Skip to main content

View Post [edit]

Poster: brewster Date: Apr 12, 2004 4:06pm
Forum: etree Subject: Download Speeds: how you can help

Thank you all for the feedback. We are trying to figure out how we can get the most from we have.
If we figure it out together we all win. Here are a few things going on:

Something happened today to cause our website usage to spike. We dont know what, but here is the evidence of a 2x spike:
http://homeserver.archive.org/http_rrd/www.html

Jon Aizen and the ops guys at the archive figured out how to speed up the website dramatically last friday. Yippie. This might be retaining some users that got sick of the past delays.

Also, we are constantly hitting our bandwidth cap from our ISP:
http://homeserver.archive.org/new/index.html
We are negotiating to buy more, but we dont want to buy too early because it costs us real money and we can use the money for other things (machines, staff, digitization, etc). Even if we wanted to, it will take us a couple of months to get more, but more is coming.
http://homeserver.archive.org/new/index.html

An Archive guru (jim shankland) wrote a bandwidth sharer on our linux router (very very cool) that seems to limit most people to the same max download rate, which seems to be more effective than the older cisco router cap used to work. this is goodness becase we can then use 90% of our bandwidth to keep everyone in music and movies.

What we could use:
* an inexpensive way to monitor our download rate to a bunch of typical user systems (dsl, cable, university). keynote for free, or something.
* sites that can offer 100Mb/sec of bandwidth or more to the commons (read: help host popular files). We have a cache system (freecache.org) that can automate the distribution of the load.
* dont download what you dont need. pls don't just hoard.
* download or stream low quality if you are just listening (and not really picky). lossless quality if you are trading.
* use the p2p systems as we set them up again (we tried 4 different ones and they were not very effective because we did not have concentrated enough hotspots)
* donate money to the Internet Archive. We really are run on donations. Please help.
* If you work for an ISP, offer us Internet transit for about $10-15 a month for a megabit/sec.
* If you work for a dsl or cable provider, offer us a free peering agreement.
* If you work for Internet2 or calren or cenic, please help us serve those users without large fees.
* in San Francisco, use sflan.org to download stuff because it does not go over our outside bandwidth.
* Anything we are missing?

Bottom line: We will get more bandwidth, but being efficient with non-profit resourses is always a good idea. Together we are making the commons work.

-brewster




Reply [edit]

Poster: Randy D Date: Apr 14, 2004 4:17am
Forum: etree Subject: Re: Download Speeds: how you can help


Couple more thoughts/suggestions.

I think you are seeing a spike in intrest due to the advent of the Grateful Dead shows, I've scene chat on many boards about it.

May I suggest people download stuff at night, just before you go to bed. I was looking at some of the stats and there seems to be a low from 9:00PM to 6:00AM (not sure what time zone that chart was for). . .plus what do you really care if it takes a long time if you're sleeping anyways.

Another suggestions, is there a way to disable the higher stream if too much bandwidth is being used at one time?? Basically only allow the lower MP3 to be streemed (I guess I'm assuming that there's a lot of people streeming music durring the day, if this is not the case it probably wouldn't save anything.)

One last thing, is there an "instant" graph or way to see how many users vs how much bandwidth. . .if so putting a link on the main Audio page to it and a suggestion that people check before DL may help you save.

Just suggestion, love the Archive (not only the audio section).
Randy

Reply [edit]

Poster: datgeek Date: Apr 14, 2004 3:33pm
Forum: etree Subject: Re: Download Speeds: how you can help

How about the BitTorrent thing that was done in the past, how hard was that to impliment on archive again ? or would we need to have the tracker site off of archive ?

Reply [edit]

Poster: Tradewalker Date: Apr 14, 2004 10:45pm
Forum: etree Subject: Re: Download Speeds: how you can help

I know this may sound a bit controversial, but how about charging a few for either each show download, or on a monthly basis (but only for the lossless shows).

It needs only be a dollar a show. Let's face it I-tunes seems to be coping well at 99 cents per track (I know they are out to make money, but the revenue generated could be ploughed back into hardware or bandwidth costs).

I've only just joined this group after noting a post on a newsgroup about the fabulous download speeds compared to torrent sites such as www.sharingthegroove.org. However, I must say I'm seeing very little difference in speeds between the two, so allowing downloads via torrents as well as FTP, could be a decent solution.

Reply [edit]

Poster: lazarus0314 Date: Apr 15, 2004 2:50am
Forum: etree Subject: Re: Download Speeds: how you can help

just my quick $.02...to impiment a charge for D/Ling shows, my go against many of the policys for trade-friendly bands and strictly non-commercial trade. just a thought.
And i'm by all means not a pro, but it may be possible to remain a non-commecial not-for-profit organization and still impiment a "membership fee" or something of the sort. It may not be as profitable as a per show fee but could stop the freeloaders and generate some sort of funds and hopefully increase bandwidth and equiptment.

James (truely grateful for archive.org)Boncek

Reply [edit]

Poster: ssj4android Date: Apr 15, 2004 3:19am
Forum: etree Subject: Re: Download Speeds: how you can help

What happened to freecache? Doesn't that reduce the bandwidth by a lot?

Reply [edit]

Poster: brewster Date: Apr 15, 2004 1:34pm
Forum: etree Subject: Re: Download Speeds: how you can help

Freecache was designed to help, but getting folks that would be up for hosting freecache sites has been slow. Larry Lessig (yippie) signed up, but otherwise it was slow going.

if anyone has a serious pipe to share that would be great. for instance, universities on internet2 would be perfect.

-brewster

Reply [edit]

Poster: abulluck Date: Apr 13, 2004 12:02pm
Forum: etree Subject: Re: Download Speeds: how you can help

Who's the moron who had the brilliant idea to upload every Dead show on the archive? Why not do it a year at a time? Keep them up there for two weeks to a month, then throw up the next year. I feel sorry for the bums who are going to spend every waking our downloading all of those shows. It's really sad. Why not give your money to charity?

Reply [edit]

Poster: brewster Date: Apr 13, 2004 12:06pm
Forum: etree Subject: Re: Download Speeds: how you can help

Well, that decision was by all of us, of which I am the digital librarian, so I hold alot of the responsibility.

While we have growing pains, I am proud of the progress on bringing lots of interesting material to lots of people. The Grateful Dead was seminal in non-commercial starting the tape trading to a mainstream practice-- I am proud that we have the ability to store and provide access to their creative works.

I think we can figure out how to make this all work, and not even go broke. Thank you all for helping.

-brewster

Reply [edit]

Poster: norvarchivist Date: Apr 13, 2004 12:22pm
Forum: etree Subject: Re: Download Speeds: how you can help

I am thrilled to see all these Dead shows sufacing in PHENOMINAL quality...in the days of analog, we all knew that someone had the "serious shit" unbelievable...plus definitely take note of the sources...these early tapers
(wise,glassburg, others) are myy heros...they were all pulling amazing tapes DECADES ago with a fraction of the technolgy available to modern tapers...

Thank you to anyone involved w/ this project

Reply [edit]

Poster: abulluck Date: Apr 13, 2004 12:20pm
Forum: etree Subject: Re: Download Speeds: how you can help

Didn't mean to sound too harsh, but it is an extremely bold plan. Hope you all can find a way to remedy this because these speeds are depressing.

Reply [edit]

Poster: bisq Date: Apr 13, 2004 1:35pm
Forum: etree Subject: Re: Download Speeds: how you can help

Once again, I don't think you can blame all of your speed problems on just the dead being here. I don't think it is that different with or without them and if they are having an effect on you it will probably die down in a little while.

Reply [edit]

Poster: HipGerm Date: Apr 13, 2004 2:32pm
Forum: etree Subject: Re: Download Speeds: how you can help

Downloading a gig at a time is pretty serious stuff - i used to use Furthurnet and thought it was the best invention every made...One can get a show in about 5-6 days or if one was lucky 1-2 days...Here at the Archive it took me about 2-3 hours (that was about 6 months ago running at about 160-180 kb/sec - i think) now it takes about 6-8 hours to get a show...@ about 60-80 So i lost about 100 kb/sec of transfer rate... big deal. But i know others have it worse off. I can't complain and others shouldnt either if they can still download a show when they go to bed and wake up in the morning to a complete download. I commend all the efforts to control the bandwidth so all have a fair share... here is my take: I recommend a user agreement that a user MUST read...so everyone who downloads is aware of how to use the archive responsibly - this may sound silly but this place will only get bigger and more prone to traffic jams... A responsible user doesn't download every show just BECAUSE he/she can. Picking and choosing... how to choose - stream or download lossy first to check if you might like the show...Sure, i am guilty of downloading a show and deleting becuase there was too much crowd noise or too many errors...thats why we have lossy, people to review and rate and also stream options, The user agreement could show tutorials on how to use p2p or bit torrents or the like The more aware the users are, the more conscious that everyone can do their part by helping will greatly increase the chances of good experiences...and further promote the wonderfullness here...a Also, I know a lot of us used to trade through mail...at about 2-3 bucks per trade, maybe there could be a trading forum on the archive, - not only would you get the best shows , you would, in away, be donating to the archive by not using up its bandwidth, - just a thought... Some people that i know download a show and then put up a B&P offer on the bands website forum A one show per day limit might be in order if you don't use p2p or bit torrent ... or maybe a bandwidth cap is in order... I apologize if the technical know how is not all there, i am not an expert. But these are some ideas It seems as though we have too good of a thing going on here and some people are abusing it, while ruining it for the responsible users...however some are ignorant that they are actually doing wrong. Maybe there will be a cap or limit - and people might get pissed for a little while but, hey, know your role and find other means and methods, friends don't let friends drink and download, especially in a heavy traffic area be responsible later all ...gotta check on the 5 sound tribe shows i just downloaded
This post was modified by HipGerm on 2004-04-13 21:32:58