Skip to main content

View Post [edit]

Poster: Melkor Date: Aug 16, 2003 2:42pm
Forum: web Subject: robots.txt question

This is a problem that has just come,
when a site dies and the domain name goes back into the list of availible names,
Here is the problem someone else 'buys' the name and sets up a robots.txt,
>what happens to all the old entries in the Wayback database?

Reply [edit]

Poster: Igor Ranitovic Date: Aug 18, 2003 3:42am
Forum: web Subject: Re: robots.txt question

robots.txt rules are followed retroactively.

Reply [edit]

Poster: ancient Date: Aug 29, 2003 3:23am
Forum: web Subject: Re: robots.txt question

Here's an associated issue. I recently added a robots.txt file to a site. Someone had used files from IA (long deleted from the active web site) for bad purposes. I am now considering getting rid of the URL. If someone reactivates it WITHOUT a robots.txt file, will my old files show up again?

Reply [edit]

Poster: Igor Ranitovic Date: Aug 29, 2003 7:45am
Forum: web Subject: Re: robots.txt question

Could you please forward this question to info@archive.org.
Thank you.
i.

Reply [edit]

Poster: ancient Date: Aug 31, 2003 12:22pm
Forum: web Subject: Re: robots.txt question

Hi Igor, I tried info@archive.org. The mailer-daemon replied with a permanent error (mailbox full). Any alternatives? Has this come up before? Does anyone know the answer?

Reply [edit]

Poster: molly Date: Sep 2, 2003 3:32am
Forum: web Subject: Re: robots.txt question

hello! our info@archive box is overflowing with the spam virus. we are trying to get a handle on it and sort through all the messages so we don't miss any exclusion requests. if you recieve a bounce message, please please send your request again!
thanks for your patience,
molly