Re: squid-dev on Google

On Mon, 12 Aug 2002, Andres Kroonmaa wrote:

> On 13 Aug 2002, at 0:10, Robert Collins <robertc@squid-cache.org> wrote:
>
> > > I was trying to search squid-dev archive, and it appears that it
> > > isn't in Google. Is that intentional?
> >
> > Yes, it's excluded in robots.txt. I'm not sure when the decision was
> > made though.
> >
> > > Would it be bad if Google indexed squid-dev archive?
> >
> > IMO, no. It could be quite useful.
>
> IMO too it would very useful.
> Duane, Alex, care to comment on why its excluded?

I do not know the reason, but I suspect that it was due to (other?)
robots harvesting for e-mail addresses and abusing the site. If site
performance allows, I agree that it would be good to have the archive
indexed by Google and other well-behaving engines.

Alex.
Received on Tue Aug 13 2002 - 10:48:45 MDT