Article

12 juillet 2004
Sources diverses

What I like better about MSN compared to Google… so far

What I like better about MSN compared to Google

What mack said. If a page is in robots.txt, we won’t crawl it, but we can still return it as a search result if we have good evidence that the page is relevant to a query. In this case, we’ll return just the url (no title and no cached page because we didn’t fetch the page itself).
Here’s a good example of why that can help users. For a long
Source: http://googleguy.zorgloob.com/2004/07/what-i-like-better-about-msn-compared.asp