What I like better about MSN compared to Google… so far
What I like better about MSN compared to Google
What mack said. If a page is in robots.txt, we won’t crawl it, but we can still return it as a search result if we have good evidence that the page is relevant to a query. In this case, we’ll return just the url (no title and no cached page because we didn’t fetch the page itself).
Here’s a good example of why that can help users. For a long