A list ofsearch engine crawlers that support wildcards, and the Allow: extension in robots.txt configuration files.
Continued entry »

There are thousands of blogs on how web developers can improve their search ranking. But there are (to my knowledge) non on how the search engine users can do better searches to get better results! Now is not that a bit odd?
Continued entry »

Even in sites' robots.txt files sitemaps have start making there presence known! Ask! Search, Yahoo! Search, Windows Live, and Google have all said they will start looking for sitemaps in the robots.txt files. Sitemaps auto-discovery have been born!
Continued entry »