Unfortunately, not all search bots and spiders comply with robots exclusion rules; nor do they have to either. While we’re not lawyers (and we could be wrong), as far as we’re aware, there is no U.S. law prohibiting search engines from ignoring robots.txt exclusions on websites. However, that doesn’t mean there’s no point in using them; as most of the major search engines comply with the robots.txt exclusions, including Google and Bing/Yahoo!
What search engines do not comply with robots.txt exclusions?
We have suspicion to believe Baidu, a popular search engine in China, does not comply with robots.txt exclusions. However, some smaller search engines worldwide may also not comply with robots.txt exclusions.