Unfortunately, not all search bots and spiders comply with robots exclusion rules; nor do they have to either. While we’re not lawyers (and we could be wrong), as far as we’re aware, there is no U.S. law prohibiting search engines from ignoring robots.txt exclusions on websites. However, that doesn’t mean there’s no point in using them; as most of the major search engines comply with the robots.txt exclusions, including Google and Bing/Yahoo!
We have suspicion to believe Baidu, a popular search engine in China, does not comply with robots.txt exclusions. However, some smaller search engines worldwide may also not comply with robots.txt exclusions.
The only plausible way to prevent is likely to blacklist their IPs on your Dedicated Server or your individual hosting account. You may also need to blacklist an IP range in order for it to be more effective. Otherwise, password-protect the specific directory.
* Anchor Text
Anchor text is a text which connects a link to another web page or resource. In the HTML link, like : Here is an example, Website hosting, in which “Website hosting” is the anchor text.
* Back Link
Backlink is the incoming link to a website or web page. Backlink is well-known as inbound link.
* Reciprocal link
It is one kind of link which connects one website to another website. It is a link building process and now it has become a vital part of achieving high search engine rankings.
* Link Farm
Link Farm is a set of web pages which have been mainly created for the sole purpose of growing number of incoming links to a web site. It is done in order to increase link popularity and search engine rankings. These types of pages are penalized by almost all search engines.
* Keyword / Keyphrase
It is one word or phrase that is reserved by a program, the cause is that it has special meaning. You type the keyword in when you are searching for information in the search engines.
* Keyword density
“Keyword density”is the density of keywords or number of times the phrase appears on a web page compared to the total number of words on the page. In the context of search engine optimization it can be utilized as a factor in determining whether a web page is relevant to a specified keyword or keyword phrase.
* Keyword Stuffing
Maximum SEO experts think that Keyword stuffing is considered to be a unethical SEO technique. It is the needless repetition of a keyword or keyword phrase in order to get higher ranking on search engines and this keyword or keyphrase may or may not be visible to human visitors.
* Title Tag
Title Tag is one of the very crucial parts of search engine optimization. Search engines rely heavily on title tag for relevancy of a web page against a search query. In Search Engine algorithms, Title tag provides very high relevancy for high ranking.
* Meta tag
Meta tag is HTML code. It is inserted into the header on a web page, after the title tag. In the context of search engine optimization, when people refer to meta tags, they are normally referring to the meta description tag and the meta keywords tag.