How does the search engine robot
On this subject there is a detailed faq on Yandex at http://help.yandex.ru/webmaster/?id=995296
Detailed, but quite informative. For example, a direct question, and given himself: "What is a search engine robot, and what he does?" Yandex himself and says:
Robot (English crawler) maintains a list of URL, which can be indexed, and regularly pumps out the corresponding documents. If the analysis of the document robot detects [...]
Pp. 1 of 1 1