Crawler-based search engines have three major elements. The first is the spider, also called the crawler, which visits a web page, reads it, and then follows links to other pages within the site.
Source: wiktionary
Crawler
Synonyms for "crawler" (19 found)
Ranked by relevance and common usage.
Related word relations
OpenGloss and ConceptNet supply richer edges like generalizations, collocations, and derivations.
Sample sentences
2 total sentences available.
These serve as an algorithmic way to judge the “mobileness” of a site, and similar algorithms are likely to be used by the search crawlers.
Source: wiktionary