A web crawler or spider is a computer program that automatically fetches the contents of a web page. The program then analyses the content, for example to index it by certain search terms. Search engines commonly use web crawlers.
Related pages[change | change source]
- HTTrack – a web crawler released in 1998