Libraries that analyze the content of websites.
Scalable crawler with downloading, url management, content extraction and persistent.
Simple and lightweight web crawler.