Chilkat Spider ActiveX for Windows

Name: Chilkat Spider ActiveX
Rating: 1 (1 reviews)
Author: Chilkat Software

By Chilkat Software
Free
User Rating

Download now

Used Chilkat Spider ActiveX for Windows?

Developer’s Description

By Chilkat Software

Cache, "avoid" patterns, handle HTTP URLs automatically.

Crawl a Web site with this free ActiveX spidering component. Advanced features include caching, "avoid" patterns, and robots.txt compliance. Automatically handles HTTPS URLs. Includes safety measures for avoiding large / infinite size Web pages or infinitely growing-in-length URLs. Includes a wind-down features for crawling a maximum number of pages per site.