WebLight for Linux for Linux

WebLight for Linux is a web crawler that lets you efficiently maintain xml sitemaps and find markup, CSS and link problems so you can maintain error free, fully indexed sites without sacrificing productivity. The heart of WebLight is its fast web crawler that makes validating and mapping entire sites, even very large sites, easy. You simply tell WebLight where to start and use rules like robots.txt to control the resources it validates and the links it follows. You can use as many starting URL's and rules as necessary to scan your all of the resources you want and none that you don't. Unlike most web crawlers that only scan public html pages, WebLight can scan most commonly used web resources - CSS, (x) html, atom, RSS, and xml sitemaps - on local disks, public, and private web sites.