Icon of program: sCrawler

sCrawler for Windows

Download now

Used sCrawler for Windows?

Editors’ Review

Download.com staff
SqrBox excels at collecting data from directory websites. It automates the process of gathering large amounts of intricate information efficiently.

  • Pros

    • Collects data from directory websites.
    • Automated and methodical browsing.
    • Gathers large amounts of information.
    • Gathers intricate information.
    • Performs URL normalization to avoid duplicates.

  • Cons

    • Focused crawling performance depends on link richness.
    • May ignore resources with a query string.
Icon of program: sCrawler

sCrawler for Windows

Download now

Used sCrawler for Windows?

Explore More

Full Specifications

GENERAL
Release
Latest update
Version
1.0
OPERATING SYSTEMS
Platform
Windows
Operating System
Windows 10
Additional Requirements
.NET Framework 3.5
POPULARITY
Total Downloads
424
Downloads Last Week
1
Report Software

Program available in other languages

Last Updated

User Reviews

1/5

1 User Votes

Developer’s Description

By SqrBox
Collect data from directory websites.
There is an endless number of reasons why a person or company would want to use web crawler software. This type of program browses the web in a designated fashion which could be automated, methodical or in an orderly way. If you're new to the term web crawler software, perhaps you've heard of spiders, bots, ants, automatic indexes, robots or scutters? They're all basically the same thing!The Purpose of Web Crawler SoftwareWhen you think of web crawling software, you probably picture the big name search engines like Google, Bing and Yahoo. Their bots crawl through web pages to determine content, relevance and indexing. By creating a copy of visited pages, they can provide faster and more accurate searches. SqrBox will tell you that you certainly do not need to be a search engine to have a need for web crawler software. You simply have to be someone who has the need to gather large amounts or extremely intricate information.Types of Web Crawler SoftwareIf you plan on using the services of a professional company such as SqrBox, you don't really need to be concerned with all the complicated lingo regarding web crawler software. Still, it's helpful to understand a few things about it.Focused Crawling - The purpose of this type of web crawler software is to download pages that appear to contain similar information. There are often some flaws associated with this method though and the actual performance of the crawler and outcome are dependent on how rich the links are on that specific topic that is being searched. This type of web crawler software is often used as a starting point to narrow down searches for further crawling. URL Normalization - web crawler software will often perform some level of URL normalization which helps reduce repetitive crawling of the same source more than once. Restricting Followed Links - In some cases, web crawler software may want to avoid certain web content and only seek out .html pages. To do this, the URL is often examined and then resources will only be requested if there are certain characters in the URL such as .html, .asp, .htm, .php, .aspx, .jspx or .jsp. web crawler software will typically ignore resources with a "?" to avoid spider traps.
Download.com
Your review for sCrawler