Apps

Web Crawler and Scraper for Files and Links


About Web Crawler and Scraper

Web Crawler can be used to get links, emails, images and files from a webpage or site. Web Crawler has a simple and intuitive interface.

The crawler is multithreaded and optimized for performance. It scans the webpage based on MIME types and file extensions, so it can find hidden links.

Two applications are included in the package. A Windows Forms application and a new WPF application with extended functionality. The “Deep crawl” feature allows the crawler to search all the linked pages from the selected website.

After crawling, the Web Crawler will save all links and e-mail addresses to the selected folder, along with all the crawled files.

The WPF crawler/scraper allows the user to input a regular expression to scrape through the webpages. The new application gives the user a greater control over the crawling process.

How to use the Windows Forms crawler

On the top is a box for entering the URL to crawl. Underneath the URL box is a folder in which to save the crawled files. The last box is for file extensions that the crawler should look for. If the file extensions box is left empty, then the program only looks for links and e-mails on the page and saves them to the linkList.txt and emailList.txt files in the output directory.

The application is primarily meant for subpage crawling, but can crawl a whole website when the “deep crawl” option is checked. This option is very resource intensive as it tries to make parallel connections to the server for better performance.

How to use the WPF crawler and scraper

The WPF has a similar interface to the Windows Forms crawler/scraper. The first three boxes have the same functionality. The last box is optional. It can be used to enter a regular expression by which to search each crawled webpage for anything that can be matched by a regular expression. This can be used to search for phone numbers, names, locations etc.

The crawler is multithreaded and optimized for performance. It scans the webpage based on MIME types and file extensions, so it can find hidden links. There is some support for AJAX calls. The new engine allows for more control over what is crawled and the depth and scope of the crawl. The user can also control the number of concurrent threads that the program will use to scrape webpages.

About the rating

It seems that only people who do not like the product or could not use it properly decide to rate it. If you like the application, you can help the developer by rating it up.

Download Web Crawler and Scraper for Files and Links For Free

You can download Web Crawler and Scraper for Files and Links for free. Download the web-crawler-and-scraper-for-files-and-links.zip file using the links below.

How can you install?

For detailed information, you can read the README.MD file in the zip file.

Download web-crawler-and-scraper-for-files-and-links.zip

I have given 1 main and 5 alternative download links below for you. You can download it for free using the links.

File Information

  • Category: Apps
  • File Size: ~3 MB
  • File Type: .zip (compressed)
  • File Name: web-crawler-and-scraper-for-files-and-links.zip

Download Links

1. Direct Links
WebSite Source Link
Direct Download Direct Click Here
Direct Download 2 Direct Click Here
Direct Download (Proxy) Direct Click Here
Google Drive Alternative Click Here
Yandex Disk Alternative Click Here
pCloud Alternative Click Here
MediaFire Alternative Click Here
Dropbox Alternative Click Here
Mega Alternative Click Here
One Drive Alternative Click Here
Amazon S3 Alternative Click Here
Tags:
Categories
Site Templates
14800
WordPress
14459
Mobile
7525
UI Templates
7144
HTML5
6716
Mobile Games
5640
eCommerce
3029
Marketing
2381
PHP Scripts
2305
CMS Themes
1334
Template Kits
1322
JavaScript
1287
Plugins
1010
CSS
713
Muse Templates
703
.NET
646
Blogging
338
Apps
300
Static Site Generators
91
Skins
73
Edge Animate Templates
71
Forums
20