gkScraper

gkScraper is a web scraping library for .NET Framework written in VB.NET.

It can navigate programmatically on web sites looking for links and information and collecting them.
Data are located using XPath syntax and extracted as node, HTML or text.

Navigation features include:

• GET/POST requests subission,
• http/https support,
• cookies support,
• redirection response,
• custom headers attributes,
• multipart form data submission,
• ftp protocol helper methods

Data scraping features:

• Absolute and relative XPath searches,
• Find single node o nodes collection,
• HTML o text extraction,
• Parsing and cleaning functions,
• include or exclude html tag and Attributes

Debugging and Tracing

To enable debug features there is a .Debug property in gkScrapeBot class.
Setting this property true, every navigation the parser write down (and overwrite) both the original html document received from host and the parsed XML document.
These files are located in the running process folder ("debug.html", "parser_orig.xml").

The libary implements also logging and tracing functionalities making use of NLOG library.
Setting the .Debug propery enable Debug logging level.
Setting the .Trace propery enable Trace logging level.

Test Scraping

This solution contains also a test project.
It scrapes products data from a demo e-commerce site and write results to a XML file and insert them into an MS Access Database.
Products are visible only to logged users.
The Test Project firstly securely log in and after collect data from the first page of displayed products.

Note

The library contains an helper class to insert data into a MS Access Database.
In order to use MS Access features you need to install the Microsoft.ACE.OLEDB.16.0 provider distributed with MS Access o Runtime library. Be sure to target the right platform x86 or x64 according to the runtime installed.
If you don't want make use of this features you can comment lines without affecting the scraping.