There are a lot of pages that are indexed because they have distinct url's, but their content is not distinct.

In such a situation, I'd prefer we keep the shorter URL version of the page.

It is my opinion that the best modification would be to the function above. We'll add a local cache of the page content, as well as the page URL (we can just change it to a std::pair of QStrings for simplicity).

Then we can check the database should it have been discovered by another node.

It will become significantly easier to link in asw_neural_net once this too is switched to the CMake build system.

It would be nice to look at this again and decide what parts can be removed.

There's a lot of code here, and I'm pretty sure most of it is not being used.

|      59950 | https://dota2.gamepedia.com/Pangolier | Pangolier - Dota 2 Wiki | Kt-1delete tniilength%Kttntenfunction btreturnstringtypeof t0 rConfig                  iadbinoitceteddlohserhtrHHt0ntfunction iterHtenFtevar rthisyeAfunctiontivttSeAfunctiontiwttReAfunctiontibttforvar o0o thisj599 itNteHteHtIeHtJitt tHtH                  tegtegtItt tgtgtttaddEventListenerunloadfunctionnWndaJnConfigiadbitemeletdbatniopdneyrnfthisccbfunctiontifnfztpushteiftnWt1evar oncmolaoNvoid 0nchmtnC                  onfigiadbilufelbanenoitcetedlvoid 0nchEtvoid 0nchStvoid 0nchjtvoid 0nchDtvoid 0nchNtvoid 0nchgtvoid 0nchktt0tndaJnConfigiadbitemele |

this is less than desirable. Maybe a dictionary lookup should be imposed, and as long as x percent of the content is not like the above, it may be stored (otherwise, it slides the window)?

It would be nice to use PostgreSQL for the backend database instead of MySQL, or, better still, have a setting to change the backend database and maintain support for both.

I'm a MySQL fan myself, but, since it lacks good Qt support (at least as of when I last checked -- around Qt 5.5, I think) for threaded insertion (cannot insert from multiple threads, even synchronously).

It would be cool to set up a self-contained docker image that includes a clean database for testing purposes/making it easier to play around with this.