[Feature Request] Integrate IPinfo's free database for ASN+country enrichment, filters, and eliminating HTTP calls about maltrail HOT 4 CLOSED

@stamparm, I really appreciate your thoughts. I thought the data would provide a better user experience. I will close the ticket.

If in the future you want to explore IPinfo's free data for this project or any other, please reach out to me or reopen the ticket. I will be extremely happy to help and be a part of the project. Thank you very much for reviewing the proposal.

from maltrail.

Furthermore, a large number of HTTP API calls can slow down the load time. This issue can be easily addressed by switching to the MMDB database. <- there is no large number of HTTP API Calls. only when user hovers over the IP, the call is being made.

AFAICS, your proposition would be a huge performance hit on the server side (as all IPs would require an IP->ASN call), while the basic premise of the Maltrail's client was that all the post-processing of sensor data should be offloaded to the client (to relieve the server machine which is usually running the sensor in parallel)

from maltrail.

Thank you for reviewing the request, @stamparm. I have a very basic understanding of Maltrail and have only been using it on a single server, which serves as both a sensor and a server on the same machine. I tried to install it on another machine, but the process keeps getting killed because it is a micro VM (low ram, process getting killed). My apologies if I did not conduct thorough research beforehand.

there is no large number of HTTP API Calls. only when user hovers over the IP, the call is being made.

I believe at the initial load, all the country-level information for the flags is being provided from https://stat.ripe.net/data/geoloc/data.json

Then, subsequent calls are made to the https://stat.ripe.net API endpoint as well upon hover. The information returned is WHOIS data.

At least for the onload call to get the country information, we could obtainthe information from the IPinfo database, as the database provides both ASN and country information.

while the basic premise of the Maltrail's client was that all the post-processing of sensor data should be offloaded to the client (to relieve the server machine which is usually running the sensor in parallel)

If you implement the IPinfo database, you will be able to eliminate the need to get ASN and country-level information. The sensor is downloading the threat intel feeds on their side. Is it possible to download the MMDB file on the sensor end and provide the data from there instead?

My recommendation is that by using the IP to Country ASN database, you are extending country-level identification with ASN information in one single database. Maltrail will not require calls to the RIPE Stats endpoint on load to get both country and ASN level information from the local IPinfo DB.

Please let me know what you think.

from maltrail.

1. those IP to country calls are made for the current page of results, that's true. nonetheless, they are quite fast (i believe that you noticed) because RIPE is providing such a service for everybody
2. golden rule of engineering is if it works, don't mingle. honestly, i don't see any improvement by downloading a 3rd party DB to the maltrail server - also, it would introduce the pre-processing of log entries, which is not inline with the "fat client" story i gave in previous comment

from maltrail.