lolepezy / rpki-prover Goto Github PK
View Code? Open in Web Editor NEWYet another RPKI validator
License: BSD 3-Clause "New" or "Revised" License
Yet another RPKI validator
License: BSD 3-Clause "New" or "Revised" License
See https://tools.ietf.org/html/draft-ietf-sidrops-6486bis-03#section-6.7
Current implementation doesn't care about the fetch on which certain object was downloaded.
So if newly downloaded MFT is broken, next top-down validation will try to use it again, instead
of trying to use the previous valid version.
We need some concept of "roll-back to previous version of the MFT" to avoid dropping the whole
bunch of VRPs at once.
rpki-prover-$git-SHA
?
Or rpki-prover-$git-revision-date
?
Or rpki-prover-$version
?
It can be very impractical and complex but may be a decent security feature.
If a manifest mentioned in lastValidMft
is not used for long time, it will be removed. Also objects that it refers to may be cleaned up, so it will not be valid anymore.
Apparently, it's an important thing
NLnetLabs/routinator#461
There's still some small difference in the number of VRPs compared to routinator and rpki-client.
Figure out why is it there and fix it.
Options.Generic uses type-level descriptions and it's not possible to lift default values to type level. Replace it with something more classical like optparse-applicative, using term level only.
It's way too slow and servant-cassava
is not properly maintained.
Example: https://rpki.caramelfox.net/rrdp/notification.xml
doesn't respond to a connection using IPV6 address, but DNS returns IPv6 as the first one. As a result connection times out and we fall back to rsync. Http client should use "happy eyeballs DNS" or something similar to handle this case.
Hello,
i wanted to update my instance of rpki-prover to the latest changes in this repository. I used the same environment as one month before, but a failure occurs. Should I use a completely new installation for the compilation? Does the last compilation affect this new compilation?
Here's the error output of stack:
[59 of 60] Compiling RPKI.TopDown
/home/ubuntu/rpki-prover/src/RPKI/TopDown.hs:63:1: error:
Could not find module ‘RPKI.Fetch’
Use -v (or `:set -v` in ghci) to see a list of the files searched for.
|
63 | import RPKI.Fetch
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/AppContext.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/AppMonad.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/AppState.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/CommonTypes.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Config.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Domain.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/Api.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/HttpServer.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/Messages.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/Types.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/UI.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Logging.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Metrics.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Orphans/Generics.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Orphans/Serialise.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parallel.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/Cert.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/Common.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/CRL.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/GBR.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/MFT.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/ROA.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/SignedObject.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Parse.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Reporting.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Repository.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Resources/IntervalSet.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Resources/Resources.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Resources/Types.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RRDP/Http.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RRDP/Parse.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RRDP/RrdpFetch.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RRDP/Types.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Rsync.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RTR/Pdus.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RTR/RtrServer.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RTR/RtrState.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RTR/Types.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/AppLmdbStorage.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/AppStorage.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/InMemory.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/LMDB.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/Map.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/MultiMap.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/Storable.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/Storage.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Database.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/MakeInMemory.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/MakeLmdb.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Repository.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Sequence.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/TAL.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Time.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Util.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Validation/Crypto.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Validation/ObjectValidation.hi Decoding failure: Invalid magic: e49ceb0f
Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Validation/ResourceValidation.hi Decoding failure: Invalid magic: e49ceb0f
Completed 208 action(s).
-- While building package rpki-prover-0.1.0.0 using:
/home/ubuntu/.stack/setup-exe-cache/aarch64-linux/Cabal-simple_mPHDZzAJ_3.2.1.0_ghc-8.10.4 --builddir=.stack-work/dist/aarch64-linux/Cabal-3.2.1.0 build lib:rpki-prover exe:rpki-prover --ghc-options " -fdiagnostics-color=always"
Process exited with code: ExitFailure 1
Some tests in RPKI.ResourcesSpec
are still failing sometimes.
At the moment it is still a little big of a moving target
https://tools.ietf.org/html/draft-ietf-sidrops-6486bis-03
Strict MFTs, 'failed fetch' concept, etc.
Create Deb and RPM (using static binary?)
Currently it is 8gb, which but can be too small for certain cache lifetimes. Figure out some simple heuristics that would allow to calculate it?
Messages like (on Linux)
rpki-prover:w[2193433]: segfault at 179 ip 00007f9c8878602d sp 00007f9b48ff4db0 error 4 in liblmdb.so.0.0.0[7f9c88783000+10000]
The same happens on macos.
It can be reproduced by sending an async exception to a thread running RO lmdb transaction. Doesn't happen every time, but happens pretty reliably. Workaround -- do not cancel
asyncs running transactions. It probably need to be fixed in the lmdb-high-level library.
Currently logging happens with MVar locking, which is not a very scalable option.
A more efficient thing would be to use co-log-concurrent (e.g. https://gist.github.com/qnikst/f38bbaee033aaa3df8a9d115c951182a).
Maybe as a CLI option?
Generally, it would be beneficial to have
fetchURI uri1 <|> ... fetchURI uriN
logic regardless of which one is RRDP and which one is rsync to support all potential extensions of the RFCs, multi-URL repositories or whatever comes up over time.
Most (all?) TALs contain two URIs. Currently, if the first URI is HTTPS, and the second is rsync, it will try the rsync one.
RFC 8630 recommends preferring the HTTPS URI over the rsync one if both are present:
https://datatracker.ietf.org/doc/html/rfc8630#section-4
If an object is referenced multiple times it will be marked as creating "reference cycle" which is not necessary true. Reference cycles can only happen between certificates and manifests.
It can be ugly, but compact and informative.
Metrics that are currently exposed with "/api/metrics" should (also) be exposed as prometheus metrics.
See, http://hackage.haskell.org/packages/search?terms=prometheus
and ask around which library doesn't suck nowadays.
It should understand that the data in cache cannot be deserialised by an updated version. In this case it should automatically wipe the whole DB (or just the relevant part?), complain about it in the log, re-dowload repositories and continue instead of complaining about weird deserialisation errors.
I will allow to safely do "withdraw" by checking is the withdrawing repository is the same as published the object.
It will help to
Generate Linux static binary and push docker image to dockerhub for every 'releases/x.x.x' tag.
Don't re-download notification.xml in case ETag says that it's not needed.
RFC insists on validating that XML corresponds to the schema. So error message should be more predictable and say that the whole file is not what RRDP parser expects at all.
That happens after copying the data to a new environment, i.e. using
RPKI.Store.AppStorage.defragmentStorageWithTmpDir
. New environment is
After restarting the process the new environment is perfectly usable.
It probably has to do with some LMDB quirks when one has more than one environment in one process.
Since "rsync --delete" is not used, there's potential for these copies to grow indefinitely.
Clean them up every few days/weeks.
These libraries are used in RPKI.RRDP.Http
.
They are present in the Stackage 16.27 LTS, but not in nightly and are not updated for long time.
Since we don't really care how to stream-download files, just replace it with http-conduit or pipes, ]
or anything that is better supported.
Current implementation uses serialise
and it's pretty not bad. The hot-spot in profiling is deserialising objects read from LMDB, so it's better to chose a library that does it with least CPU time necessary.
store
instead of serialise
and benchmarkThat happens for ROAs coming from rpki.cnnic.cn that by some reason includes them on the manifest.
Read https://tools.ietf.org/html/draft-ietf-sidrops-6486bis-03 carefully once again and decide if it's a bug
or should be done this way.
In the following scenario things can go really wrong.
--cache-lifetime
the ROA is GC-ed, because it was never touched by a top-down validation.While in practice this scenario doesn't make any sense, it is still technically valid behaviour of a RRDP repository and must be taken into consideration.
Current implementation using hs-certificate has exactly the same problem of interpreting signature algorithm without parameters and with NULL as equal.
See
http://sobornost.net/~job/arin-manifest-issue-2020.08.12.txt,
https://www.arin.net/announcements/20200812/.
This needs modifying hs-certificate or implementing another custom parser for signature algorithm.
Implement proper signal handling so that the process reliably stops. Review exception propagation in concurrent/parallel code. That is required for systemd integration.
RSS of the process on Linux looks like it indefinitely(?) growing and it's hard to say why.
Haskell heap seems to be perfectly fine, but the total RSS grows over time.
That does not seem to happen on MacOS, so there's a chance that it's some sort of misinterpretation.
https://rpki.admin.freerangecloud.com/rrdp/notification.xml
is fetched 3 times, once for each TA.
Sharing TVar PublicationPoints
between TA threads leads to weird race,
use something better.
For installations that don't have rsync
, rpki-prover builds and starts fine, but will not function. This error will be seen in the output
[Error] [21 Apr 2021 15:33:59.193 +00:00] [<unknown loc>] [ThreadId 11] Something weird happened rsync: startProcess: runInteractiveProcess: exec: does not exist (No such file or directory), exiting.
Recommendation: document requirement in README.md. Probably obvious; opening this issue in case others search for this error.
https://tools.ietf.org/html/draft-ietf-sidrops-6486bis-03#section-6.1
A manifest SHOULD contain exactly one CRL (.crl) file and it MUST be at the location specified in the CRLDP in the manifest's EE certificate.
Add --log-level
parameter with WARN, INFO (default), DEBUG levels.
It needs to check that the prefix on ROA is covered by the resource set on the EE certificate.
There are still TODOs to fix that: an object (defined by hash) can have multiple locations but only one is stored. It would be very useful to store them all to see duplication.
Should the script mkdirs.sh
be run with an argument?
#!/bin/sh
ROOT=$1
mkdir -p ${ROOT}/cache
...
run without an argument, the script tries to make directories in /
This needs to be implemented https://tools.ietf.org/html/rfc8416.
Applied assertions and BGP filters the VRPs of the validateTA
function so that the results gets into RTR server.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.