Giter Club home page Giter Club logo

rpki-prover's People

Contributors

dadepo avatar lolepezy avatar wibisono avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

rpki-prover's Issues

Replace Options.Generic with something simpler

Options.Generic uses type-level descriptions and it's not possible to lift default values to type level. Replace it with something more classical like optparse-applicative, using term level only.

Handle IPv6/IPv4 fallbacks

Example: https://rpki.caramelfox.net/rrdp/notification.xml doesn't respond to a connection using IPV6 address, but DNS returns IPv6 as the first one. As a result connection times out and we fall back to rsync. Http client should use "happy eyeballs DNS" or something similar to handle this case.

Error occurs while compiling

Hello,

i wanted to update my instance of rpki-prover to the latest changes in this repository. I used the same environment as one month before, but a failure occurs. Should I use a completely new installation for the compilation? Does the last compilation affect this new compilation?

Here's the error output of stack:

[59 of 60] Compiling RPKI.TopDown
/home/ubuntu/rpki-prover/src/RPKI/TopDown.hs:63:1: error:
    Could not find module ‘RPKI.Fetch’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
63 | import           RPKI.Fetch
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^


Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/AppContext.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/AppMonad.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/AppState.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/CommonTypes.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Config.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Domain.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/Api.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/HttpServer.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/Messages.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/Types.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Http/UI.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Logging.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Metrics.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Orphans/Generics.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Orphans/Serialise.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parallel.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/Cert.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/Common.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/CRL.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/GBR.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/MFT.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/ROA.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Internal/SignedObject.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Parse/Parse.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Reporting.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Repository.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Resources/IntervalSet.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Resources/Resources.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Resources/Types.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RRDP/Http.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RRDP/Parse.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RRDP/RrdpFetch.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RRDP/Types.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Rsync.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RTR/Pdus.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RTR/RtrServer.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RTR/RtrState.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/RTR/Types.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/AppLmdbStorage.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/AppStorage.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/InMemory.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/LMDB.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/Map.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/MultiMap.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/Storable.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Base/Storage.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Database.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/MakeInMemory.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/MakeLmdb.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Repository.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Store/Sequence.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/TAL.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Time.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Util.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Validation/Crypto.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Validation/ObjectValidation.hi Decoding failure: Invalid magic: e49ceb0f

Warning: Failed to decode module interface: /home/ubuntu/rpki-prover/.stack-work/dist/aarch64-linux/Cabal-3.2.1.0/build/RPKI/Validation/ResourceValidation.hi Decoding failure: Invalid magic: e49ceb0f
Completed 208 action(s).

--  While building package rpki-prover-0.1.0.0 using:
      /home/ubuntu/.stack/setup-exe-cache/aarch64-linux/Cabal-simple_mPHDZzAJ_3.2.1.0_ghc-8.10.4 --builddir=.stack-work/dist/aarch64-linux/Cabal-3.2.1.0 build lib:rpki-prover exe:rpki-prover --ghc-options " -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1

Segfault in LMDB

Messages like (on Linux)
rpki-prover:w[2193433]: segfault at 179 ip 00007f9c8878602d sp 00007f9b48ff4db0 error 4 in liblmdb.so.0.0.0[7f9c88783000+10000]

The same happens on macos.

It can be reproduced by sending an async exception to a thread running RO lmdb transaction. Doesn't happen every time, but happens pretty reliably. Workaround -- do not cancel asyncs running transactions. It probably need to be fixed in the lmdb-high-level library.

Implement RRDP -> rsync fall-back

Maybe as a CLI option?

Generally, it would be beneficial to have

fetchURI uri1 <|> ... fetchURI uriN logic regardless of which one is RRDP and which one is rsync to support all potential extensions of the RFCs, multi-URL repositories or whatever comes up over time.

Fix "reference cycle"

If an object is referenced multiple times it will be marked as creating "reference cycle" which is not necessary true. Reference cycles can only happen between certificates and manifests.

Introduce DB versioning

It should understand that the data in cache cannot be deserialised by an updated version. In this case it should automatically wipe the whole DB (or just the relevant part?), complain about it in the log, re-dowload repositories and continue instead of complaining about weird deserialisation errors.

Fix the 'invalid cursor' problem

That happens after copying the data to a new environment, i.e. using
RPKI.Store.AppStorage.defragmentStorageWithTmpDir. New environment is

  • successfully created
  • filled in with data
  • old env is closed
  • new one is re-opened
  • new one is unusable (with LMDB code 22)

After restarting the process the new environment is perfectly usable.

It probably has to do with some LMDB quirks when one has more than one environment in one process.

Replace `streaming-utils` and `json-stream` with something more alive.

These libraries are used in RPKI.RRDP.Http.
They are present in the Stackage 16.27 LTS, but not in nightly and are not updated for long time.
Since we don't really care how to stream-download files, just replace it with http-conduit or pipes, ]
or anything that is better supported.

Test/benchmark different serialisation libraries

Current implementation uses serialise and it's pretty not bad. The hot-spot in profiling is deserialising objects read from LMDB, so it's better to chose a library that does it with least CPU time necessary.

  • try store instead of serialise and benchmark
  • try TH instead of Generic deriving and benchmark

GC-ing unused objects can lead to invalid repositories

In the following scenario things can go really wrong.

  • A RRDP repository publishes a ROA and a manifest that doesn't refer to the ROA.
  • After time defined by --cache-lifetime the ROA is GC-ed, because it was never touched by a top-down validation.
  • After that the repository publishes a new manifest that now refers to the ROA
  • Top-down validation considers the manifest and the CA invalid, since it has a dangling reference.
  • Earth collapses to a black hole.

While in practice this scenario doesn't make any sense, it is still technically valid behaviour of a RRDP repository and must be taken into consideration.

Ctrl+C doesn't always stop the process

Implement proper signal handling so that the process reliably stops. Review exception propagation in concurrent/parallel code. That is required for systemd integration.

Fix memory issue on Linux, reduce RSS

RSS of the process on Linux looks like it indefinitely(?) growing and it's hard to say why.
Haskell heap seems to be perfectly fine, but the total RSS grows over time.

That does not seem to happen on MacOS, so there's a chance that it's some sort of misinterpretation.

Document requirement for rsync

For installations that don't have rsync, rpki-prover builds and starts fine, but will not function. This error will be seen in the output

[Error]   [21 Apr 2021 15:33:59.193 +00:00] [<unknown loc>] [ThreadId 11] Something weird happened rsync: startProcess: runInteractiveProcess: exec: does not exist (No such file or directory), exiting.

Recommendation: document requirement in README.md. Probably obvious; opening this issue in case others search for this error.

Add validation result history with search

  1. Decouple VRP history from validation result history -- it doesn't make so much sense to store 100 versions of VRPs
  2. Add API calls to search through validation result history, there's an obvious use case for it "figure out if and when a problem with this URL/prefix/hash happened"
  3. Add some UI for it

Store all locations for every object

There are still TODOs to fix that: an object (defined by hash) can have multiple locations but only one is stored. It would be very useful to store them all to see duplication.

Does mkdirs.sh require an arguement?

Should the script mkdirs.sh be run with an argument?

#!/bin/sh

ROOT=$1

mkdir -p ${ROOT}/cache
...

run without an argument, the script tries to make directories in /

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.