As a Holochain Core team we need to know how much of our code has tests written for it.

In the go implementation we had a code coverage that ran to check our testing. We need to:

• find a fust code coverage tool for our CI
• add it to travis
• decide on code coverage % targets

in the go version all headers had a type and there were helper fns like TopType that gave types first class support

i think we can add the type, but get away without forcing TopType into the source chain trait

child of #117

As an container developer and as a core developer, I want web access to rust library documentation, so as to be able to understand and navigate the codebase easily.

cargo doc builds all the docs and puts them into the target directory which is (rightfully) ignored in .gitignore

• setup github pages on this repo (or other solution) to serve the api docs.
• create a build script for release (maybe in travis) that deploys the docs build

it's not considered in hashing (because it's in the hash of the Header), so should it be considered in equality?

current errors:

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build0.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build1.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build10.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build11.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build12.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build13.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build14.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build15.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build2.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build3.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build4.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build5.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build6.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build7.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build8.rcgu.o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.build_script_build9.rcgu.o" "-o" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd" "/root/holochain/target/debug/build/num-integer-ee7fbdeb912bb4dd/build_script_build-ee7fbdeb912bb4dd.crate.allocator.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "-L" "/root/holochain/target/debug/deps" "-L" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-68ef171a75dfd5dc.rlib" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-4a380244b10d6a5c.rlib" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-cd20fbe618e0ef78.rlib" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-6b45cc0c1f58bda3.rlib" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_system-72362966f5e95133.rlib" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-7e3cdca8cafd62fc.rlib" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-00d127618b8240f9.rlib" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-6ac02a98cd149424.rlib" "-Wl,--end-group" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-2743c400e5187c22.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util" "-l" "util"
  = note: /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-cd20fbe618e0ef78.rlib(ctl.pic.o): In function `ctl_lookup':
          /checkout/src/liballoc_jemalloc/../jemalloc/src/ctl.c:909: undefined reference to `__rawmemchr'
          /checkout/src/liballoc_jemalloc/../jemalloc/src/ctl.c:832: undefined reference to `__rawmemchr'
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

error: Could not compile `num-integer`.

Flesh out the app init process

Currently the returned serialized parameter from a function call overwrites the input parameters (which is probably ok), but if the return value were longer than one page of memory this would be a problem.

• add malloc so we can increase page size
• use malloc in the serialization of the output params

current get implementation is slow as it iterates through the whole chain with a filter to implement get.

similar to #35 this is simply due to running everything through a single vector and could be improved by leveraging additional data structures internally

once for make test and once for cargo tarpaulin --out Xml in codecov handling

flagged in #51

the pair is cloned for both validation and the final insert, perhaps this can be done more efficiently without the cloning...

e.g. https://doc.rust-lang.org/std/hash/trait.Hash.html

looks like rust natively supports data hashing itself

also this hashing is used by the native hashmap implementation IIRC, which would make lookups quite neat

discussed with @zippy this morning

we currently have three hash references in the header:

• previous header
• entry for this header
• previous header with the same type

and entries do their own hash referencing, e.g. with links

would it make sense to have a single property in the header links or refs that covers all of this?

child of #117

child of #86
Handling error cases where not all of the Zomes have done their genesis() for some reason and we would want to call a second InitApplication Action to complete the Init and do genesis() only on the remaining Zomes.

Checklist

A checklist to keep track of the chain.go basic port to the source_chain module:

• a concept of a "chain" of headers and entries
  • implemented as source_chain::Pair
• chains can be saved/loaded from the file system #231
• headers and entries use the correct hashing algo #20
• headers and entries reference each other by hash
  • implemented as common::entry::Entry and common::entry::Header
• helper fn top to get top header (most recently added)
• helper fn nth to get nth most recently added header
• helper fn TopType to get most recently added header of given type
• ability to provide an entry + key and have it appended with correct headers + signatures
• helper fn get to lookup header by hash: #30
• helper fn getEntry to lookup entry by entry hash: #30
• helper fn getEntryHeader to lookup header by entry hash: #30
  • this should be merged with getEntry if all lookups return pairs
• ability to serialize/deserialize a chain: #61
• ability to "dump" a chain to a string
• ability to walk a chain from most recent to least recent
  • implemented move and ref for loop as chain.iter(), (&chain).into_iter() and chain.into_iter()
• ability to validate a chain: #31
• ability to JSON stringify a chain #61
• ability to graphvis a chain
• helper fn Length to get the number of entries in a chain
• helper fn BundleStarted to get the start index of the active bundle
• ability to start/close bundles

Diff

summary of major differences planned/implemented between the go/rust versions:

• multiple implementations of the source chain trait, so memory/file/db chains can all be independently maintained
• dump/JSON/graphvis functionality will all come under the serialize/deserialize banner
• rust chains:
  • append entries (internally generating headers/pairs)
  • return pairs
  • perform direct lookups by hash
  • prefer composable iteration methods/traits over ad-hoc DSL (e.g.filter vs. TopType 😬 )
  • leverage immutability much more

Ongoing

things that will likely come up again in the future, that are not solved in the go version currently.

• source chains that support concurrency

current memory source chain implementation only supports moving into_iter() not iter() or references so we can't currently iterate by reference - which means lots of inefficient cloning of the source chain to get anything done

current source chain implementation is in memory only, need a way to save/load to disk

setting a volume rather than COPY means we can do edits and watch tests instead of build/run being a one shot process in docker

source chain tests could be much neater with a little destructuring:

        let p1 = test_pair(None, "foo");
        let p2 = test_pair(Some(&p1), "bar");
        let p3 = test_pair(Some(&p2), "baz");

could be more like:

    let (p1, p2, p3) = test_pairs(["foo", "bar", "baz"]);

As Holochain core dev, I want to be able to log from different parts of the application and enable them variably.

• decide on standards for using log crate target variable for different contexts
• implement a complex logger that can handle a complex target for multi-coloring for the use case of hcdev using that standard

currently we insert new pairs at the start of a vector, which is O(n)

other collection types can do O(1) or O(log n) for inserts

https://doc.rust-lang.org/std/collections/#sequences

see chain::chain::SourceChain::push() for an example of what i'm talking about...

See discussion at #69 (comment)

current implementation involves lots of cloning of data, rather than having newer pairs simply reference older pairs and be immutable

https://docs.rs/im/10.2.0/im/ looks like something we might want here

child of #117

As the README says, we are currently targetting Rust 1.26.2, which is very specific. I think we should be able to always target the current stable version of Rust since it is developed in a backwards compatible manner.

In the current implementation of #71 each function call results in two additions to the state via the actions: ExecuteZomeFunction(FunctionCall),
ReturnZomeFunctionResult(FunctionResult),

this means the state object will grow in size for each function call, which means that eventually it has to get cleaned up or garbage collected.

Also there's discussion of the history in the state in PR #37 that would affect this.

currently validation is only based on hashes - pushing invalid hashes onto a source chain is a panicable offense, because hashes should never fail to line up at that point, that's always a bug in core.

however, once validation can be farmed out to zomes, this may no longer be the case so we should return Result instead.

There are some kinds of weirdnesses with clippy in travis. The first time it ran on the a branch it complained with an error like no such subcommand: clippy This was fixed by adding

- cargo +nightly install clippy

Into the install section of the .travis.yml file. But then on next push it would fail the travis run with this error:

cargo +nightly install clippy
    Updating registry `https://github.com/rust-lang/crates.io-index`
  Installing clippy v0.0.209
error: binary `cargo-clippy` already exists in destination as part of `clippy v0.0.209`
binary `clippy-driver` already exists in destination as part of `clippy v0.0.209`

Furthermore clippy was being too conservative and requiring Boxing an Enum that we don't want to box right now, so I didn't want it to fail the tests just because of that.

followup of #71

the golang version had things like addEntry and prepareHeader that sort-of built up pairs (there were no explicit pairs in go)

source_chain::memory::tests has a helper fn test_pair that does something like what we might want here

as a core dev I want access to testing utilities that help me set up things like DNA, holochain instances etc.

• investigate solutions eg (https://stackoverflow.com/questions/41700543/can-we-share-test-utilites-between-crates)
• chose one
• fix use of test_utils crate in hc_core

#117