Now we print the node status in the log as a heartbeat every 10 secs: https://github.com/iotexproject/iotex-core/blob/master/server/itx/heartbeat.go

It's better to add a JSON-RPC API to let ops to query the node status. You're supposed to define an API here: https://github.com/iotexproject/iotex-core/blob/master/explorer/idl/explorer.idl, and implement it. You could refer to heartbeat.go for what metrics we want to expose.

Create an e2e test:

• Start 4-server testnet in one process
• Run roll-DPoS consensus
• Put data in memory
• Inject actions into the testnet
• Verify every nodes get a good blockchain copy

What I did
Build and run container

make docker
docker run -t -i IMAGENAME /bin/bash

Inside docker container i attempt to run iotex-server

root@4fa675318289:/go# iotex-server -log-path=/var/log/iotex/server.log -config=/etc/iotex/config.yaml
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
github.com/iotexproject/iotex-core/common/utils.GetFileAbsPath(0xa2567d, 0x14, 0xc42002c500, 0xc42009c198)
        /go/src/github.com/iotexproject/iotex-core/common/utils/fileutils.go:32 +0x296
github.com/iotexproject/iotex-core/blockchain.NewGenesisBlock(0xc4201a8900, 0x0)
        /go/src/github.com/iotexproject/iotex-core/blockchain/genesis.go:82 +0xe55
github.com/iotexproject/iotex-core/blockchain.createAndInitBlockchain(0xa94660, 0xc4201894c0, 0xa94420, 0xc4200c0300, 0xc4201a8900, 0x9bfc20, 0xc4200c0300)
        /go/src/github.com/iotexproject/iotex-core/blockchain/blockchain.go:512 +0x1cb
github.com/iotexproject/iotex-core/blockchain.CreateBlockchain(0xc4201a8900, 0xa94420, 0xc4200c0300, 0xa94420, 0xc4200c0300)
        /go/src/github.com/iotexproject/iotex-core/blockchain/blockchain.go:405 +0x97
github.com/iotexproject/iotex-core/server/itx.NewServer(0xc4200949e0, 0x9, 0xc420094a10, 0xf, 0x77359400, 0x2540be400, 0x3b9aca00, 0x12a05f200, 0x3b9aca00, 0x0, ...)
        /go/src/github.com/iotexproject/iotex-core/server/itx/itxserver.go:45 +0x158
main.main()
        /go/src/github.com/iotexproject/iotex-core/server/main.go:50 +0x84

After running it manually the second time i get the following error

root@4fa675318289:/go# iotex-server -log-path=/var/log/iotex/server.log -config=/etc/iotex/config.yaml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb0 pc=0x86d67c]

goroutine 1 [running]:
github.com/iotexproject/iotex-core/blockchain.createAndInitBlockchain(0xa94660, 0xc42018b4c0, 0xa94420, 0xc4200c2300, 0xc4201aa900, 0x9bfc20, 0xc4200c2300)
        /go/src/github.com/iotexproject/iotex-core/blockchain/blockchain.go:495 +0x12c
github.com/iotexproject/iotex-core/blockchain.CreateBlockchain(0xc4201aa900, 0xa94420, 0xc4200c2300, 0xa94420, 0xc4200c2300)
        /go/src/github.com/iotexproject/iotex-core/blockchain/blockchain.go:405 +0x97
github.com/iotexproject/iotex-core/server/itx.NewServer(0xc4200949e0, 0x9, 0xc420094a10, 0xf, 0x77359400, 0x2540be400, 0x3b9aca00, 0x12a05f200, 0x3b9aca00, 0x0, ...)
        /go/src/github.com/iotexproject/iotex-core/server/itx/itxserver.go:45 +0x158
main.main()
        /go/src/github.com/iotexproject/iotex-core/server/main.go:50 +0x84

1. What i did

Tried to run

make fmt; make build

2. What i saw

iotex-core/logger/logger.go:10:2: cannot find package "context" in any of:

3. What i expected to see
   Build work properly.

It would appear as though context is imported but not available, it was not installed with other glide installation like in glide.yaml

- package: golang.org/x/net

make test
go fmt ./...
makego test -short ./...
blockdb/blockdb.go:13:2: cannot find package "github.com/golang/glog" in any of:
/home/chensc/go/src/github.com/iotexproject/iotex-core/vendor/github.com/golang/glog (vendor tree)
/home/chensc/go/go/src/github.com/golang/glog (from $GOROOT)
/home/chensc/go/src/github.com/golang/glog (from $GOPATH)
/opt/go/src/github.com/golang/glog
Makefile:70: recipe for target 'test' failed
make: *** [test] Error 1

make run
go build -o ./bin/server -v ./server
export LD_LIBRARY_PATH=D_LIBRARY_PATH:/home/chensc/go/src/github.com/iotexproject/iotex-core/crypto/lib
./bin/server -config=e2etest/config_local_delegate.yaml -log-level=debug
./bin/server: error while loading shared libraries: libsect283k1_ubuntu.so: cannot open shared object file: No such file or directory
Makefile:129: recipe for target 'run' failed
make: *** [run] Error 127

The method is a legacy code, which is only used in the test cases now. We should remove it from the interface.

It's not necessary to be the IP address, but also the host name.

1. what i did
   built the server as per instructions and ran make server
   I tried this both on a dedicated Centos7 build as well as using Docker container

[iotex@localhost ~]$ docker attach 75df525c0e90
root@75df525c0e90:/go/src/github.com/iotexproject/iotex-core# make run
go build -o ./bin/server -v ./server
./bin/server -config=e2etests/config_local_delegate.yaml -log-level=debug
./bin/server: error while loading shared libraries: libsect283k1_ubuntu.so: cannot open shared object file: No such file or directory
Makefile:59: recipe for target 'run' failed
make: *** [run] Error 127

There is also no running process in docker container, by running the following inside of container or in my build directory

root@c8290f10f0c2:/go# iotex-server
iotex-server: error while loading shared libraries: libsect283k1_ubuntu.so: cannot open shared object file: No such file or directory

The anchor is at <project_root>/proto/blockchain.proto

message TransferPb {
    // TransferPb should share these three fields with other Actions
    // TODO: extract these three fields to ActionPb
    uint32 version = 1;
    uint64 nonce = 2;
    bytes signature = 3;

    // used by state-based model
    bytes amount  = 4;
    string sender = 5;
    string recipient = 6;
    bytes payload  = 7;
    bytes senderPubKey = 8;
    bool isCoinbase = 9;
}

message VotePb {
    // VotePb should share these three fields with other Actions
    // TODO: extract these three fields to ActionPb
    uint32 version = 1;
    uint64 nonce = 2;
    bytes signature = 3;

    uint64 timestamp = 4;
    bytes selfPubkey = 5;
    bytes votePubkey = 6;  // the pubkey this node is voting for
}

message ActionPb {
    oneof action {
        TxPb tx = 1; // To be deprecated
        TransferPb transfer = 2;
        VotePb vote = 3;
    }
}

Run protoc --go_out=plugins=grpc:. *.proto command to regenerate the proto code stub

If you plan to support Windows OS in the future, I found that the server builds well, except that the SYS_EXIT constant used in syscall.Exit() is not defined on Windows.

I quickly fixed that with syscall.Exit(60), though I'm not a Go developer therefore a more suitable replacement is probably available.

Transaction is deprecated already. We removed most of the code, but not the protobuf definition as well as Tx struct. We should cleanup the unused code.

Currently, when any system error happens in consensus FSM, we will return an sInvalid state, and the fsm will stay at the current state. This makes the consensus be at the risk of being stuck unexpectedly. A better way is to move fsm back to epoch_start, which consensus could from restart from a fresh state.

Trying to build for ARM (arm version 5), everything is fine except that the build can't be completed due to the following error:

crypto/lib/blslib/libtblsmnt_ubuntu.so: file not recognized: File format not recognized

can you provide those libs for ARM architecture, or provide references about where to find the code to build them?

libtblsmnt_ubuntu.so
libsect283k1_ubuntu.so  

We use boltDB to persist the blockchain data. As the external service, we need to our blockchain node to be tolerant to transit failures by talking to it.

Blockchain has a clock. When minting a new block, it just needs to pass the timestamp generated from the clock into NewBlock func, instead of passing the clock itself into the func

Now when the server starts, it will output a heartbeat log every 10 second. It has included the high-level health metrics from dispatcher, consensus and blockchain:

These are the good CLI function to add:

getBlockActionCount(blockId)
getPendingTransfers()
getPendingVotes()
getTransfersFromBlock(blockId)
getTransfer(transferId)
getVotesFromBlock(blockId)
getVote(voteId)

Feel free to propose other useful APIs

txvm code is useless now, as we have already move to support evm on our blockchain. we could cleanup txvm package as well as other related code.

I m trying to install iotex-core on Ubuntu 16.04, I got an error on make test command:
/tmp/go-build454528752/b001/keystore.test: error while loading shared libraries: libtblsmnt_ubuntu.so: cannot open shared object file: No such file or directory FAIL github.com/iotexproject/iotex-core/accounts/keystore 0.003s /tmp/go-build454528752/b166/subchain.test: error while loading shared libraries: libtblsmnt_ubuntu.so: cannot open shared object file: No such file or directory FAIL github.com/iotexproject/iotex-core/action/subchain 0.005s /tmp/go-build454528752/b171/actpool.test: error while loading shared libraries: libtblsmnt_ubuntu.so: cannot open shared object file: No such file or directory FAIL github.com/iotexproject/iotex-core/actpool 0.002s /tmp/go-build454528752/b177/address.test: error while loading shared libraries: libtblsmnt_ubuntu.so: cannot open shared object file: No such file or directory FAIL github.com/iotexproject/iotex-core/address 0.001s
How can I solve it? Thanks

When I run make test, there is an error:

make test go fmt ./... go test -short ./... blockdb/blockdb.go:13:2: cannot find package "github.com/golang/glog" in any of: /Users/sggb/go/src/github.com/iotexproject/iotex-core/vendor/github.com/golang/glog (vendor tree) /usr/local/Cellar/go/1.10.1/libexec/src/github.com/golang/glog (from $GOROOT) /Users/sggb/go/src/github.com/golang/glog (from $GOPATH) make: *** [test] Error 1

when i forked the project, and run make build, a error below:
make build go build -o ./bin/server -v ./server github.com/fishmeat/iotex-core/vendor/github.com/golang/protobuf/proto github.com/fishmeat/iotex-core/server #github.com/fishmeat/iotex-core/server server/main.go:79:60: cannot use bcb (type func("github.com/fishmeat/iotex-core/vendor/github.com/golang/protobuf/proto".Message) error) as type func("github.com/iotexproject/iotex-core/vendor/github.com/golang/protobuf/proto".Message) error in argument to rpcservice.NewChainServer make: *** [build] Error 2

actioninjector.go test coverage is under 50%. There should be some easy test cases we could add to it.

Please comment on this thread by answering the following questions. Thanks!

What did you do?

What did you expect to see?

What did you see instead?

Search for the "log" import in the codebase

I'm building the master branch on Mac OS 10.14 (go1.11.2).

I can build for Mac without any issue, but when I try to cross compile for my Linux server I get the following:

$ GOOS=linux make build

go build -o ./bin/server -v ./server
vendor/github.com/CoderZhi/go-ethereum/crypto/signature_cgo.go:27:2: build constraints exclude all Go files in /Users/simone/Source/Go/src/github.com/iotexproject/iotex-core/vendor/github.com/CoderZhi/go-ethereum/crypto/secp256k1
make: *** [build] Error 1

There's no sense to have MockExplorer, which could be completely erased. We have used mockgen to generate the mock stub for explorer interface already.

testnet_actions.yaml defines the actions to be included in the genesis block (genesis.go). The pain point of using this is that every time we update the action struct or the address, we have to re-generate the config file again. However, if we keep to only use pub key for src/dst, generate address and signature dynamically. We should be able to get rid of always updating the file.

In actpool.go, actPool.allActions stores the serialized (proto) actions. However, since we put all actions in memory, we don't need to serialize them to proto bytes, and then deserialize it when loading them from actpool. We could use action interface instead, and store the action struct directly

In actpool_test.go, local_test.go, local_actpool_test.go, we have quite similar generator methods, such as (create)SignedTransfer/Vote. It's better to uniform the code.

go version go1.10.3 linux/amd64
Ubuntu 16.04.5 LTS (Windows subsystem)
commit b6bad29 (had same issue with v2.0 zip)

# $ make run
go build -o ./bin/server -v ./server
export LD_LIBRARY_PATH=/home/james/go/src/github.com/iotexproject/iotex-core/crypto/lib/::/home/james/go/src/github.com/iotexproject/iotex-core/crypto/lib
./bin/server -config-path=e2etest/config_local_delegate.yaml -log-colorful=true
panic: invalid page type: 0: 4

goroutine 1 [running]:
github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt.(*Cursor).search(0xc420257738, 0xc4201fba30, 0x7, 0x8, 0x4)
        /home/james/go/src/github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt/cursor.go:256 +0x388
github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt.(*Cursor).seek(0xc420257738, 0xc4201fba30, 0x7, 0x8, 0x0, 0x0, 0xc420257710, 0x4cbce9, 0xab79e0, 0xc420232860, ...)
        /home/james/go/src/github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt/cursor.go:159 +0xa5
github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt.(*Bucket).CreateBucket(0xc420136638, 0xc4201fba30, 0x7, 0x8, 0xc4201fba37, 0x1, 0x0)
        /home/james/go/src/github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt/bucket.go:172 +0xfa
github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt.(*Bucket).CreateBucketIfNotExists(0xc420136638, 0xc4201fba30, 0x7, 0x8, 0x7, 0x8, 0xc4201fba30)
        /home/james/go/src/github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt/bucket.go:206 +0x4d
github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt.(*Tx).CreateBucketIfNotExists(0xc420136620, 0xc4201fba30, 0x7, 0x8, 0x7, 0x8, 0x429ee9)
        /home/james/go/src/github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt/tx.go:115 +0x4f
github.com/iotexproject/iotex-core/db.(*boltDB).PutIfNotExists.func1(0xc420136620, 0xa6d608, 0xc420136620)
        /home/james/go/src/github.com/iotexproject/iotex-core/db/db.go:137 +0x71
github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt.(*DB).Update(0xc4200ba1e0, 0xc420257920, 0x0, 0x0)
        /home/james/go/src/github.com/iotexproject/iotex-core/vendor/github.com/boltdb/bolt/db.go:598 +0x90
github.com/iotexproject/iotex-core/db.(*boltDB).PutIfNotExists(0xc420232100, 0xa3ddec, 0x7, 0xc420230dc0, 0x20, 0x20, 0xc4201963c0, 0x140, 0x140, 0xd3506d5d43f640bb, ...)
        /home/james/go/src/github.com/iotexproject/iotex-core/db/db.go:136 +0xd0
github.com/iotexproject/iotex-core/trie.(*trie).putPatriciaNew(0xc420136380, 0xabb960, 0xc420234a40, 0x274a6c8ca32285bf, 0x115a25d95389566d)
        /home/james/go/src/github.com/iotexproject/iotex-core/trie/trie.go:572 +0x3ec
github.com/iotexproject/iotex-core/trie.(*trie).upsert(0xc420136380, 0xc420230ce5, 0x14, 0x1b, 0xc420139b00, 0xf3, 0xff, 0x0, 0xc4202373b0)
        /home/james/go/src/github.com/iotexproject/iotex-core/trie/trie.go:266 +0x2bb
github.com/iotexproject/iotex-core/trie.(*trie).Upsert(0xc420136380, 0xc420230ce5, 0x14, 0x1b, 0xc420139b00, 0xf3, 0xff, 0x0, 0x0)
        /home/james/go/src/github.com/iotexproject/iotex-core/trie/trie.go:119 +0xb2
github.com/iotexproject/iotex-core/state.(*factory).CreateState(0xc4200e6e70, 0xa588b4, 0x31, 0x2540be400, 0x1, 0x0, 0x996a40)
        /home/james/go/src/github.com/iotexproject/iotex-core/state/factory.go:172 +0x1b1
github.com/iotexproject/iotex-core/blockchain.(*blockchain).initStateFactory(0xc4200ab540, 0x0, 0x0)
        /home/james/go/src/github.com/iotexproject/iotex-core/blockchain/blockchain.go:244 +0x77
github.com/iotexproject/iotex-core/blockchain.NewBlockchain(0xc4200af900, 0xc420257ed0, 0x2, 0x2, 0x0, 0x1f4)
        /home/james/go/src/github.com/iotexproject/iotex-core/blockchain/blockchain.go:199 +0x25e
github.com/iotexproject/iotex-core/server/itx.NewServer(0xc4200af900, 0x8)
        /home/james/go/src/github.com/iotexproject/iotex-core/server/itx/itxserver.go:39 +0x79
main.main()
        /home/james/go/src/github.com/iotexproject/iotex-core/server/main.go:47 +0x87
Makefile:129: recipe for target 'run' failed
make: *** [run] Error 2

Currently, a node will only passively listen to the flying blocks in the P2P network, request the missing blocks between the flying block height and its own top block height. It could happen that there's no flying blocks, but a node is not up-to-date in terms of blockchain. In this case, it should be able to proactively ask neighbors for the top block height they see

ErrNotImplemented is a single var in errcode package, but is only used in consensus. Let's move it there, and delete errcode package.

There're other K/V store choices options:

• https://github.com/dgraph-io/badger
• https://github.com/etcd-io/bbolt
• https://github.com/syndtr/goleveldb

We need to investigate if these are good alternative we should migrate to. Some considerations in my mind:

1. Operation cost
2. Read/write performance
3. Storage footprint

Ideally, there is no way to tell if a hash is a block hash or a tx hash without searching the database, e.g., in ETH's case
Tx:
0xb88f9a1c0e950ac160db166150b2cf3ff064deda9ec4e1b71e7a001fc3f0157d
0x328e629b47bce2f71095ed498abd37d8306d3956f2920ba3d5f53c8966bc178d

Block:
0xc79cad12e5eb9c0530a6084f26dd3b40191996d7105b24ac1fa708d1d40c2cc8
0x3c5a4f77e959758c0146e9cdf750ff402e56a92f33305611481f715b7dcbd66f

So the backend should provide an API named GetBlockOrActionByHash(hash string) that can be used by the frontend to search by a hash string, which makes me think we could normalize the underlying storage for hash=>block and hash=>action in boltdb.

Run following commands and then see the error.

$ dep ensure; make fmt;
$ make build
...
# iotex-core/vendor/github.com/iotexproject/iotex-core/crypto vendor/github.com/iotexproject/iotex-core/crypto/sig.go:9:24: fatal error: lib/ecckey.h: No such file or directory //#include "lib/ecckey.h"
...
Makefile:49: recipe for target 'build' failed make: *** [build] Error 2

Any suggestion on the build? Thanks!

A recent logger change makes logs output as json strings by default no matter to console or to file: d149e03. However, it should still make sense to print logs colorfully in tests. Ideally we should find a clean way to set it once so that it could affect all tests.

Check the codecov report: https://codecov.io/gh/iotexproject/iotex-core/tree/master/blockchain/action

In fact, we haven't any test cases to cover the action instance creation, serializing to/ deserializing from proto or json format.

We have two code path that a block could be appended to the blockchain: 1) consensus module commits a block after reaching consensus; 2) block sync receives a block from peers. The race condition could happen between the two path. However, it should not matter if a block has been put successfully already, and the second put fails because the block exists already. We need to catch the error on both side and check the error type. If the error is that block exists already, we shouldn't log an error

Background: We have a basic implementation of CLI but we have recently added a bunch of new APIs (see explorer.idl for more details). However, CLI only wraps over a very limited amount of them. We need to support all or most our APIs from the command line. And a step back, we need to rethink the framework/layout of CLI.

Requirement: We need a command line tool that interfaces with the blockchain APIs that can:
-- be configured to talk to a proper endpoint
-- get blockchain states
-- issue txs
-- create/update/manage accounts

This could simplify a lot of error handling in our code base, and make our code more condense

The code is at <project_root>/logger/logger.go. We need to prevent people from using Print and Printf. If they want to debug, they should use debug level logging.

What I did

1. Pull out iotex-core
2. Run "docker -f Dockerfile ."

What i saw

go clean
rm -f ./bin/server
rm -f ./bin/txinjector
rm -f chain.db
rm -f block.dat
go build -o ./bin/server -v ./server
db/db.go:14:2: cannot find package "github.com/boltdb/bolt" in any of:
        /usr/local/go/src/github.com/boltdb/bolt (from $GOROOT)
        /go/src/github.com/boltdb/bolt (from $GOPATH)
server/main.go:19:2: cannot find package "github.com/golang/protobuf/proto" in any of:
        /usr/local/go/src/github.com/golang/protobuf/proto (from $GOROOT)
        /go/src/github.com/golang/protobuf/proto (from $GOPATH)
db/db.go:12:2: cannot find package "github.com/pkg/errors" in any of:
        /usr/local/go/src/github.com/pkg/errors (from $GOROOT)
        /go/src/github.com/pkg/errors (from $GOPATH)
logger/logger.go:15:2: cannot find package "github.com/rs/zerolog" in any of:
        /usr/local/go/src/github.com/rs/zerolog (from $GOROOT)
        /go/src/github.com/rs/zerolog (from $GOPATH)
crypto/cryptosort.go:13:2: cannot find package "golang.org/x/crypto/blake2b" in any of:
        /usr/local/go/src/golang.org/x/crypto/blake2b (from $GOROOT)
        /go/src/golang.org/x/crypto/blake2b (from $GOPATH)
proto/rpc.pb.go:11:2: cannot find package "golang.org/x/net/context" in any of:
        /usr/local/go/src/golang.org/x/net/context (from $GOROOT)
        /go/src/golang.org/x/net/context (from $GOPATH)
proto/rpc.pb.go:12:2: cannot find package "google.golang.org/grpc" in any of:
        /usr/local/go/src/google.golang.org/grpc (from $GOROOT)
        /go/src/google.golang.org/grpc (from $GOPATH)
network/utils.go:17:2: cannot find package "google.golang.org/grpc/credentials" in any of:
        /usr/local/go/src/google.golang.org/grpc/credentials (from $GOROOT)
        /go/src/google.golang.org/grpc/credentials (from $GOPATH)
config/config.go:15:2: cannot find package "google.golang.org/grpc/keepalive" in any of:
        /usr/local/go/src/google.golang.org/grpc/keepalive (from $GOROOT)
        /go/src/google.golang.org/grpc/keepalive (from $GOPATH)
network/rpcserver.go:17:2: cannot find package "google.golang.org/grpc/peer" in any of:
        /usr/local/go/src/google.golang.org/grpc/peer (from $GOROOT)
        /go/src/google.golang.org/grpc/peer (from $GOPATH)
rpcservice/rpcservice.go:17:2: cannot find package "google.golang.org/grpc/reflection" in any of:
        /usr/local/go/src/google.golang.org/grpc/reflection (from $GOROOT)
        /go/src/google.golang.org/grpc/reflection (from $GOPATH)
config/config.go:16:2: cannot find package "gopkg.in/yaml.v2" in any of:
        /usr/local/go/src/gopkg.in/yaml.v2 (from $GOROOT)
        /go/src/gopkg.in/yaml.v2 (from $GOPATH)
make: *** [build] Error 1
Makefile:25: recipe for target 'build' failed

What I expected to see
A clean build of the iotex-core

IoTeX blockchain is using roll DPoS consensus, so that it's more accurate call the machine who produces blocks block producer than miner. Let's find the miner in the codebase and replace it with block producer.

Now block#NewBlock takes the clock as the param, and then call clock.Now().Unix() to get the current unixtime to set the block. Instead, we could pass in the unixtime directly into NewBlock