casper-network / casper-node Goto Github PK

View Code? Open in Web Editor NEW

388.0 34.0 214.0 59.66 MB

Reference client for CASPER protocol

Home Page: https://casper.network

License: Apache License 2.0

Rust 91.01% Makefile 0.08% Shell 5.86% TypeScript 2.45% JavaScript 0.17% Python 0.43%

cspr casper casper-network blockchain rust

casper-node's People

Contributors

Stargazers

Watchers

Forkers

fraser999 edhastingscasperassociation fizyk20 marc-casperlabs sacherjj afck goral09 dwerner ashok-cl narimantos arcolife mpapierski darthsiroftardis xcthulhu ilhamm02 hysmagus engrgeorgemarc alive29 andreyvelde smartstake joesixpack sun1534 silentnoname jimmycheung1993 btcnadzor uniqnodes hodlshark huluyisheng razumv waynewayner shivlim bi23com sideeffect117 sharagoov sentrylink honeywest neilyamit soyerso pablovader blockblox-md khjcph 0xmaverick apiprotocol dipmr mikeyhodl mabahma alipostaci2001 spingreekgod mcdee ihor past2017 deuszex aaa379 jimdotzel drayanaindra sbnair sujper leo1286 jesuscoins776 nilz3ro mrinal-manohar killianh vivalldi25 staking7pc mt2721 rafal-ch r2d2shark voogarix doom75 loctrinh lambert41 kukurian r0ter nvlinh90 dartspacephysiker manhsu91 alikarabulut58 ulsgutten eltann nguyenminhhieu12041996 h0psing benjaminmongin ian-d-crypto hunkel haoth thanhtahnhdrfgd medocapital maramelz erictop3 tuantadev adamkluk web3validator pabloortegat jetlag666 criptomauro haithambasim supremeleader00 armyids teemaja asiermp

casper-node's Issues

Bid contract deployment with an error

I've tried to deploy a contract from newly fresh compiled sources but the deployment was completed with an error.
I've tried to use the contract that was assembled when starting Delta-1 - everything works fine

The same error was in delta-2
"error_message": "Interpreter error: host module doesn't export function with name casper_call_contract"

Current [ DELTA 6 ] have peers connected from [ DELTA 5 ]

Current seed node have 7 peers connected, where one of this peer i check he is from DELTA 5 and he have about 30 peers connected from previous network. Possibility of cross chain talk.

Intruder curl -s http://116.203.214.160:8888/status | jq

{
  "api_version": "1.0.0",
  "chainspec_name": "casper-delta-5",
  "genesis_root_hash": "a665..29c6",
  "peers": [
    {
      "node_id": "NodeId::Tls(08ba..cc65)",
      "address": "157.90.21.155:35000"
    },
    {
      "node_id": "NodeId::Tls(08d4..0273)",
      "address": "168.119.250.90:35000"
    },
    {
      "node_id": "NodeId::Tls(1237..8332)",
      "address": "159.69.218.20:35000"
    },
    {
      "node_id": "NodeId::Tls(17d5..2f0f)",
      "address": "168.119.123.213:35000"
    },
    {
      "node_id": "NodeId::Tls(193d..23ab)",
      "address": "157.90.26.198:35000"
    },
    {
      "node_id": "NodeId::Tls(2fc9..7c20)",
      "address": "116.202.24.91:35000"
    },
    {
      "node_id": "NodeId::Tls(35b2..a27e)",
      "address": "78.47.110.240:35000"
    },
    {
      "node_id": "NodeId::Tls(3e9f..87e4)",
      "address": "116.203.182.37:35000"
    },
    {
      "node_id": "NodeId::Tls(4037..0b1e)",
      "address": "88.99.171.225:35000"
    },
    {
      "node_id": "NodeId::Tls(4214..1f70)",
      "address": "135.181.40.95:35000"
    },
    {
      "node_id": "NodeId::Tls(4a9b..2834)",
      "address": "88.99.120.29:35000"
    },
    {
      "node_id": "NodeId::Tls(52f0..b54c)",
      "address": "94.130.224.158:35000"
    },
    {
      "node_id": "NodeId::Tls(5a06..7fe9)",
      "address": "168.119.173.204:35000"
    },
    {
      "node_id": "NodeId::Tls(5e61..d73f)",
      "address": "135.181.42.149:35000"
    },
    {
      "node_id": "NodeId::Tls(64b6..aec8)",
      "address": "116.203.206.169:35000"
    },
    {
      "node_id": "NodeId::Tls(64c9..abb4)",
      "address": "88.99.36.131:35000"
    },
    {
      "node_id": "NodeId::Tls(6758..caac)",
      "address": "49.12.96.112:35000"
    },
    {
      "node_id": "NodeId::Tls(6d85..298b)",
      "address": "78.46.121.69:35000"
    },
    {
      "node_id": "NodeId::Tls(75e5..f7d6)",
      "address": "78.47.9.189:35000"
    },
    {
      "node_id": "NodeId::Tls(874f..9bcb)",
      "address": "95.217.154.113:35000"
    },
    {
      "node_id": "NodeId::Tls(90eb..7a13)",
      "address": "95.216.147.88:35000"
    },
    {
      "node_id": "NodeId::Tls(9435..f7d9)",
      "address": "78.47.18.35:35000"
    },
    {
      "node_id": "NodeId::Tls(98b6..dd10)",
      "address": "78.46.186.163:35000"
    },
    {
      "node_id": "NodeId::Tls(a9e7..4b5e)",
      "address": "168.119.59.12:35000"
    },
    {
      "node_id": "NodeId::Tls(ac63..e930)",
      "address": "168.119.169.105:35000"
    },
    {
      "node_id": "NodeId::Tls(b27e..72d6)",
      "address": "168.119.177.37:35000"
    },
    {
      "node_id": "NodeId::Tls(b328..035e)",
      "address": "168.119.126.225:35000"
    },
    {
      "node_id": "NodeId::Tls(b44b..ad4e)",
      "address": "78.47.131.222:35000"
    },
    {
      "node_id": "NodeId::Tls(dc8d..7985)",
      "address": "116.202.106.191:35000"
    },
    {
      "node_id": "NodeId::Tls(e39e..bab8)",
      "address": "78.46.198.83:35000"
    },
    {
      "node_id": "NodeId::Tls(ea95..f095)",
      "address": "95.217.213.84:35000"
    },
    {
      "node_id": "NodeId::Tls(f887..28cd)",
      "address": "157.90.26.194:35000"
    },
    {
      "node_id": "NodeId::Tls(fbd5..e151)",
      "address": "168.119.59.64:35000"
    },
    {
      "node_id": "NodeId::Tls(fef3..6370)",
      "address": "195.201.148.78:35000"
    }
  ],
  "last_added_block_info": {
    "hash": "6225f5a1130929f4e5f59f96d96c8436c007c67a949b3d1b63672d81287c5bab",
    "timestamp": "2020-12-27T09:03:14.944Z",
    "era_id": 221,
    "height": 4396,
    "state_root_hash": "d735ceeb4e9ec2f31a9f19e63cdab4445e5df85a7f805d69f30f6c2fd9dfd3f2",
    "creator": "01b952f866b554dbb80b420348bffcdc1033b5f97444e47dc8630db2d05da255d6"
  },
  "build_version": "0.3.1"
}

Legit seed curl -s http://13.57.200.251:8888/status | jq

{
  "api_version": "1.0.0",
  "chainspec_name": "casper-delta-6",
  "genesis_root_hash": "9eb4..aef6",
  "peers": [
    {
      "node_id": "NodeId::Tls(1905..f268)",
      "address": "129.146.102.188:35000"
    },
    {
      "node_id": "NodeId::Tls(2455..759c)",
      "address": "129.146.58.40:35000"
    },
    {
      "node_id": "NodeId::Tls(3601..bb91)",
      "address": "161.97.136.153:56588"
    },
    {
      "node_id": "NodeId::Tls(3d16..a5d8)",
      "address": "158.101.3.185:35000"
    },
    {
      "node_id": "NodeId::Tls(59da..01e6)",
      "address": "18.191.239.36:35000"
    },
    {
      "node_id": "NodeId::Tls(7a60..f706)",
      "address": "101.36.118.242:35000"
    },
    {
      "node_id": "NodeId::Tls(f03f..1d84)",
      "address": "18.144.176.168:35000"
    }
  ],
  "last_added_block_info": {
    "hash": "2a48f3e41a2ee7eaf1b2d04eb8b1b3afa90091bc597872c858445fe328191b74",
    "timestamp": "2020-12-30T12:12:07.552Z",
    "era_id": 18,
    "height": 308,
    "state_root_hash": "ffd1515f875a1e27a19db90176ce01cfb6b7368ce5c2ac0be7838755abd4e596",
    "creator": "013f774a58f4d40bd9b6cce7e306e53646913860ef2a111d00f0fe7794010c4012"
  },
  "build_version": "0.4.0"
}

Update 3

Current peers situation

 STATUS     IP ADDRESS       TRUSTED HASH           CHAIN NAME       ERA   HEIGH     VERSION
--------------------------------------------------------------------------------------------
 Trusted    18.144.176.168   c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Trusted    13.57.200.251    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
--------------------------------------------------------------------------------------------
 HOST       127.0.0.1        null                   casper-delta-6  null   null      0.4.0
 Useless    49.12.96.112     6225f .... 7c5bab      casper-delta-5   221   4396      0.3.1
 Available  13.57.200.251    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Available  129.146.102.188  c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Useless    95.216.147.88    6225f .... 7c5bab      casper-delta-5   221   4396      0.3.1
 Available  129.146.58.40    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Available  158.101.3.185    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Available  18.191.239.36    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Useless    135.181.147.7    6225f .... 7c5bab      casper-delta-5   221   4396      0.3.1
 Available  101.36.118.242   c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Useless    168.119.177.37   6225f .... 7c5bab      casper-delta-5   221   4396      0.3.1
 Available  18.144.176.168   c13a2 .... baa532      casper-delta-6    19    318      0.4.0

Trusted hash: c13a20dc256787c5a64c8dc63636638d0548815f8b728a5d1fa226515abaa532

PeersCount:     12

Useless peers:  4
Blocked access: 0
Good condition: 8

Update 4

Ok, so this above happen on the time of the sync, after sync complete situation change and ancient peers been disconnected from the node accordingly:

STATUS     IP ADDRESS       TRUSTED HASH           CHAIN NAME       ERA   HEIGH     VERSION
--------------------------------------------------------------------------------------------
Trusted    18.144.176.168   c13a2 .... baa532      casper-delta-6    19    318      0.4.0
Trusted    13.57.200.251    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
--------------------------------------------------------------------------------------------
HOST       127.0.0.1        c13a2 .... baa532      casper-delta-6    19    318      0.4.0
Available  13.57.200.251    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
Blocked    88.99.120.29                                                                  
Available  18.144.176.168   c13a2 .... baa532      casper-delta-6    19    318      0.4.0

Trusted hash: c13a20dc256787c5a64c8dc63636638d0548815f8b728a5d1fa226515abaa532

PeersCount:     3

Useless peers:  0
Blocked access: 1
Good condition: 2

Main process exited, code=killed, status=9/KILL

Happen Dec 26 02:52:25 UTC

I noticed Scheduler Queue overload, peak happen 1:00 UTC one hour before node crash:

DELTA BALANCER: scheduler_queue_api_count: 0
DELTA BALANCER: scheduler_queue_network_count: 47.3333
DELTA BALANCER: scheduler_queue_network_incoming_count: 4947216.0417
DELTA BALANCER: scheduler_queue_regular_count: 23.125
DELTA BALANCER: scheduler_queue_total_count: 4947286.5

Update 1

Trying to recover and restart node so far unsuccessfully:

Main process exited, code=killed, status=6/ABRT

Node can't sync above "height": 4215,

Weird behavior on genesis generation. After "Consensus(Shutdown)" node shutdown.

On the start of the Delta - 3 exact on the moment of genesis generation, my validator lost peers.

Status replay:

{
  "api_version": "1.0.0",
  "chainspec_name": "casper-delta-3",
  "genesis_root_hash": "e81d..b4c0",
  "peers": {},
  "last_added_block_info": null,
  "build_version": "1.6.0"
}

Expected:

{
    "api_version": "1.0.0",
    "chainspec_name": "casper-delta-3",
    "genesis_root_hash": "e81d..b4c0",
    "peers": {
        "45cf..1364": "13.57.242.68:35000",
        "4e5d..4e53": "194.19.235.78:35000",
        "716d..9a06": "167.99.74.47:35000",
        "db09..0b9b": "54.183.27.75:35000",
        "e52c..55ac": "207.180.221.204:35000"
    },
    "last_added_block_info": null,
    "build_version": "1.6.0"
}

Error

24: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
      <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
      std::sys::unix:🧵:Thread:🆕:thread_start
             at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/sys/unix/thread.rs:87:17
  25: start_thread
  26: clone

Error

"DEBUG","fields":{"event":"consensus: Shutdown if current era is inactive","q":"Regular"},"target":"casper_node::reactor","span":{"ev":8053,"name":"dispatch events"},"spans":[{"ev":8053,"name":"crank"},{"ev":8053,"name":"dispatch events"}]}
{"timestamp":"Nov 20 22:05:00.003","level":"TRACE","fields":{"event":"Consensus(Shutdown)","q":"Regular"},"target":"casper_node::reactor","span":{"ev":8053,"name":"dispatch events"},"spans":[{"ev":8053,"name":"crank"},{"ev":8053,"name":"dispatch events"}]}

P.S. Peers was present before genesis time.

People can't sync [ DELTA 8 ]

Many guys experience sync issues. I observe network and found Scheduler Queue Total Count constantly increasing:

See spike on gossiper here:

Which correlated with peers been joining my node and then disconnected in same frequence here:

Where is also a glitch in metrics which can indecate restart of my validator unit or can be related to this issue we have presiously, where metrics with all endpoints can't be served properly and fall off periodically.

Update:

This error recently popups in my log:

{
  "timestamp": "Jan 10 05:15:47.654",
  "level": "DEBUG",
  "fields": {
    "message": "connection error: connection closed before message completed"
  },
  "target": "hyper::server::conn::spawn_all"
}

error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`

When running:
$ cargo build -p casper-node --release

I'm getting:

Compiling casper-node v0.9.0
error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
--> node/src/utils/rlimit.rs:23:32
|
23 |     const LIBC_RESOURCE: libc::__rlimit_resource_t;
|                                ^^^^^^^^^^^^^^^^^^^ not found in `libc`

error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
--> node/src/utils/rlimit.rs:32:32
|
32 |     const LIBC_RESOURCE: libc::__rlimit_resource_t  = 7;
|                                ^^^^^^^^^^^^^^^^^^^ not found in `libc`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0412`.
error: could not compile `casper-node`

Key issue while joining delta testnet

While attempting to join the delta testnet, I provided public hex key 01329cc15cb92fca5d5ce6d7fcc881010161547f452e2caad4dff565708b00b8c7, which was successfully funded. Upon launching the node and attempting to bond, I received some warnings of this form:

Nov  5 14:51:39 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:51:39.169","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"2725..4368","peer_address":"135.181.73.218:34858"},"target":"casper_node::components::small_network"}
Nov  5 14:51:39 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:51:39.169","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"2725..4368","peer_address":"135.181.73.218:34553"},"target":"casper_node::components::small_network"}
Nov  5 14:53:49 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:53:49.348","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"725c..21c8","peer_address":"135.181.73.218:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 14:57:36 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:57:36.031","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"99c4..8c92","peer_address":"135.181.73.218:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:01:36 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:01:36.250","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"4b5b..adef","peer_address":"135.181.73.218:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:06:30 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:06:30.501","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"4712..58c3"},"target":"casper_node::components::small_network"}
Nov  5 15:06:30 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:06:30.501","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"4712..58c3","peer_address":"135.181.73.218:43550"},"target":"casper_node::components::small_network"}
Nov  5 15:06:30 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:06:30.501","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"4712..58c3","peer_address":"135.181.73.218:34553"},"target":"casper_node::components::small_network"}
Nov  5 15:11:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:11:32.363","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"6e1a..8a85"},"target":"casper_node::components::small_network"}
Nov  5 15:11:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:11:32.363","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"6e1a..8a85","peer_address":"164.68.114.22:40472"},"target":"casper_node::components::small_network"}
Nov  5 15:11:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:11:32.364","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"6e1a..8a85","peer_address":"164.68.114.22:34553"},"target":"casper_node::components::small_network"}
Nov  5 15:13:42 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:13:42.144","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"b56a..6ef1","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:15:42 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:15:42.292","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"f9ce..270e","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:16:15 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:16:15.752","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"3937..faf0"},"target":"casper_node::components::small_network"}
Nov  5 15:16:15 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:16:15.753","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"3937..faf0","peer_address":"135.181.2.250:41402"},"target":"casper_node::components::small_network"}
Nov  5 15:16:15 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:16:15.753","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"3937..faf0","peer_address":"135.181.2.250:34553"},"target":"casper_node::components::small_network"}
Nov  5 15:17:48 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:17:48.079","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"e8f1..99eb","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:19:48 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:19:48.185","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"3024..ecc5","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:21:48 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:21:48.511","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"7ddf..85d6","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:23:49 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:23:49.251","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"fe6a..d28b","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:35:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:35:32.681","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"1a63..5169"},"target":"casper_node::components::small_network"}
Nov  5 15:35:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:35:32.681","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"1a63..5169","peer_address":"135.181.2.250:37742"},"target":"casper_node::components::small_network"}
Nov  5 15:35:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:35:32.681","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"1a63..5169","peer_address":"135.181.2.250:34553"},"target":"casper_node::components::small_network"}
Nov  5 15:42:28 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:42:28.365","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"8cd6..a450"},"target":"casper_node::components::small_network"}
Nov  5 15:42:28 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:42:28.365","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"8cd6..a450","peer_address":"164.68.114.22:41120"},"target":"casper_node::components::small_network"}
Nov  5 15:42:28 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:42:28.365","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"8cd6..a450","peer_address":"164.68.114.22:34553"},"target":"casper_node::components::small_network"}

Which soon turned into a ton of errors of this form:

Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.753","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"63dc..34cb","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"f1f7..7af1","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"6ab1..2807","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"d353..5ddf","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"0269..7cfd","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"da45..ead8","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"ff54..4652","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.755","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"1be7..3688","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.764","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"cae0..79ee","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.775","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"fc9b..46a9","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}

Which ended in a panic of this form:

Nov  5 18:06:21 casper-node casper-node[24645]:    0: casper_node::panic_hook
Nov  5 18:06:21 casper-node casper-node[24645]:    1: core::ops::function::Fn::call
Nov  5 18:06:21 casper-node casper-node[24645]:    2: std::panicking::rust_panic_with_hook
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/panicking.rs:530:17
Nov  5 18:06:21 casper-node casper-node[24645]:    3: rust_begin_unwind
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/panicking.rs:437:5
Nov  5 18:06:21 casper-node casper-node[24645]:    4: std::panicking::begin_panic_fmt
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/panicking.rs:391:5
Nov  5 18:06:21 casper-node casper-node[24645]:    5: casper_node::effect::EffectBuilder<REv>::fatal::{{closure}}
Nov  5 18:06:21 casper-node casper-node[24645]:    6: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
Nov  5 18:06:21 casper-node casper-node[24645]:    7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Nov  5 18:06:21 casper-node casper-node[24645]:    8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Nov  5 18:06:21 casper-node casper-node[24645]:    9: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
Nov  5 18:06:21 casper-node casper-node[24645]:   10: tokio::runtime::task::raw::poll
Nov  5 18:06:21 casper-node casper-node[24645]:   11: std::thread::local::LocalKey<T>::with
Nov  5 18:06:21 casper-node casper-node[24645]:   12: tokio::runtime::thread_pool::worker::Context::run_task
Nov  5 18:06:21 casper-node casper-node[24645]:   13: tokio::runtime::thread_pool::worker::Context::run
Nov  5 18:06:21 casper-node casper-node[24645]:   14: tokio::macros::scoped_tls::ScopedKey<T>::set
Nov  5 18:06:21 casper-node casper-node[24645]:   15: tokio::runtime::thread_pool::worker::run
Nov  5 18:06:21 casper-node casper-node[24645]:   16: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
Nov  5 18:06:21 casper-node casper-node[24645]:   17: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
Nov  5 18:06:21 casper-node casper-node[24645]:   18: tokio::runtime::task::harness::Harness<T,S>::poll
Nov  5 18:06:21 casper-node casper-node[24645]:   19: tokio::runtime::blocking::pool::Inner::run
Nov  5 18:06:21 casper-node casper-node[24645]:   20: tokio::runtime::context::enter
Nov  5 18:06:21 casper-node casper-node[24645]:   21: tokio::runtime::handle::Handle::enter
Nov  5 18:06:21 casper-node casper-node[24645]:   22: std::sys_common::backtrace::__rust_begin_short_backtrace
Nov  5 18:06:21 casper-node casper-node[24645]:   23: core::ops::function::FnOnce::call_once{{vtable.shim}}
Nov  5 18:06:21 casper-node casper-node[24645]:   24: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
Nov  5 18:06:21 casper-node casper-node[24645]:       <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
Nov  5 18:06:21 casper-node casper-node[24645]:       std::sys::unix::thread::Thread::new::thread_start
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/sys/unix/thread.rs:87:17
Nov  5 18:06:21 casper-node casper-node[24645]:   25: start_thread
Nov  5 18:06:21 casper-node casper-node[24645]:   26: clone
Nov  5 18:06:21 casper-node casper-node[24645]: panicked at 'fatal error [node/src/components/consensus/era_supervisor.rs:724]: Consensus shutting down due to inability to participate in the network', node/src/effect.rs:366:9

Block validator doesn't check deploy dependencies.

License matters, clarification on usage, and GPL violations

I've been following CasperLabs' work since I heard a talk about it in DevCon V. I really like the CBC protocol and its designs. However, on license matters, I think there's something I must bring up, as I couldn't find discussions on this elsewhere. There are things that have to be fixed (GPL violations), and things that many developers and me would really appreciate clarifications, so that we can properly utilize the codebase, and avoid any "misuse" defined in your license terms.

License matters

If you knew MongoDB, you know that "open source" is a somewhat sensitive word -- as people expect codebases under an "open source" license would comply with the open source definition. OSI reviews provided some reasons why they think the current license is not appropriately "open source". While that's only according to their OSI's definitions, I couldn't find any responses from CasperLabs on how those critics are addressed, and what's CasperLabs' definition of open source. Without this, the word "open source" in the license title "CapserLabs Open Source License" is going to be a source of trouble when the software is made aware by the wider open source community.

The license also contains some reference to "FLOSS", which is supposed to mean free/libre and open source software. The current license also heavily deviates from what the community commonly understand it is.

A recommendation from mine would be if the term "open source" can be removed from the license title and preamble. Why not simply call it "free and open"? No one will question that whether the license is "free and open" at all, but many will question whether the license is "open source" or "FLOSS".

Clarification on usage

I also hope if there can be a more concrete summary page on what is allowed and what is not allowed under the current license terms, as after reading the whole license, I still find it quiet confusing. Not only does this benefit developers like me, but also ecosystems and enterprises heavily.

GPL violations

The license this codebase use (COSLv1) is an open-codebase license. As it's not compatible with GPL, no code in this codebase shall depend on it. However, currently there's one violation of it:

colour, licensed under GPL-3.0.

That dependency has to be removed from the current codebase to avoid the violation. Given it's only a simple dependency, I think this would be relatively easy to do.

Separate transfers and other deploys in the block proposer.

Split up the pending deploys in the block proposer, to separate transfers and other deploys from the start.

Possibly remove DeployType, and split up AppendableBlock::add into two methods, that verify that it's a transfer or not, as expected.

https://casperlabs.atlassian.net/browse/NDRS-1131

error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`

First, thanks for working on this product!

When following the build instructions I get

Compiling warp-json-rpc v0.2.0
error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
  --> node/src/utils/rlimit.rs:23:32
   |
23 |     const LIBC_RESOURCE: libc::__rlimit_resource_t;
   |                                ^^^^^^^^^^^^^^^^^^^ not found in `libc`

error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
  --> node/src/utils/rlimit.rs:31:32
   |
31 |     const LIBC_RESOURCE: libc::__rlimit_resource_t = libc::RLIMIT_NOFILE;
   |                                ^^^^^^^^^^^^^^^^^^^ not found in `libc`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0412`.
error: could not compile `casper-node`

Log files invalid format

Periodically jq refuse to parse with the following error

parse error: Invalid numeric literal at line ...

This is the case of the issue:

Another problem I seen is this, often make grep thinks is a binary file

This above artifacts possible coming from incorrect log rotation.

" Failed sending Request" error when executing transfer subcommand

When i try to run :

cargo run --release -- transfer \
    --node-address=http://localhost:50101 \
    --secret-key=../utils/nctl/assets/net-1/nodes/node-3/keys/secret_key.pem \
    --amount=1234567 \
    --target-account=$PUBLIC_KEY \
    --chain-name=casper-net-1 \
    --payment-amount=10000

I got this error:

Failed sending Request(Request { jsonrpc: "2.0", method: "account_put_deploy", params: Some(Map({"deploy": Object({"approvals": Array([Object({"signature": String("01701aa67c643dd16098e830151b5e6555acb83bb58a1fa75324df9d99f2c8ed646b93d410f6ae4a561bcc94d95e6b4b36b9adcc8a7df2b4e0fa9d43ca1137b50d"), "signer": String("016222cc3d4dc35c387d3d2deff214674be8f750a921b97667efadf6b0c6826226")})]), "hash": String("65e0e5a356038e088dca524734fd61aba7dc92229d88fdd8fc6b9753e015e7fc"), "header": Object({"account": String("016222cc3d4dc35c387d3d2deff214674be8f750a921b97667efadf6b0c6826226"), "body_hash": String("889297a958b5385f092f1918bd3b2f04507286076ad9124148505339703d6a21"), "chain_name": String("casper-net-1"), "dependencies": Array([]), "gas_price": Number(1), "timestamp": String("2021-05-06T20:08:21.389Z"), "ttl": String("1h")}), "payment": Object({"ModuleBytes": Object({"args": Array([Array([String("amount"), Object({"bytes": String("021027"), "cl_type": String("U512"), "parsed": String("10000")})])]), "module_bytes": String("")})}), "session": Object({"Transfer": Object({"args": Array([Array([String("amount"), Object({"bytes": String("0387d612"), "cl_type": String("U512"), "parsed": String("1234567")})]), Array([String("target"), Object({"bytes": String("fa522ee1af717bcd71259d336666017000eb3f659346806f4cdcdab848724bd4"), "cl_type": Object({"ByteArray": Number(32)}), "parsed": String("fa522ee1af717bcd71259d336666017000eb3f659346806f4cdcdab848724bd4")})]), Array([String("id"), Object({"bytes": String("00"), "cl_type": Object({"Option": String("U64")}), "parsed": Null})])])})})})})), id: Num(-799331664064629152) })

My local network is running and the output of nctl-status is the flowing:

validators-1:casper-net-1-node-1    RUNNING   pid 6266, uptime 5:24:14
validators-1:casper-net-1-node-2    RUNNING   pid 6267, uptime 5:24:14
validators-1:casper-net-1-node-3    RUNNING   pid 6268, uptime 5:24:14
validators-2:casper-net-1-node-4    RUNNING   pid 6309, uptime 5:24:12
validators-2:casper-net-1-node-5    RUNNING   pid 6310, uptime 5:24:12
validators-3:casper-net-1-node-10   STOPPED   Not started
validators-3:casper-net-1-node-6    STOPPED   Not started
validators-3:casper-net-1-node-7    STOPPED   Not started
validators-3:casper-net-1-node-8    STOPPED   Not started
validators-3:casper-net-1-node-9    STOPPED   Not started

Any idea how to fix this issue please ?

Dictionary

This task is about implementing support for improved local keys, which intends to resolve past problems that lead to their removal. Contract developers will use this to store data in a separate keyspace prefixed by the URef address provisioned to a user by the system.

CEP for reference: casper-network/ceps#39

Docker-composer and Github Docker Images

Hello,

Is it possible to create a docker-compose yml?

The releases are also in docker Hub, would it be possible to publish them in Github Docker Images?

docs: https://docs.github.com/es/actions/guides/publishing-docker-images

Thanks

Redesign Synchronizer (Part 2 of n)

after making node as validator it works for couple seconds then crush

node version 1_0_0
Suppose that stop state issued by:
"node panicked: Bootstrap failed for node, but it was not in the set of pending connections
Error: "/var/lib/casper/bin/1_0_0/casper-node" "validator" "/etc/casper/1_0_0/config.toml" exited with error"
Using white IP and VDS. ufw disabled. Ubuntu 20.04

casper-node.stderr.lognode

casper-node.log

Block provided by BlockAdded event does not match block specification

The block specification in the JSON-RPC API shows the era report containing lists of rewards and validator weights. This shows up as expected if a block is fetched, e.g.:

          "era_report": {
            "equivocators": [],
            "inactive_validators": [],
            "rewards": [
              {
                "amount": 1202458594726,
                "validator": "01026ca707c348ed8012ac6a1f28db031fadd6eb67203501a353b867a08c8b9a80"
              },
              {
                "amount": 19956734910,
                "validator": "010427c1d1227c9d2aafe8c06c6e6b276da8dcd8fd170ca848b8e3e8e1038a6dc8"
              },

where rewards is an array of reward objects. However, if a block is received on the events stream with the BlockAdded event the rewards and next_era_validator_weights are returned as maps with the key being the validator public key.

This discrepancy makes life difficult to use a standard process for parsing blocks, and appears to serve no purpose. Could the block presented with the BlockAdded event use the same format at that returned by the JSON-API?

Add test for crashed-node reconnection regression

#1413 fixed a regression that caused a node not to reconnect to a crashed node. The conditions are somewhat tough to replicate outside of our integration, but it could be possible to add these to the networking tests using testing::Network. We should definitely try to replicate this bug (by temporarily turning off the fix) and add a test for it.

Avoid restarting event stream event ID from 0 on each restart

Currently, the event stream indexes every event it produces. The numbering starts at 0 and increases by 1 for every subsequent event. However, if the node stops and restarts (which it would do e.g. during an upgrade) then the numbering starts from 0 again.

We want to persist the indexing and on the restart, continue from where we left off.

fatal error via control announcement

Found in the nightly nctl tests: Happens consistently in itst01.

Drone: https://drone-auto-casper-network.casperlabs.io/casper-network/casper-node/202/7/2


cladmin@admin:~$ tail stdout.log
{"timestamp":"May 09 18:05:13.599","level":"DEBUG","fields":{"event":"small network: request: gossip: AddressGossiper::(gossip(gossiped-address 127.0.0.1:34557))","q":"Network"},"target":"casper_node::reactor","span":{"ev":157,"name":"dispatch events"},"spans":[{"ev":157,"name":"crank"},{"ev":157,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:13.599","level":"DEBUG","fields":{"event":"address gossiper: gossiped gossiped-address 127.0.0.1:34557 to ","q":"Regular"},"target":"casper_node::reactor","span":{"ev":158,"name":"dispatch events"},"spans":[{"ev":158,"name":"crank"},{"ev":158,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:13.599","level":"DEBUG","fields":{"message":"paused gossiping since no more peers to gossip to","item":"gossiped-address 127.0.0.1:34557"},"target":"casper_node::components::gossiper","span":{"ev":158,"name":"dispatch events"},"spans":[{"ev":158,"name":"crank"},{"ev":158,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"DEBUG","fields":{"event":"linear chain: Initialize timeout","q":"Regular"},"target":"casper_node::reactor","span":{"ev":159,"name":"dispatch events"},"spans":[{"ev":159,"name":"crank"},{"ev":159,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"INFO","fields":{"message":"hasn't downloaded any blocks in expected time window. Shutting down…"},"target":"casper_node::components::linear_chain_sync","span":{"ev":159,"name":"dispatch events"},"spans":[{"ev":159,"name":"crank"},{"ev":159,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"DEBUG","fields":{"event":"control: fatal error [node/src/components/linear_chain_sync.rs:824]: no syncing progress, shutting down…","q":"Control"},"target":"casper_node::reactor","span":{"ev":160,"name":"dispatch events"},"spans":[{"ev":160,"name":"crank"},{"ev":160,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"ERROR","fields":{"message":"fatal error via control announcement","file":"node/src/components/linear_chain_sync.rs","line":"824","msg":"no syncing progress, shutting down…"},"target":"casper_node::reactor","span":{"ev":160,"name":"dispatch events"},"spans":[{"ev":160,"name":"crank"},{"ev":160,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"INFO","fields":{"message":"shutting down socket, no longer accepting incoming connections","our_id":"NodeId::Tls(0254..87d6)"},"target":"casper_node::components::small_network"}
{"timestamp":"May 09 18:05:18.589","level":"INFO","fields":{"message":"exiting casper-node","exit_code":"101"},"target":"casper_node"}
{"timestamp":"May 09 18:05:18.591","level":"WARN","fields":{"message":"failed running \"/home/cladmin/cl/git/casper-node/utils/nctl/assets/net-1/nodes/node-5/bin/1_0_0/casper-node\" \"validator\" \"/home/cladmin/cl/git/casper-node/utils/nctl/assets/net-1/nodes/node-5/config/1_0_0/config.toml\"","exit_status":"exit code: 101"},"target":"casper_node_launcher::utils"}

node panicked: Failed to download linear chain.

After recent crash and no success with join it again I try to run validator from scratch.

Service exit error:

Main process exited, code=killed, status=6/ABRT

Valdator log is overloaded and produced about 4G since last 30 minutes, mostly:

message received","msg":"payload: AddressGossiper::(gossip-response
message received","msg":"payload: AddressGossiper::(gossip(gossiped-address

Warnings present in a log:

"WARN","fields":{"message":"Finality signatures not handled in joiner reactor"},"target":"casper_node::reactor::joiner","span":{"ev":7206551,"name":"dispatch events"},"spans":[{"ev":7206551,"name":"crank"},{"ev":7206551,"name":"dispatch events"}]}

"WARN","fields":{"message":"network announcement ignored.","other":"Consensus(Protocol { era_id.0: 214, .. })"},"target":"casper_node::reactor::joiner","span":{"ev":7239826,"name":"dispatch events"},"spans":[{"ev":7239826,"name":"crank"},{"ev":7239826,"name":"dispatch events"}]}

Most of the warning spam is:

"WARN","fields":{"message":"NodeId::Tls(7e9d..fa72): outgoing connection failed","peer_address":

Just two warnings about:

{"timestamp":"Dec 26 14:23:37.920","level":"WARN","fields":{"message":"large event size, consider reducing it or boxing","event_size":"160"},"target":"casper_node::reactor"}

{"timestamp":"Dec 26 14:23:38.245","level":"WARN","fields":{"message":"large event size, consider reducing it or boxing","event_size":"544"},"target":"casper_node::reactor"}

Full error message output:

   0: casper_node::panic_hook
   1: core::ops::function::Fn::call
   2: std::panicking::rust_panic_with_hook
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:597:17
   3: std::panicking::begin_panic::{{closure}}
   4: std::sys_common::backtrace::__rust_end_short_backtrace
   5: std::panicking::begin_panic
   6: <casper_node::components::linear_chain_sync::LinearChainSync<I> as casper_node::components::Component<REv>>::handle_event
   7: <casper_node::reactor::joiner::Reactor as casper_node::reactor::Reactor>::dispatch_event
   8: casper_node::reactor::Runner<R>::crank::{{closure}}
   9: casper_node::cli::Cli::run::{{closure}}
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: std::thread::local::LocalKey<T>::with
  12: tokio::runtime::enter::Enter::block_on
  13: tokio::runtime::thread_pool::ThreadPool::block_on
  14: tokio::runtime::context::enter
  15: tokio::runtime::handle::Handle::enter
  16: casper_node::main
  17: std::sys_common::backtrace::__rust_begin_short_backtrace
  18: std::rt::lang_start::{{closure}}
  19: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:381:40
      std::panicking::try
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:345:19
      std::panic::catch_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panic.rs:396:14
      std::rt::lang_start_internal
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/rt.rs:51:25
  20: main
  21: __libc_start_main
  22: _start

node panicked: Failed to download linear chain.

hasn't downloaded any blocks in expected time window

Validator crash after some transactions been processed trough.

Been sending some small spam trough my node, was a sequence 1000 transaction.
Observing event on my monitor it was approximatly 870 pass only.

Crash message:

   0: casper_node::panic_hook
   1: core::ops::function::Fn::call
   2: std::panicking::rust_panic_with_hook
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:597:17
   3: std::panicking::begin_panic_handler::{{closure}}
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:499:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/sys_common/backtrace.rs:141:18
   5: rust_begin_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:495:5
   6: std::panicking::begin_panic_fmt
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:437:5
   7: <casper_node::components::gossiper::Gossiper<T,REv> as casper_node::components::Component<REv>>::handle_event
   8: <casper_node::reactor::validator::Reactor as casper_node::reactor::Reactor>::dispatch_event
   9: casper_node::cli::Cli::run::{{closure}}
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: std::thread::local::LocalKey<T>::with
  12: tokio::runtime::enter::Enter::block_on
  13: tokio::runtime::thread_pool::ThreadPool::block_on
  14: tokio::runtime::context::enter
  15: casper_node::main
  16: std::sys_common::backtrace::__rust_begin_short_backtrace
  17: std::rt::lang_start::{{closure}}
  18: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:381:40
      std::panicking::try
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:345:19
      std::panic::catch_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panic.rs:396:14
      std::rt::lang_start_internal
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/rt.rs:51:25
  19: main
  20: __libc_start_main
  21: _start

panicked at 'internal error: entered unreachable code: can't have gossiped if we don't hold the complete data', node/src/components/gossiper.rs:269:17

Latest log entry content

{"timestamp":"Jan 04 12:46:15.800","level":"DEBUG","fields":{"event":"deploy gossiper: check gossip timeout for deploy-hash(d008..134f) with NodeId::Tls(bad4..0f7d)","q":"Regular"},"target":"casper_node::reactor","span":{"ev":37307377,"name":"dispatch events"},"spans":[{"ev":37307377,"name":"crank"},{"ev":37307377,"name":"dispatch events"}]}

Log attached

tx-crash.log

Update

Join back was very easy, no issue.

Update

Network dead, we not be able to produce any blocks, nodes crashing with:

panicked at 'fatal error [node/src/components/consensus/era_supervisor.rs:811]: Consensus shutting down due to inability to participate in the network', node/src/effect.rs:423:9

v1.0.0 rpc port 7777 issue

Hi leader：

without 7777 port。what's the problem？ please help me！！！

[error] cargo build -p casper-node --release

I get the following error when trying to build casper node with:

cargo build -p casper-node --release

Block Proposer Must Check Deploy Hashes Back to the Last TTL

When the block proposer initializes, it currently gets all of the deploy hashes going back to genesis. It must only get deploy hashes as far back as the TTL.

Crash during catchup with trusted hash

message.txt

See the attached file for log from the node.
Crash can be reproduced by installing everything on a clean box and syncing the node with trusted hashes.

Tested: Around 20 Resyncs
Crashes: 6
Tested with different trusted hashes - Crash is not always happening (So hash is valid)
One of the hashes tested with: 3b0e4ae85f096b0be2e28402c3f32c471d0f5f4a470d2709c731e0febf3d26ec
Environment: Ubuntu 18.04 (Docker - can provide build scripts)

rozśmieszyła mnie elewacja na początku art

https://ebvalaim.pl/2020/05/19/mierzenie-pozycji-iss/

Query status takes very long time.

After network made above 100 validators query status end point takes ages to respond.

Reproducing error:

curl -s http://127.0.0.1:8888/status

Expected:

Fast replay

Update 1

After first very long wait queries back to normal, fast replay.

Update 2

This not only happen with /status query, but also present on Prometheus /metrics

/metrics:

/status

Update 3

People reporting issues with deploys, takes a while to be able to succeed, expected errors:

FailedToGetResponse(reqwest::Error { kind: Request, url: Url { scheme: "http", host: Some(Ipv4(127.0.0.1)), port: Some(7777), path: "/rpc", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) })', client/src/deploy/put.rs:56:31
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

Peers cross talk possibility for attack vector

Recently peers jump from 50 to 118 for exact 3 minutes. ( speaking dry-run )

Start of event: 2021-03-28 07:03:00 UTC
End of event: 2021-03-28 07:06:00 UTC

This seen in a log:

"message":"dropping connection due to network name mismatch","our_id":"NodeId::Tls(8735..e0b2)","peer_id":"NodeId::Tls(bf5c..c2d2)","our_network":"\"casper-dryrun\"","their_network":"\"delta-11\""

We aware, some validators run dry-run release on the same IP which been used in Delta's and they are still listed as trusted IP in Delta-11 config.toml. Possible this nodes act as relay. We can remove them from trusted list, but this issue still introcude possibility of exploit.

Checksum for target account hash

Implement CEP-57.

There is already a draft PR #1557

Needs the following touch-ups before it is merged:

Tests fixed since @afck and I changed the scheme to use our hash entropy efficiently
Rebase to dev
Optimizations
- Replace blake2bhash which returns a Vec with something that returns [u8; 32] so we avoid using the heap
- Reduce the number of passes over the hex string in decode to at most 2
- Make it so decode doesn’t do any heap allocations
- Simplify for loops to use iteration instead
Remove all hex external crate dependencies
Benchmarks: run cargo bench --bench transfer_bench -- --save-baseline 'dev' on dev branch, then on this branch run cargo bench --bench transfer_bench -- --baseline 'dev'

CLType of `{"ByteArray":32}` returned in deploy

When querying the deploy with hash f81d0a32890b47a5dfbb55c7bffe75d2a9301c442f7857b1908409b8e9921e83 on mainnet returns a JSON structure with the following:

        "session": {
          "Transfer": {
            "args": [
              ...
              [
                "target",
                {
                  "cl_type": {
                    "ByteArray": 32
                  },
                  "bytes": "94664ce59fa2e6eb0cc69d270fc91dd9dd2ba02a1f7964c69f036ef4a68bb96f",
                  "parsed": "94664ce59fa2e6eb0cc69d270fc91dd9dd2ba02a1f7964c69f036ef4a68bb96f"
                }
              ],
              ...
            ]
          }
        },

however cl_type is meant to be an enum rather than an object. Given the byte array size can be inferred from the bytes field can this be altered so the cl_type is just a simple ByteArray value as per other CL types?

Node Crash [ DELTA 8 ]

Based on:

I've found the "scheduler_queue_total_count" to be a more useful one tbh. It shows the current total backlog of these kind of events waiting to be processed. We aim for it to always be very low - ideally single or double digits. When we saw the recent OOM issues, before the node crashed, that value would invariably skyrocket to hundreds of thousands - meaning we had a huge backlog of events

Here is the moment of actual crash: ( 2021-01-10 14:11:52 UTC )

Since we sure this is OOM most luckily:

Here is no doubts, the reason for node to loose ability providing us with HTTP and RPC end point is coming from the same source. We can observe it on mostly every graph, as example I post graphs below.

From here we can see, issue start accumulating before the actual catastrophe ( 2021-01-10 06:27:00 UTC )

Update 1

I been tracking network activity, and here is some in case can be usefull.

From here we can see, since my crawler still crawling after node colapse, new IP addresses still joining chain.

At moment of crash my node have 69 peers connected, auction order book have 65 bids and network coun 132 unique IP addresses. My validator crash happen when 133 addresses been count, it was exactly on the moment when new IP join network, not sure if this can be related to some specific missconfigured node joining.

Update 2

Last IP's joined network:

130 95.216.224.235
131 95.217.0.233
132 95.217.154.113
133 95.217.154.200
134 95.217.213.84
135 95.217.84.84
136 95.217.94.83

Update 3

List of current "still alive peers":

 104.197.195.68  Era: 101       Height: 2647  
 104.236.61.140  Era: 101       Height: 2647  
 107.22.247.79   Era: 101       Height: 2647  
 116.202.106.191 Era: 83        Height: 2277  
 116.202.24.91   Era: 80        Height: 2205  
 116.203.157.37  Era: 101       Height: 2647  
 116.203.206.169 Era: 101       Height: 2647  
 116.203.214.160 Era: 101       Height: 2647  
 116.203.254.245 Era: 101       Height: 2647  
 116.203.69.88   Era: 101       Height: 2647  
 129.146.102.188 Era: 101       Height: 2647  
 129.146.102.28  Era: 101       Height: 2647  
 129.146.137.190 Era: 101       Height: 2647  
 129.146.166.34  Era: 101       Height: 2647  
 129.146.17.202  Era: 101       Height: 2647  
 129.146.243.139 Era: 101       Height: 2647  
 129.146.58.40   Era: 101       Height: 2647  
 129.146.83.137  Era: 101       Height: 2647  
 134.209.16.172  Era: 101       Height: 2647  
 135.181.129.210 Era: 101       Height: 2647  
 135.181.143.233 Era: 97        Height: 2605  
 135.181.163.79  Era: 101       Height: 2647  
 135.181.192.149 Era: 101       Height: 2647  
 135.181.25.45   Era: 101       Height: 2647  
 135.181.30.191  Era: 101       Height: 2647  
 135.181.35.187  Era: 101       Height: 2647  
 135.181.40.95   Era: 101       Height: 2647  
 135.181.89.76   Era: 101       Height: 2647  
 135.181.97.186  Era: 101       Height: 2647  
 13.68.180.213   Era: 101       Height: 2647  
 138.197.65.212  Era: 101       Height: 2647  
 143.110.225.98  Era: 101       Height: 2647  
 144.76.199.44   Era: 101       Height: 2647  
 144.91.109.12   Era: 101       Height: 2647  
 157.90.19.142   Era: 101       Height: 2647  
 157.90.26.194   Era: 101       Height: 2647  
 158.101.3.185   Era: 101       Height: 2647  
 158.247.204.136 Era: 101       Height: 2647  
 161.97.124.55   Era: 101       Height: 2647  
 161.97.152.70   Era: 101       Height: 2647  
 161.97.82.77    Era: 101       Height: 2647  
 161.97.96.132   Era: 101       Height: 2647  
 167.86.75.50    Era: 101       Height: 2647  
 168.119.123.213 Era: 101       Height: 2647  
 168.119.173.204 Era: 101       Height: 2647  
 168.119.179.150 Era: 101       Height: 2647  
 168.119.59.64   Era: 101       Height: 2647  
 172.105.13.176  Era: 101       Height: 2647  
 173.212.205.2   Era: 101       Height: 2647  
 194.19.235.78   Era: 101       Height: 2647  
 195.201.148.78  Era: 96        Height: 2565  
 195.201.220.118 Era: 101       Height: 2647  
 207.180.198.56  Era: 101       Height: 2647  
 207.180.245.251 Era: 101       Height: 2647  
 3.19.76.184     Era: 101       Height: 2647  
 34.121.43.83    Era: 101       Height: 2647  
 34.65.104.178   Era: 101       Height: 2647  
 35.198.176.85   Era: 101       Height: 2647  
 35.233.165.80   Era: 101       Height: 2647  
 40.75.89.48     Era: 101       Height: 2647  
 46.4.15.204     Era: 101       Height: 2647  
 49.12.109.102   Era: 101       Height: 2647  
 49.12.66.192    Era: 97        Height: 2605  
 49.12.96.112    Era: 101       Height: 2647  
 52.247.126.146  Era: 101       Height: 2647  
 52.247.219.167  Era: 101       Height: 2647  
 78.46.121.69    Era: 101       Height: 2647  
 78.46.186.163   Era: 101       Height: 2647  
 78.47.10.119    Era: 101       Height: 2647  
 78.47.110.240   Era: 101       Height: 2647  
 78.47.9.189     Era: 101       Height: 2647  
 88.198.227.113  Era: 101       Height: 2647  
 88.99.120.29    Era: 101       Height: 2647  
 88.99.36.131    Era: 101       Height: 2647  
 94.130.224.158  Era: 101       Height: 2647  
 95.216.147.88   Era: 101       Height: 2647  
 95.216.164.122  Era: 101       Height: 2647  
 95.217.154.113  Era: 101       Height: 2647  
 95.217.154.200  Era: 101       Height: 2647  
 95.217.213.84   Era: 101       Height: 2647  
 95.217.84.84    Era: 101       Height: 2647

Dead peers:

103.212.121.38                               
116.202.17.49                                
116.203.157.213                              
116.203.182.37                               
135.181.118.39                               
135.181.147.7                                
135.181.152.29                               
135.181.42.149                               
13.57.200.251                                
144.76.111.74                                
144.91.123.104                               
144.91.83.59                                 
157.90.21.155                                
157.90.26.198                                
158.101.12.238                               
159.146.54.127                               
159.69.218.20                                
161.97.136.153                               
161.97.143.67                                
167.99.74.47                                 
168.119.126.225                              
168.119.169.105                              
168.119.177.31                               
168.119.177.37                               
168.119.250.90                               
168.119.49.220                               
168.119.59.12                                
173.212.200.158                              
173.249.31.34                                
18.144.176.168                               
18.217.179.179                               
20.57.161.70                                 
206.189.4.64                                 
207.180.199.157                              
207.180.213.119                              
207.180.243.57                               
34.122.124.107                               
34.123.112.124                               
34.75.153.42                                 
35.236.56.221                                
35.239.228.18                                
45.61.52.142                                 
46.4.91.24                                   
62.171.165.37                                
78.46.198.83                                 
78.46.200.17                                 
78.47.131.222                                
78.47.18.35                                  
78.47.218.195                                
88.198.200.113                               
88.99.171.225                                
88.99.227.165                                
95.216.224.235                               
95.217.0.233                                 
95.217.94.83

Last delta-8 geomap:

Geodata CSV attached
delta-8_geo_data.txt

Authenticate with validator key when connecting

This will allow us to prioritize validator nodes over others in the future.

https://casperlabs.atlassian.net/browse/NDRS-1231

Enhance deploy acceptor to validate associated keys

Upgraded node won't rejoin after being stopped and restarted

Use different namespaces for different networks.

Currently a node uses a single directory to hold its LMDB database files and various "unit" files created by consensus. To avoid problems if a node is run against e.g. the testnet and later the mainnet, we need to create a subdirectory in the storage folder which is named as per the given network, i.e. it will use the chainspec.network.name value for the subdir name.

There are a few aspects to this task:

need to pass the network name in via Storage::new(). This will be done similarly to the protocol_version: it's also specified in the chainspec, and the initializer reactor (which constructs the Storage component) accesses the chainspec by calling the ChainspecLoader component.
the ContractRuntime component's constructor should be changed. Currently it takes a copy of the Storage component's config (storage_config: WithDir<StorageConfig>), but only needs that to deduce the path in which to create its own LMDB files. We should replace that variable with a simple PathBuf or &Path which will provided via a new getter on the Storage component. The initializer reactor has access to the newly-constructed Storage just before it constructs the ContractRuntime.
We need to ensure when nodes upgrade from the current version to this new one that they move their existing database files. This code should only need to exist while we're transitioning from the currently running version of node to the next one, so the simplest way to achieve this is to have the the Storage component move the 4 relevant files in its constructor. The files are data.lmdb, data.lmdb-lock, storage.lmdb and storage.lmdb-lock. They just need to be moved from what is currently root to the new subdir of root. That code should have a TODO added to help remind us to remove it after the next version is published, as we don't want this behaviour to persist after the upgrade in which this change is introduced.
All existing tests should be updated if required.
Should add a new test to the existing suite of Storage unit tests to exercise the behaviour of moving the LMDB files. Note that the data.lmdb and data.lmdb-lock files need not be actual LMDB files for the purposes of this test. They're the DB files backing global state (owned by ContractRuntime), but the test only needs to ensure they're moved appropriately - they won't be actually read as DB files. We tend to use tempfile::tempdir() for testing such filesystem-related things.

Node cant start and communicate with local api after / is full

I have installed casperlabs in / mountpoint.
The node was running as a validator and the mount point became 100% full and the node crashed. I have cleared the mount point and tried restarting the node but it fils with the below error.

CHain : Delta-10

0: casper_node::panic_hook
1: core::ops::function::Fn::call
2: std::panicking::rust_panic_with_hook
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:595:17
3: std::panicking::begin_panic_handler::{{closure}}
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:497:13
4: std::sys_common::backtrace::__rust_end_short_backtrace
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/sys_common/backtrace.rs:141:18
5: rust_begin_unwind
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:493:5
6: std::panicking::begin_panic_fmt
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:435:5
7: casper_node::effect::EffectBuilder::load_state::{{closure}}::{{closure}}
8: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
9: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
10: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
11: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
12: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
13: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
14: <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
15: tokio::runtime::task::raw::poll
16: std::thread::local::LocalKey::with
17: tokio::runtime::thread_pool::worker::Context::run_task
18: tokio::runtime::thread_pool::worker::Context::run
19: tokio::macros::scoped_tls::ScopedKey::set
20: tokio::runtime::thread_pool::worker::run
21: tokio::loom::std::unsafe_cell::UnsafeCell::with_mut
22: tokio::runtime::task::core::Core<T,S>::poll
23: <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
24: tokio::runtime::task::harness::Harness<T,S>::poll
25: tokio::runtime::blocking::pool::Inner::run
26: tokio::runtime::context::enter
27: std::sys_common::backtrace::__rust_begin_short_backtrace
28: core::ops::function::FnOnce::call_once{{vtable.shim}}
29: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/alloc/src/boxed.rs:1328:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/alloc/src/boxed.rs:1328:9
std::sys::unix::thread::Thread::new::thread_start
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/sys/unix/thread.rs:71:17
30: start_thread
31: clone

panicked at 'could not deserialize state from storage type name "core::option::Option<casper_node::components::block_proposer::deploy_sets::BlockProposerDeploySets>" err InvalidTagEncoding(61)', node/src/effect.rs:1252:13
Error: "/var/lib/casper/bin/1_0_0/casper-node" "validator" "/etc/casper/1_0_0/config.toml" exited with error

After the node synchronizes to the highest block, could not deserialize state from storage type name

When the node synchronizes to the highest block, check the block height，in 3 hours

this is my log

look like this node always restart

and this is another log

Finalized block that hasn't been proposed

Logs:
casper-logs.log

This Finalized block that hasn't been proposed error started showing up after I raised my bid in the auction.

Logs after restarting the node:
https://pastebin.com/a0GUqhzZ

System specs:
AMD Ryzen 9 3900 12c/24t
128GB DDR4 ECC
2x 2TB NVMe SSDs RAID0

The problem and solution of node startup failure + peer is 0

node status，Attempt to restart but failed

check peer is 0

check casper-node.stderr.log,Prompt is related to the config.toml

check casper-node.log ,Found that a node is down

solution：
After deleting the node that reported the error, the service started normally, with more than 40 peers

Gossiper on rampage

Observing last 6 hours on my monitoring solution, I notice table items are significantly increased.
First thoughts it was someone spamming allot of deploys, but observing deploys count I figure out this is not the issue.

Here is some screenshots from my monitor ( 6 hour frame ):

Items finished:

Items current:

Deploys onward:

Table items finished:

Unpleasant consequences:

CPU utilization:

Memory usage:

Interrupts/s:

On load average we can observe strange behavior where in same 6 hours time frame 5 and 15 minutes decrease, but small momentum 1 minute spikes is growing.

Some stepped memory decrease ( is upside down in zabbix ):

current_era metrics is temporarily set to 0 in the very first era after an upgrade.

could not find the port 7777

timestamp format is invalid for key `network.timestamp` at line 208 column 1

hello, when running-one-node), I get this error

Mar 17 02:45:30.638 INFO  [casper_node::cli cli.rs:149] node starting up; version=0.9.0-cb1d20ad
Mar 17 02:45:30.638 WARN  [casper_node::reactor reactor.rs:437] large event size, consider reducing it or boxing; event_size=272
Error: chainspec error: decoding from TOML error: timestamp format is invalid for key `network.timestamp` at line 208 column 1

Caused by:
    0: decoding from TOML error: timestamp format is invalid for key `network.timestamp` at line 208 column 1
    1: timestamp format is invalid for key `network.timestamp` at line 208 column 1

➜  casper-node pwd
/etc/casper-node
➜  casper-node ls
chainspec.toml  config.toml

chainspec.toml and config.toml copy from chainspec.toml and config.toml

[network]
# Human readable name for convenience; the genesis_hash is the true identifier.  The name influences the genesis hash by
# contributing to the seeding of the pseudo-random number generator used in contract-runtime for computing genesis
# post-state hash.
name = 'casper-example'
# Timestamp for the genesis block.  This is the beginning of era 0. By this time, a sufficient majority (> 50% + F/2 —
# see finality_threshold_percent below) of validator nodes must be up and running to start the blockchain.  This
# timestamp is also used in seeding the pseudo-random number generator used in contract-runtime for computing genesis
# post-state hash.
timestamp = '${TIMESTAMP}'

How should I do?

Node dead after start [ DELTA 6 ]

Trying to join delta-6 recently, after couple of minutes node crash with provided error:

   0: casper_node::panic_hook
   1: core::ops::function::Fn::call
   2: std::panicking::rust_panic_with_hook
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:597:17
   3: std::panicking::begin_panic_handler::{{closure}}
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:499:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/sys_common/backtrace.rs:141:18
   5: rust_begin_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:495:5
   6: std::panicking::begin_panic_fmt
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:437:5
   7: casper_node::components::consensus::<impl casper_node::components::Component<REv> for casper_node::components::consensus::era_supervisor::EraSupervisor<I>>::handle_event
   8: <casper_node::reactor::validator::Reactor as casper_node::reactor::Reactor>::dispatch_event
   9: casper_node::cli::Cli::run::{{closure}}
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: std::thread::local::LocalKey<T>::with
  12: tokio::runtime::enter::Enter::block_on
  13: tokio::runtime::thread_pool::ThreadPool::block_on
  14: tokio::runtime::context::enter
  15: casper_node::main
  16: std::sys_common::backtrace::__rust_begin_short_backtrace
  17: std::rt::lang_start::{{closure}}
  18: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:381:40
      std::panicking::try
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:345:19
      std::panic::catch_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panic.rs:396:14
      std::rt::lang_start_internal
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/rt.rs:51:25
  19: main
  20: __libc_start_main
  21: _start

panicked at 'fatal error [node/src/components/consensus/era_supervisor.rs:811]: Consensus shutting down due to inability to participate in the network', node/src/effect.rs:423:9

Update

Probably getting this from nodes which are on previous delta still.

{
  "timestamp": "Dec 30 09:32:32.937",
  "level": "INFO",
  "fields": {
    "message": "NodeId::Tls(3da7..bf59): dropping connection to NodeId::Tls(4037..0b1e) due to genesis config hash mismatch",
    "our_hash": "42edd6b09c101a65d7aa3626bf0811cb39fa4d2d3957d8a28868edd4aa8d1a9e",
    "their_hash": "0e9f4c541abeb97a34708f732b6d0acc7345dbda394ae115ffef213461d1a892"
  },

Update 2

Attached logs are in pure JSON format, can be easily to operate, this is full cycle from start to crash.

Inside zip is tar.xz archive, GitHub only support zip, sorry for it

delta-6.zip

Unable to obtain historical stake

I'm attempting to obtain historical (balance+stake) information from the JSON-RPC interface. I can obtain historical balance (which is currently always 0 as all balance is staked) and current stake (from the state_get_auction_info method) but not historical stake.

Is historical stake available somewhere? Or some other method to calculate an account's (balance+stake), for example obtaining per-era reward data?

Peers remote port reported from incoming instead of outgoing connection

The /status endpoint seems to currently report the wrong port for bidirectional nodes, using incoming instead of outgoing connections. This behavior should be reverted to be in line with the previous one, where an outgoing connection always wins out over an incoming one.

Accidentally first filed as https://github.com/casper-network/Governance/issues/7

Limit outgoing traffic to a fixed amount per node

See https://hackmd.io/Y99IYqXVSWaGAwj_u4wqkA for details, section "3 steps plan".