trezor / blockbook Goto Github PK

There is an issue (caused by the design decision in Blockbook) with pay to public key addresses P2PK: bitcoin type coins in Blockbook use public key scripts, not addresses (as the value returned by by the abstraction bchain.AddressDescriptor from parsers) to index data. When searching for an address in explorer, Blockbook must convert the address to the script before the lookup.
The problem with P2PK address is that it is the same as pay to public key hash, but has different script (which cannot be obtained from the address). Blockbook therefore does not find the transactions of P2PK address and also computes balances of P2PK and P2PKH separately, with different results than returned some other explorers.

Fortunately there is a way how to change this behavior: detect in the parser.GetAddrDescFromVout and parser.GetScriptFromAddrDesc P2PK script and convert it to P2PKH script.

When I stop bitcoind by

systemctl stop backend-bitcoin.service

On the next start, I get tons of "rolling forward" messages and rolling forward takes hours.

I can find this issue here

bitcoin/bitcoin#11600

Citing the comment from Blue Matt there:

This happens when bitcoind isnt allowed to shut down cleanly after connecting many blocks with a large dbcache. Ideally we'd have a mechanism for background flushing the cache to disk so that we dont end up in this state, but if you do a full, normal, clean shutdown, this shouldn't happen.

In my opinion, systemctl stop backend-bitcoin.service is not shutting down the bitcoind cleanly.

Hello,

Any thoughts of a UTXO index, or the ability to query available UTXO's similar to how insight used to provide?

ex. https://btc.faa.st/insight-api/addr/1Hz96kJKF2HLPGY15JWLB5m9qGNxvt8tHJ/utxo

Thanks

  ✗  github.com/gorilla/handlers

However, these projects are not direct dependencies of the current project:
they are not imported in any .go files, nor are they in the 'required' list in
Gopkg.toml. Dep only applies [[constraint]] rules to direct dependencies, so
these rules will have no effect.

Either import/require packages from these projects so that they become direct
dependencies, or convert each [[constraint]] to an [[override]] to enforce rules
on these projects, if they happen to be transitive dependencies,

Warning: Gopkg.lock is out of sync with Gopkg.toml or the project's imports.
Fetching sources...

I put in my Gopkg.toml the gorilla handlers to [[override]]
Is this temporarily or can a commit be made to put it to override ?

README.md mentions we can install blockbook using Docker:

Install using Docker:

git clone https://github.com/jpochyla/blockbook.git
cd blockbook/docker
./build.sh

But there is no docker folder inside the repository, so you can't cd into it and therefore can't find build.sh

Confirmed the following OSX Homebrew package/formula dependencies used to build and run blockbook that connected to a Dash Core version v0.12.3.2 (64-bit) full node:

1. pkg-config:
2. zlib:
3. bzip2:
4. lz4:
5. gflags:
6. go:
7. lxc: go
8. dep: go
9. zeromq: pkg-config 
10. snappy: pkg-config
11. rocksdb: snappy lz4 gflags     
12. wget: pkg-config libidn2 openssl

Additional dependency details above resulted from a quick brew info <formula> dependency analysis.

When I try to send an raw Ethereum transaction via websockets sendTransaction method, I get error 'SendRawTransaction: not implemented'. https://github.com/trezor/blockbook/blob/master/bchain/coins/eth/ethrpc.go#L502-L504 - as long its not implemented, how to send transactions in blockbook for Ethereum?

I am currently implementing support for Myriad but because the DecodeAddress function in github.com/btcsuite/btcutil checks all registered networks to determine if it is a P2PKH or P2SH address it throws ErrAddressCollision.

We get an address conflict between Litecoin and Myriad (first letter in Litecoin P2SH is same as first letter in Myriad P2PKH).

I saw that Monacoin solved this by registering a dummy network. I do not think that solution is very good. How would you advise?

I have tried to install blockbook according to this manual

https://wiki.trezor.io/User_manual:Running_a_local_instance_of_Trezor_Wallet_backend_(Blockbook)

after installing bitcoind and waiting about a week to sychronize blockchain, I try to install blockbook. gdebi writes me

This package is uninstallable
Dependency is not satisfiable: libstdc++6 (>= 6)

....I think blockbook (or the backend installer) could have told me this first, before I synched the blockchain.

"Terms" in explorer footer lead to wallet.trezor.io terms

I am not sure if that is intended or not

How to enable SSL/disable SSL/put in actual (not self-signed) certificate?

It would be really useful if the whole thing ran as a docker container. You need to install docker anyway, might as well make it so that we can build a docker image and run it anywhere.

Would be nice blockbook will listen on it's public port also due initial sync. It can says something like bitcoind do "Syncing...".

Reason is if you're using some upstream proxy with more /alias URLs, then load-balancer consider whole virtualhost down because root URL does not reply valid response.

Thanks

Following the manual build instructions I run into the following error on ARM 32-bit (armhf):

$ cd $GOPATH/src/blockbook
$ go build
# blockbook/vendor/github.com/tecbot/gorocksdb
vendor/github.com/tecbot/gorocksdb/db.go:199: type [1073741824]*_Ctype_char larger than address space
vendor/github.com/tecbot/gorocksdb/db.go:199: type [1073741824]*_Ctype_char too large

I opened an issue with the gorocksdb project: tecbot/gorocksdb#145

Hi,
I'm trying to add GameCredits support to Blockbook, so that we can move forward with Trezor integration (for Electrum) by getting our coin added to trezor-common.
Using Litecoin as a base, since that is what the currency is forked from, I've customized several of the files to match the GameCredits network's configuration. You can see the changes at my repo here: https://github.com/samadsajanlal/blockbook

I believe this is because we share the mainnet magic packet with Litecoin (0xdbb6c0fb), and unfortunately that was due to an error way back when the code was created (before I took over development). Can this be confirmed? What options do I have here to get blockbook working with GAME?

However, when I attempt to launch it, I get the following error.

root@server2:~/go/src/blockbook# ./blockbook -sync -blockchaincfg=build/blockchaincfg.json -internal=:9044 -public=:9144 -certfile=server/testcert -logtostderr
I1008 04:18:49.245092   10369 blockbook.go:136] Blockbook: {Version:unknown GitCommit:unknown BuildTime:unknown GoVersion:go1.10.3 OSArch:linux/amd64}, debug mode false
I1008 04:18:49.254941   10369 mq.go:60] MQ listening to tcp://127.0.0.1:38344
I1008 04:18:49.255062   10369 mempool_utxo.go:66] mempool: starting with 8*2 sync workers
I1008 04:18:49.255117   10369 gamecreditsrpc.go:39] Chain name main
panic: duplicate Bitcoin network

goroutine 1 [running]:
blockbook/bchain/coins/gamecredits.GetChainParams(0xc4200a87d0, 0x4, 0x2)
	/root/go/src/blockbook/bchain/coins/gamecredits/gamecreditsparser.go:57 +0x569
blockbook/bchain/coins/gamecredits.(*GameCreditsRPC).Initialize(0xc4200c8050, 0x1a6, 0x1c0)
	/root/go/src/blockbook/bchain/coins/gamecredits/gamecreditsrpc.go:40 +0x109
blockbook/bchain/coins.NewBlockChain(0xc4200a8470, 0xb, 0x7fffdcb16748, 0x18, 0x1024060, 0xc4200f2780, 0x0, 0xeea8c0, 0x1f, 0xef658d)
	/root/go/src/blockbook/bchain/coins/blockchain.go:95 +0x196
main.getBlockChainWithRetry(0xc4200a8470, 0xb, 0x7fffdcb16748, 0x18, 0x1024060, 0xc4200f2780, 0x3c, 0x0, 0x10700e0, 0xc4201a5590, ...)
	/root/go/src/blockbook/blockbook.go:108 +0xc4
main.main()
	/root/go/src/blockbook/blockbook.go:167 +0x686
root@server2:~/go/src/blockbook# 

Currently, we detect the chain by

• looking at 0 height
• if it is bitcoin, we have some hard-coded rules by reading "version" from some networkInfo call

However, since we don't plan to be compatible with Bitcore in the future and so don't plan to keep compatibility with third party insight servers, we can make this easier by explicitly returning the coin name in some simple API.

And the best thing would be coin_name from https://github.com/trezor/trezor-common/tree/master/defs/coins

Hi, Insight api has this call: https://insight.bitpay.com/api/txs?address=3FRpHy6ZwWDTUGrXY9YFTqnC1P9WMho7MF while blockbook doesn't seem to support it:

https://btc-bitcore1.trezor.io/api/txs?address=3FRpHy6ZwWDTUGrXY9YFTqnC1P9WMho7MF

Perhaps you can suggest another API request that would accomplish the task of getting new transactions to an address.

Hey folks,

I was directed here by the customer support. I noticed that you now provide a full Dogecoin wallet interface along with the Trezor support that has been there before. As Dogecoin dev that made me wonder what version of Dogecoin Core you're running given ZMQ support is currently only available in the alpha version of 1.14. Maybe I'm misunderstanding but blockbook seems to rely on ZMQ connections to the coin daemon, no?

Anyway, if that is the case I'd be happy to have any feedback about that version given you'd be the first large scale service to run it.

Added a PR trezor/python-trezor#344 to assume locktime:0 if none is provided. This seems to break some trezorlib sign_tx apis for versions < 0.11. Exposures may be limited to PAYTOADDRESS and trezorctl

Electrum does not seem to be effected.

Please add prometheus gauge metrics at least as follows:

Blockbook version:
blockbook{coin="bitcoin", version="0.0.6", "gitcommit"="hash", ...} 1

Backend version
backend{coin="bitcoin", version="0.16.2" (or similar), subversion="satoshi/0.16.2", ...} 1

Name of variables can be changed for some reason, but must be same across blockbooks/backends

Thank you !

Docker version 18.06.1-ce on armhf:

$ make all-bitcoin
rm -rf build/pkg-defs
rm -f build/*.deb
rm -f .bin-image .deb-image
make .bin-image .deb-image
make[1]: Entering directory '/opt/src/trezor/blockbook'
docker build --no-cache=false -t blockbook-build build/docker/bin
Sending build context to Docker daemon   5.12kB
Step 1/20 : FROM debian:9
 ---> b4717fbcbbda
Step 2/20 : RUN apt-get update &&     apt-get upgrade -y &&     apt-get install -y build-essential git wget pkg-config lxc-dev libzmq3-dev                        libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev                        liblz4-dev graphviz &&     apt-get clean
 ---> Using cache
 ---> 085d7047591c
Step 3/20 : ENV GOLANG_VERSION=go1.10.linux-amd64
 ---> Using cache
 ---> 92746d74b368
Step 4/20 : ENV GOPATH=/go
 ---> Using cache
 ---> ad00698b786a
Step 5/20 : ENV PATH=$PATH:$GOPATH/bin
 ---> Using cache
 ---> a1ec994aaaf3
Step 6/20 : ENV CGO_CFLAGS="-I/opt/rocksdb/include"
 ---> Using cache
 ---> 4930dba434a2
Step 7/20 : ENV CGO_LDFLAGS="-L/opt/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4"
 ---> Using cache
 ---> d97fd0576e83
Step 8/20 : RUN mkdir /build
 ---> Using cache
 ---> 9677a3f86dcd
Step 9/20 : RUN cd /opt && wget https://storage.googleapis.com/golang/$GOLANG_VERSION.tar.gz &&     tar xf $GOLANG_VERSION.tar.gz
 ---> Using cache
 ---> 8c3e122d14f6
Step 10/20 : RUN ln -s /opt/go/bin/go /usr/bin/go
 ---> Using cache
 ---> c6749f8cdb99
Step 11/20 : RUN mkdir -p $GOPATH
 ---> Using cache
 ---> d2380b3be514
Step 12/20 : RUN echo -n "GO version: " && go version
 ---> Running in 72e8f2d44bb1
GO version: /bin/sh: 1: go: Exec format error
The command '/bin/sh -c echo -n "GO version: " && go version' returned a non-zero code: 2
Makefile:42: recipe for target '.bin-image' failed
make[1]: *** [.bin-image] Error 2
make[1]: Leaving directory '/opt/src/trezor/blockbook'
Makefile:38: recipe for target 'build-images' failed
make: *** [build-images] Error 2

Bitcore API returns fee, blockbook API does not.

In TXs without input values (for example nonsegwit btc) this is impossible to calculate just from the TX.

Currently, it's on 0.16.1. Please update so CVE-2018–17144 is taken care of.

This address has negative value in explorer

https:/[url]/explorer/address/3JoptT3Po7AnvuAZ7spXGtPxZxdDsEzzj3

moving issue from https://github.com/satoshilabs/mytrezor/issues/385,

When i click on "details" of BTG transaction (in wallet) i'm getting redirected from:
https://btg1.trezor.io/tx/[txid]
to
https://explorer.bitcoingold.org/insight/

and i not able to see requested transaction detail

The computer syncing blockbook to the bitcoin blockchain briefly lost power. I now get the following error message when starting blockbook:

I1127 07:33:36.032267    1075 rocksdb.go:88] rocksdb: opening /opt/coins/data/bitcoin/blockbook/db, required data version 3, cache size 1073741824, max open files 16384
E1127 07:33:36.235710    1075 blockbook.go:184] internalState: database is in inconsistent state and cannot be used
I1127 07:33:36.235752    1075 rocksdb.go:118] rocksdb: close

Is there any way to recover or restore the database? What is the best way to move forward? Thanks!

Especially during initial index import the RocksDB memory usage of Blockbook is large and unpredictable. Find a way (options) that will make the memory usage limited.

with curl http://example.compute.amazonaws.com:19036 i get answer ....
But with axios or fetch have problem
Failed to load http://example.compute.amazonaws.com:19036/: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8000' is therefore not allowed access.
blockbook have any options for resolve it?

From all the functionality in explorer, I would like some easy way to see raw tx, just hex

similar to bitcore

https://testnet.blockexplorer.com/api/rawtx/778a2656159c9542d3c3d8182e866e08c58bde08a9acd188de6cc450b2d48fb7

I encountered a strange issue. I have just installed blockbook for myriad and it seems some addresses are displayed as bitcoin addresses and some as myriad addresses.

You can clearly see the issue here:
https://blockbook.myralicious.com/block/c74c617d39d67e8cf7eb095f94670df054d36944468ef215d766168866a1abca

I cannot figure out why this occurs.
Do you have any idea?

I noticed blockbook will run an extremely IO intensive function periodically:

func (d *RocksDB) ComputeInternalStateColumnStats(stopCompute chan os.Signal) 

This is causing all the IO of my server to be taken, rendering all other processes unresponsive.

What is the reason for this? Is this operation just for statistics purpose, can it be made optional?

When making an api call GetSystemInfo is returning the exact "gitcommit" version of blockbook that is running as well as the local "host" name of system returning the call. The "host" value can give an attacker insight to the internals of your network (like addressing). Displaying the gitcommit in api call gives an easy way for attacker to determine if the running endpoint is vulnerable should an exploit become known.

Is there any technical reason these need to exist? Would you consider a configuration option to limit these fields from output? Thank you!

Is there any ready-to-use blockbook server?
I'm trying to make requests to btc-bitcore1.trezor.io and btc-bitcore2.trezor.io but they always return index response.
According to code in https://github.com/trezor/blockbook/blob/master/server/https.go the next request should return unconfirmed transactions for given address: https://btc-bitcore2.trezor.io/unconfirmedTransactions/3D9YzhPQ6xQRscNcRf9Nv9b78VLiv65s7L but it (as any other request) returns only response from index method.

A valid address with zero transactions should be shown in the explorer. Explorer now returns error "Address not found". See satoshilabs/mytrezor#454.

I want to change the "about" text :) so I don't see "trezor.io" there.

I can change it in the source (which I do) but that is not good, since it needs rebuild

I added new coin with blockbook servers to coins.json. Then, I sent some tokens to address generated by trezor emulator. Now, I would like to see balance of the account using connect's method getAccountInfo. I can clearly see transactions returned by blockbook server, but the balance is still zero. Is blockbook responsible for summing up transactions and returning final balance, or is this a problem on trezor-connect's side?

I was looking for a complete list of URLs for testnet instances run below trezor.io.

I found https://testnet-bitcore1.trezor.io in a Reddit post, are their instances for the other currencies available on wallet.trezor.io (excluding the ones that direct to third-party wallets).

It is possible to run one instance of Blockbook for more coins (RPC connections)?
Seems this is the idea of Blockbook, but not sure.

Will Blockbook be compatible with Bcoin in the future? I took a look and found that Bcoin's rpc calls have a different format than Bitcoind (different parameters and result formatting). Is there an easy way to make Blockbook send the correct rpc requests to Bcoin?

Explorer does not show OP RETURN text/data

The certificate for https only works with localhost. I can't run this software on one computer and point to it with another computer on the network.

on git master f11a690

Partial excerpt of contents of error

make[1]: Entering directory '/build/build/pkg-defs/blockbook'
[ -d /go/src/blockbook ] || cp -r /src /go/src/blockbook
make prepare-vendor
make[2]: Entering directory '/build/build/pkg-defs/blockbook'
Updating vendor
make[2]: Leaving directory '/build/build/pkg-defs/blockbook'
cd /go/src/blockbook && go test -tags unittest ./...
?       blockbook       [no test files]
?       blockbook/api   [no test files]
?       blockbook/bchain        [no test files]
?       blockbook/bchain/coins  [no test files]
ok      blockbook/bchain/coins/bch      0.057s
ok      blockbook/bchain/coins/btc      0.058s
ok      blockbook/bchain/coins/btg      0.061s
?       blockbook/bchain/coins/dash     [no test files]
ok      blockbook/bchain/coins/dogecoin 0.053s
ok      blockbook/bchain/coins/eth      0.118s
ok      blockbook/bchain/coins/litecoin 0.086s
ok      blockbook/bchain/coins/monacoin 0.108s
ok      blockbook/bchain/coins/namecoin 0.043s
?       blockbook/bchain/coins/utils    [no test files]
ok      blockbook/bchain/coins/vertcoin 0.042s
# blockbook/contrib/scripts
contrib/scripts/generate-port-registry.go:32:6: Config redeclared in this block
        previous declaration at contrib/scripts/check-ports.go:14:6
contrib/scripts/generate-port-registry.go:39:6: main redeclared in this block
        previous declaration at contrib/scripts/check-ports.go:21:6
ok      blockbook/bchain/coins/zec      0.053s
?       blockbook/build/templates       [no test files]
?       blockbook/common        [no test files]
ok      blockbook/db    0.224s
ok      blockbook/server        0.020s
Makefile:26: recipe for target 'test' failed
make[1]: *** [test] Error 2
make[1]: Leaving directory '/build/build/pkg-defs/blockbook'
dh_auto_test: make -j1 test returned exit code 2
debian/rules:6: recipe for target 'build' failed
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2

blockbook/contrib/scripts/check-ports.go
blockbook/contrib/scripts/generate-port-registry.go
Since two declarations of the same name(Config and main()) have been done, it may be better to change either.

referes to #4

@martinboehm I think you misunderstood, I use the testnet to run tests on the trezor web wallet using a trezor hardware wallet.

I see myself as a happy customer that does bug hunting in his spare time, I am asking for a list of testnet URI's to run against the different testnets. That is basically QA at no cost.

I have no problem with URI's changing, can take a 404.

This testnet address

2N8hwP1WmJrFF5QWABn38y63uYLhnJYJYTF

has very different end balance than it should.

It should have 0.2 btc, it has around 1 000 000 btc on blockbook explorer

Hi

Any ideas of support querying balances / txs / utxos from xpub( ypub / zpub)? I notice that trezor uses fastxpub to do account scanning locally, it may be faster on server side?

Thanks

Bitcoin ABC might have a hard-fork, which will make transaction be ordered differently in a block.

See https://www.reddit.com/r/btc/comments/9ehll3/a_technical_dive_into_ctor/

The problem is that with the new ordering, we might see in a block transaction before its parent/grandparent/..., which might produce some errors (the transaction will be spending an output of a transaction that we don't yet know, since it happens later in a block)

When I push invalid TX through the socket.io API, the socket disconnects instead of returning me usable error from the backend

Here we should use {{.CoinLabel}} instead of {{.CoinName}} to be consistent with this for example:

the following url:
https://bch1.trezor.io/tx/send
gives the following error message:

Tx not found, encoding/hex: invalid byte: U+0073 's'

This is misleading. A better reply would be something like:

Transaction not found: "send"
(encoding/hex: invalid byte: U+0073 's')

(alternately because /tx/send is Bitcore's send form, maybe this is worth special-casing?)

9 sync.go:242] getBlockWorker 2 connect block error hash 00000000786da017b80e0e66d55facb9f291fc5b2d26b0d1c578dc10860561cf: -1: Block not available (pruned data). Retrying...
I have bitcoin node with option prune=5000.
can the blockbook work with such data?
this problem can be solved?