yafimk / blockchain Goto Github PK

I use the following code:

    BlockChain *b = createBlockChain("/home/XXXXX/.bitcoin/blocks");
    if ( b )
    {
        const BlockChain::Block *block = b->readBlock();
        while ( block )
        {
            block = b->readBlock();
        }
        b->release();
    }

Processing of the first block (blk00000.dat) goes fine. But when the code goes 
to blk00001.dat it hits an error.

Output from the program:

Successfully opened block-chain input file 
'/home/stick/.bitcoin/blocks/blk00000.dat'
Warning: Missing block-header; early termination of this block-data file.
Successfully opened block-chain input file 
'/home/stick/.bitcoin/blocks/blk00001.dat'
Advanced to the next data file; but it does not start with a valid block.  
Aborting reading the block-chain.

What steps will reproduce the problem?
1. make on linux
2.
3.

What is the expected output? What do you see instead?
>>make
g++ *.cpp -o blockchain.out
BlockChain.cpp: In function ‘const char* getDateString(time_t)’:
BlockChain.cpp:81:84: error: ‘sprintf_s’ was not declared in this scope
  sprintf_s(scratch,"%4d-%02d-%02d", gtm->tm_year+1900, gtm->tm_mon+1, gtm->tm_mday );
                                                                                    ^
BlockChain.cpp: In constructor ‘ZombieFinder::ZombieFinder()’:
BlockChain.cpp:3057:16: warning: converting to non-pointer type ‘uint64_t 
{aka long unsigned int}’ from NULL [-Wconversion-null]
   mLastBalance = NULL;
                ^
BlockChain.cpp: In member function ‘virtual void 
BlockChainImpl::reportTransactionValues(uint32_t)’:
BlockChain.cpp:6581:73: error: ‘sprintf_s’ was not declared in this scope
    sprintf_s(scratch,512,"TransactionValues-%s.csv", getDateString(date));
                                                                         ^
main.cpp: In function ‘const char* getDateString(time_t)’:
main.cpp:24:84: error: ‘sprintf_s’ was not declared in this scope
  sprintf_s(scratch,"%4d-%02d-%02d", gtm->tm_year+1900, gtm->tm_mon+1, gtm->tm_mday );
                                                                                    ^
make: *** [blockchain.out] Error 1


What version of the product are you using? On what operating system?
Version: checked out today (Oct-08-2014)
0S:
>>uname -a
Linux APU 3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 
x86_64 x86_64 GNU/Linux


Please provide any additional information below.
>>grep sprintf_s *.*
BlockChain.cpp: sprintf_s(scratch,"%4d-%02d-%02d", gtm->tm_year+1900, 
gtm->tm_mon+1, gtm->tm_mday );
BlockChain.cpp:         sprintf_s(scratch,512,"EXPORT_%04d_%02d_%02d.csv", 
1900+gtm->tm_year, gtm->tm_mon+1, gtm->tm_mday );
BlockChain.cpp:         sprintf_s(scratch,512,"TransactionValues-%s.csv", 
getDateString(date));
main.cpp:   sprintf_s(scratch,"%4d-%02d-%02d", gtm->tm_year+1900, gtm->tm_mon+1, 
gtm->tm_mday );

What steps will reproduce the problem?
1. scan whole blockchain
2. statistics
3. process

What is the expected output? What do you see instead?
get this error msg (see screenshot)



What version of the product are you using? On what operating system?
ubuntu binary from website, ubuntu 14

Please provide any additional information below.
could this be a memory problem or is it more of a software problem? have 16GB 
RAM

You'll have to change line 

1793 of blockChain.cpp from 

"sprintf(scratch,"%s\\blk%05d.dat", mRootDir, mBlockIndex );"
to
"sprintf(scratch,"%s/blk%05d.dat", mRootDir, mBlockIndex );"

Since linux uses forward slashes instead of backslashes for path names.  After 
that

make
./blockchain.out pathtodatfile

works fine

I pulled the latest code yesterday and got this output when I ran the process 
command. Any ideas?
--------------------------------------------

....
statistics

Block Processing will gather statistics.
*** WARNING : This will consume an enormous amount of memory! ***
process

Beginning processing of 308919 blocks : Gathering Statistics=true
Processed block #10000 of 308919 total.
Processed block #20000 of 308919 total.
Processed block #30000 of 308919 total.
Gathering statistics for December 31, 2009 to January 1, 2010
Loading RenderDebug DLL
Scanning 32,607 bitcoin addresses and looking for zombies, relative to this 
date: 12, 31, 2009
Zombie Report:
Found 0 zombie addresses older than 558 days with a total balance of 0, 
excluding balances less than 0.0001
Found 29,823 living addresses newer than 558 days with a total balance of 
1,624,300, excluding balances less than 0.0001
Segmentation fault (core dumped)

What steps will reproduce the problem?
1. compile for x64 debug, execute with no options

What is the expected output? What do you see instead?

proper loading of blockchain is expected.
process is killed after "Failed to find transaction!" is printed


What version of the product are you using? On what operating system?

current code pull, compiled in VS2013 on Windows 8.1 x64 with 32GB RAM

keyboard.cpp: In function ‘const char** getInputString(uint32_t&)’:
keyboard.cpp:167:24: warning: comparison is always true due to limited range of 
data type [-Wtype-limits]
   if ( c >= 32 && c <= 127 )

What steps will reproduce the problem?
1. current version is not working with current blockchain as it has more than 
50mln addresses and more then 100mln? transactions

2. #define MAX_BITCOIN_ADDRESSES 48000000 // 40 million unique addresses.
#define MAX_TOTAL_TRANSACTIONS 76000000 // 40 million transactions.
#define MAX_TOTAL_INPUTS 268000000 // 200 million inputs.
#define MAX_TOTAL_OUTPUTS 268000000 // 200 million outputs
#define MAX_TOTAL_BLOCKS 600000     // 1/2 million blocks.

- that`s are highest numbers it could be compiled with - if go higher then:

Warning 1   warning D9025: overriding '/W3' with '/W4'   
2013\Projects\Project5\compiler\vc10win64\cl    blockchain
Error   2   error C2220: warning treated as error - no 'object' file 
generated   2013\Projects\Project5\BlockChain.cpp   3156    1   blockchain
Warning 3   warning C4307: '*' : integral constant 
overflow    2013\Projects\Project5\BlockChain.cpp   3156    1   blockchain


What version of the product are you using? On what operating system?
vs2013 win 8.1

Please provide any additional information below.

could not get a working version or recompile myslef. Any help appreciated! 
Thank you in advance.

This isn't an issue but rather a feature pull-request.

I implemented a better (but still imperfect) parsing of inputs and outputs 
public keys. 

A diff is attached.

One place in the new code is a bit messy, but anyway I believe this is patch 
could be useful. 

What steps will reproduce the problem?
1. I compiled with make then tried to run for the first time with 
"./blockchain.out ~/.bitcoin/blocks"
2. The first 200,000 or so blocks scan fine and then I started getting a lot of 
"failed to decode public key in output script" over various blocks but it keeps 
going.
3. Eventually it segfaults on block 249,957 with the below output.


WARNING: Failed to decode public key in output script. Block 249,957 : 
Transaction: 49 : OutputIndex: 0 scriptLength: 60
TRANSACTION 
HASH:587f949b39e4590e40ed2a1e06c9cfb3ae4b73a18f7261251e6e68a4a2d41896
==========================================
FAILED TO LOCATE PUBLIC KEY
ChallengeScriptLength: 73 bytes long
63 
aa 20 00 00 00 00 00 19 d6 68 9c 08 5a e1 65 83 
1e 93 4f f7 63 ae 46 a2 a6 c1 72 b3 f1 b6 0a 8c 
e2 6f 87 67 aa 20 6f e2 8c 0a b6 f1 b3 72 c1 a6 
a2 46 ae 63 f7 4f 93 1e 83 65 e1 5a 08 9c 68 d6 
19 00 00 00 00 00 87 68 
==========================================

WARNING: Failed to decode public key in output script. Block 249,957 : 
Transaction: 97 : OutputIndex: 0 scriptLength: 73
TRANSACTION 
HASH:b5765f55d18f0c0b31132c71294b40dde96d629538f2121d17f4bd4dc0c25008
Processed block #250000 of 339111 total.
blockchain.out: BlockChain.cpp:811: bool 
BlockImpl::readOutput(BlockChain::BlockOutput&): Assertion `0' failed.
Aborted (core dumped)

What version of the product are you using? On what operating system?
I'm running ubuntu 14.04 and the latest version of the code I downloaded on 
1/14/15

The program needs a huge amount of memory for the blockchain, as the size is 
determined at compile-time, I assume. This makes it currently impossible for me 
to run it on small 1&1 vservers and on EC2 micro instances. I would be ever so 
happy if the memory was dynamically allocated upon need. Even if a huge mmap'ed 
file on disk would be ok, I assume blocks are not read very often.

Thank you for your existing work, I have patched it to support Riecoin. 
https://bitbucket.org/ysangkok/blockchain

What steps will reproduce the problem?

./blockchain.out ~/.bitcoin/blocks/ | grep Timestamp | sort

Output:

Timestamp : 01/03/2009 11:15:05 MST
Timestamp : 04/24/2011 11:56:35 MST
…
Timestamp : 04/25/2011 00:03:53 MST
…
Timestamp : 04/25/2011 09:53:53 MST

(172 lines in total)

What is the expected output? What do you see instead?
I only get information for some blocks.

What version of the product are you using? On what operating system?
recent version on Linux 3.10.17
My blockchain is from bitcoin-qt v0.8.5-beta

Perhaps my blockchain is corrupt?

What steps will reproduce the problem?
1. run. scan. statistics. process
2.
3.

What is the expected output? What do you see instead?
precompiled exe process only first 30000 blocks and exits


What version of the product are you using? On what operating system?
win 8.1 32 gb ram

Please provide any additional information below.

What steps will reproduce the problem?
1. Run blockchain on a recent (November 2013) set of blocks.

What is the expected output? What do you see instead?

Segfault

What version of the product are you using? On what operating system?

SVN r27

Please provide any additional information below.

Here is a patch. It includes some extra asserts.


--- BlockChain.cpp      (revision 27)
+++ BlockChain.cpp      (working copy)
@@ -1462,7 +1462,7 @@
 // These limits work for the blockchain current as of July 1, 2013.
 // The limits can be revised when and if necessary.
 #define MAX_BLOCK_SIZE (1024*1024)*10  // never expect to have a block larger than 10mb
-#define MAX_BLOCK_TRANSACTION 1024             // never expect more than 1024 
transactions per block.
+#define MAX_BLOCK_TRANSACTION 2048             // never expect more than 1024 
transactions per block.
 #define MAX_BLOCK_INPUTS 32768                 // never expect more than 32768 total inputs
 #define MAX_BLOCK_OUTPUTS 32768                        // never expect more than 32768 total outputs
@@ -1715,6 +1715,7 @@
                bits = readU32();       // Get the bits field
                nonce = readU32();      // Get the 'nonce' random number.
                transactionCount = readVariableLengthInteger(); // Read the number of transactions
+               assert ( transactionCount < MAX_BLOCK_TRANSACTION );
                if ( transactionCount < MAX_BLOCK_TRANSACTION )
                {
                        transactions = mTransactions;   // Assign the transactions buffer pointer
@@ -2105,6 +2107,7 @@
        virtual const Block * processSingleBlock(const void *blockData,uint32_t blockLength)
        {
                const Block *ret = NULL;
+               assert ( blockLength < MAX_BLOCK_SIZE );
                if ( blockLength < MAX_BLOCK_SIZE )
                {
                        mSingleBlock.blockIndex = 0;
@@ -2122,6 +2125,7 @@
        virtual const BlockTransaction *processSingleTransaction(const void *transactionData,uint32_t transactionLength)
        {
                const BlockTransaction *ret = NULL;
+               assert ( transactionLength < MAX_BLOCK_SIZE );
                if ( transactionLength < MAX_BLOCK_SIZE )
                {
                        mSingleBlock.blockIndex = 0;

I'm running the 64 bit version you have provided in the link here: 
http://codesuppository.blogspot.se/2014/01/a-command-line-interface-for-blockcha
in.html

These are the command I run, in order: scan, statistics and process. The 
process command crashes. This is the last output I get:

-----------------
Finished processing all blocks in the blockchain.
Total Blocks: 309,031
Total Transactions: 41,857,249
Total Inputs: 98,548,695
Total Outputs: 110,057,228
39,999,999 transactions.
89,349,453 inputs.
99,999,999 outputs.
35,767,653 addresses.
Found 35,767,653 addresses which have ever been used.
Found 33,418,027 addresses with a zero balance.
Found 1,188,449 'dust' addresses (less than 1mbtc) with a total balance of 
111.93181 BTC
Found 905,605 addresses with a balance greater than 1mbtc but less than 1btc, 
total balance 84697.88281
Found 154,918 addresses with a balance greater than 1btc but less than 10btc, 
total btc: 392,905
Found 87,578 addresses with a balance greater than 10btc but less than 100btc, 
total: 3,262,611
Found 11,648 addresses with a balance greater than 100btc but less than 
1,000btc, total: 2,837,809
Found 1,328 addresses with a balance greater than 1,000btc but less than 
10,000btc, total: 2,943,814
Found 98 addresses with a balance greater than 10,000btc but less than 
100,000btc, total: 2,342,625
Found 2 addresses with a balance greater than 100,000btc, total: 255,455
Gathering final statistics.

What steps will reproduce the problem?
1. Build on OSX
2. run

What is the expected output? What do you see instead?
Expect everything to be parsed.

See segfault.

... much trimmed
Successfully opened block-chain input file '/Users/bruce/Library/Application 
Support/Bitcoin/blocks/blk00104.dat'
Scanned 277,415 block headers so far, 819 since last time.
Successfully opened block-chain input file '/Users/bruce/Library/Application 
Support/Bitcoin/blocks/blk00105.dat'
Scanned 278,284 block headers so far, 869 since last time.
Warning: Missing block-header; scanning for next one.
Process 32116 stopped
* thread #1: tid = 0x4b6280, 0x000000010000dc04 
blockchain.out`BlockChainImpl::readBlockHeader(this=0x0000000101000000) + 310 
at BlockChain.cpp:4353, queue = 'com.apple.main-thread, stop reason = 
EXC_BAD_ACCESS (code=1, address=0x10ba00000)
    frame #0: 0x000000010000dc04 blockchain.out`BlockChainImpl::readBlockHeader(this=0x0000000101000000) + 310 at BlockChain.cpp:4353
   4350                     for (uint32_t i=0; i<c; i++)
   4351                     {
   4352                         const uint32_t *check = (const uint32_t *)&temp[i];
-> 4353                         if ( *check == MAGIC_ID )
   4354                         {
   4355                             printf("Found the next block header after skipping: %s bytes forward in the file.\r\n", formatNumber(i) );
   4356                             lastBlockRead+=i; // advance to this location.

i = 10485757
temp = 0x000000010a000000
check = 0x000000010a9ffffd

The bad access is to 0x10ba00000, three bytes past check
((char*)check)[0] = 0
((char*)check)[1] = 0
((char*)check)[2] = 0
((char*)check)[3] = segfault


What version of the product are you using? On what operating system?

svn revision 55 running on OS X Mavericks


Please provide any additional information below.

I already fixed the file path separator.
The program takes less than a second to get to the point of the crash!