Giter Club home page Giter Club logo

vincenzopalazzo / spycblock Goto Github PK

View Code? Open in Web Editor NEW
19.0 4.0 6.0 85.42 MB

:octocat: :hammer: A simple Bitcoin parser to parse the raw *.dat files, and decode them into differents format :microscope: :octocat:

Home Page: https://vincenzopalazzo.github.io/SpyCblock/

License: Apache License 2.0

CMake 2.21% C 2.53% C++ 94.20% Shell 1.06%
bitcoin bitcoin-core bitcoin-parser blockchain-parser blockchain-parsing blk blk-parsing bitcoin-development

spycblock's Introduction

GitHub Matrix

🎓 SpyCBlock 🔬

SpyCBlock is an accademics software that analyses the blk file of Bitcoin blockchain Mainet. This is an explerimental version and it wants to demostrate that it is possible to work with only file blk.

SpyCBlock is a simple parser bitcoin blk file, with this parser it is possible to do some serialization, like:

  • Serialization Transaction Graph (:heavy_check_mark:);
  • Serialization complete Blockchain to JSON (:heavy_check_mark:);
  • Serialization Address Graph (:warning:).

To reduce the space of serialization Transaction graph, it is possible to use the library ZLib to compress the information; with this code it is possible to uncompress the data.

void decompressFileWithZLib()
{
    std::string filename = "YOUR_PATH/blkXXXXX.dat";
    gzFile inFileZ = gzopen(filename.c_str(), "rb");

    if(inFileZ == NULL){
        printf("Error: Failed to gzopen %s\n", filename.c_str());
        exit(0);
    }
    unsigned char unzipBuffer[8192];
    unsigned int unzippedBytes;
    std::vector<unsigned char> unzippedData;
    while (true)
    {
        unzippedBytes = gzread(inFileZ, unzipBuffer, 8192);
        if(unzippedBytes > 0) {
            unzippedData.insert(unzippedData.end(), unzipBuffer, unzipBuffer + unzippedBytes);
        }else{
            break;
        }
    }

    for(auto &character : unzippedData)
    {
        if(character == '.'){
            std::cout << "\n";
        }else{
            std::cout << character;
        }
    }
}

It Is possible to execute the parser with the library OpenMP for execution data with multi-core and this is a simple velocity benchmark.

PS: at the moment the parser uses all core of the CPU is the multicore propriety is enabled.

benchmark_image

The graph of transactions form is describe here and this is an example to visualise this with Web app, this is a simple demo.

The screenshot of transaction graph

transaction_graph

The screenshot of address graph subdivise to class (with luvain algorithm)

address_graph

⭐ JSON version Bitcoin Blockchain :octocat:

Having the version of Bitcoin blockchain to JSON is powerful because it is possible to work on the JSON format, With this is possible to create easy analysis on Bitcoin network.

An example: I created a simple analysis with AnalytcsPyBlock to get informations from type of script used in the Bitcoin blockchain; this is the result and here is avaible the web version.

analisis_script

💥 Build SpyCBlock :octocat:

Follow this guide

🏃 Configure SpyCBlock :octocat:

Follow this guide

💬 IRC channels :octocat:

  • IRC dev channel: #spycblock-dev :neckbeard:

PS: The IRC channel is deprecated in favor of the matrix channel.

Cite

You can use the research work in any way you want but please cite us!

@software{Palazzo_A_simple_Bitcoin_2021,
    author = {Palazzo, Vincenzo and Carlo, Sartiani},
    month = {12},
    title = {{A simple Bitcoin parser to parse the raw data files, and decode them into differents format}},
    url = {https://github.com/vincenzopalazzo/SpyCBlock},
    version = {1.0.0},
    year = {2021}
}

🍺 Buy me a beer 🍺

🍺 39juEyTGSNfD9JHCZdQKAjkzC3yiudce7b 🍺

spycblock's People

Contributors

vincenzopalazzo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

spycblock's Issues

May change the transaction seriealize format

In October 2020 was merged into master a couple of good things described inside this BIP341, this BIP with be available on mainet since the Bitcoin 0.21.0.

TODO

  • Reproduce the crash during the block parser in SpyCBlock, and update the tests

Logger configuration feature

Glog doesn't have the logger level 3.

Maybe I can use some string value to disable log, and not use the log level 3.

Compatibility Test Traking

The object of this issue to tracking the support of the actual network with the Bitcoin Core software.

List of compatibility test:

  • Released in December 12 2019:

    • Bitcoin Core 0.18.0:
      • JSON Serializzation ✔️
      • Serialization graph transaction ✔️
      • Serializzation graph addresss with bitcoin core help ✔️
  • Test in Agust 21 2020:

    • Bitcoin Core 0.20.1 (height blockchain 644 606):
      • JSON Serializzation ✔️
      • Serialization graph transaction ✔️
      • Serializzation graph addresss with bitcoin core help ‼️ (look 🔶)

P.S: Update (October 2020) the transaction format can change in Bitcoin Core 0.21.0 and the motivations are described inside this issue

Appendix

🔶 Bitcoin core has deprecated the P2PK script inside the version 0.18.1, this means that is not longer exists the possibility to receive the address from bitcoin core RPC interface. The alternative is to implement a parser of bitcoin script with code, a draft code is here

✔️ Deserialization completed.
🔁 Waiting for the test.
‼️ Compatibility error.

Progress bar issue

This could be a good first issue for the people that want to start to understand the code.

The progress bars don't work as an aspect. It has the following problems:

  • Displayed with logger level under 3
  • With parallel execution not running as expected, the time and the loading is wrong

Error during compilation

I'am trying to compile the project on my Kali Linux.

1- I get the repository with:
git clone https://github.com/vincenzopalazzo/SpyCblock
2- I install dependences:
apt-get update && sudo apt-get install build-essential -y
apt-get install zlib1g-dev -y
apt-get install libncurses-dev -y
apt install cmake -y
3- I try to compile from the script:
./compile.sh

This is the error that I get:

[ 53%] Built target gtest
[ 55%] Built target gtest_main
[ 56%] Building CXX object test/unittest/CMakeFiles/unittest.dir/pointertest.cpp.o
In file included from /home/kali/Documents/SpyCblock/installer/bitcoin-api-cpp/rapidjson/test/unittest/pointertest.cpp:16:
In member function ‘rapidjson::GenericPointer<ValueType, Allocator>::Ch* rapidjson::GenericPointer<ValueType, Allocator>::CopyFromRaw(const rapidjson::GenericPointer<ValueType, Allocator>&, std::size_t, std::size_t) [with ValueType = rapidjson::GenericValue<rapidjson::UTF8<> >; Allocator = rapidjson::CrtAllocator]’,
    inlined from ‘rapidjson::GenericPointer<ValueType, Allocator> rapidjson::GenericPointer<ValueType, Allocator>::Append(const Token&, Allocator*) const [with ValueType = rapidjson::GenericValue<rapidjson::UTF8<> >; Allocator = rapidjson::CrtAllocator]’ at /home/kali/Documents/SpyCblock/installer/bitcoin-api-cpp/rapidjson/include/rapidjson/pointer.h:238:30,
    inlined from ‘rapidjson::GenericPointer<ValueType, Allocator> rapidjson::GenericPointer<ValueType, Allocator>::Append(const Ch*, rapidjson::SizeType, Allocator*) const [with ValueType = rapidjson::GenericValue<rapidjson::UTF8<> >; Allocator = rapidjson::CrtAllocator]’ at /home/kali/Documents/SpyCblock/installer/bitcoin-api-cpp/rapidjson/include/rapidjson/pointer.h:255:39,
    inlined from ‘typename rapidjson::internal::DisableIf<typename rapidjson::internal::RemoveSfinaeTag<rapidjson::internal::SfinaeTag& (*)(rapidjson::internal::NotExpr<rapidjson::internal::IsSame<typename rapidjson::internal::RemoveConst<T>::Type, typename Encoding::Ch> >)>::Type, typename rapidjson::internal::RemoveSfinaeTag<rapidjson::internal::SfinaeTag& (*)(rapidjson::GenericPointer<ValueType, Allocator>)>::Type>::Type rapidjson::GenericPointer<ValueType, Allocator>::Append(T*, Allocator*) const [with T = const char; ValueType = rapidjson::GenericValue<rapidjson::UTF8<> >; Allocator = rapidjson::CrtAllocator]’ at /home/kali/Documents/SpyCblock/installer/bitcoin-api-cpp/rapidjson/include/rapidjson/pointer.h:267:62,
    inlined from ‘virtual void Pointer_Append_Test::TestBody()’ at /home/kali/Documents/SpyCblock/installer/bitcoin-api-cpp/rapidjson/test/unittest/pointertest.cpp:574:21:
/home/kali/Documents/SpyCblock/installer/bitcoin-api-cpp/rapidjson/include/rapidjson/pointer.h:887:24: error: ‘void* memcpy(void*, const void*, size_t)’ offset [0, 23] is out of the bounds [0, 0] [-Werror=array-bounds=]
  887 |             std::memcpy(tokens_, rhs.tokens_, rhs.tokenCount_ * sizeof(Token));
      |             ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [test/unittest/CMakeFiles/unittest.dir/build.make:272: test/unittest/CMakeFiles/unittest.dir/pointertest.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1683: test/unittest/CMakeFiles/unittest.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
rm: cannot remove 'rapidjson': No such file or directory
The dependecis are OK
Submodule path 'include/spycblockrpc': checked out 'a2120f0cc50f8ccbf58a939a68d9d5ef47b802fd'
fatal: No url found for submodule path 'include/spycblockrpc/lib/cpp-properties' in .gitmodules
fatal: Failed to recurse into submodule path 'include/spycblockrpc'
Already up to date.
Submodule path 'include/bitcoin-cryptography-library': merged in '68061ea1167548aca189ff240895ddd1a50b3d9f'
Already up to date.
Submodule path 'include/cpp-properties': merged in '2ad95c2d639f7114ae9677bdb1d516abe4fb1709'
Already up to date.
Submodule path 'include/progressbar': merged in '05d61b642500a3dbb00237e8e33087917c420a4e'
Updating a2120f0..82ee042
Fast-forward
 .gitmodules                            |   3 +-
 CMakeLists.txt.user.4.10-pre1          | 451 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 src/core/graph/WrapperInformations.cpp |   1 +
 3 files changed, 454 insertions(+), 1 deletion(-)
 create mode 100644 CMakeLists.txt.user.4.10-pre1
Submodule path 'include/spycblockrpc': merged in '82ee042ff768c9c0d65121ba098bc6fd2a06cffa'
Already up to date.
Submodule path 'include/spycblockrpc/lib/cpp-properties': merged in '2ad95c2d639f7114ae9677bdb1d516abe4fb1709'
Submodule path 'lib/cpp-properties': checked out '2ad95c2d639f7114ae9677bdb1d516abe4fb1709'
mkdir: cannot create directory ‘build’: File exists
CMake Error at CMakeLists.txt:10 (find_library):
  Could not find bitcoinapi using the following names: 0.3                                                                                                                                                                                 
                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                           
-- Configuring incomplete, errors occurred!
make: *** No targets specified and no makefile found.  Stop.

C++ Improvement and code library improvement

The actual version of the project it is only an alpha version and contains many many leaks in terms of code style, code managing, and so on!

This issue contains a couple of points that SpyCBlock must be considered to be considered a real alternative in the index bitcoin blockchain.

  • Move C++ version to 17 and make smarter the directory management.
  • Improve C++ Style and make the code style under the Google line guide with the help of clang-format 🚀 (Developing in this branch).
  • Introduce a command line parser, a good choice can be the following project
  • Introduce jemalloc to make the malloc operation faster.
  • Support of chainstate parsing 🚀 (Develiping in this branch)
  • Support Testnet blk parsing 📌.
  • Improve Readme file and remove the fancy (:trollface:) readme of version 1.0.0
  • Improve the benchmarck and also improve the benchmark description with the ram consumption 📈.
  • Remove the actual progress bar and change with a C++ progressbar like this (Maybe)

If you have some suggestions or comment feel free to make a comment below or write me an email you can found my email around my github profile.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.