Giter Club home page Giter Club logo

bitcoinj's Introduction

Github Build Status Travis Build Status GitLab Build Status Coverage Status

Visit our IRC channel

Welcome to bitcoinj

The bitcoinj library is a Java implementation of the Bitcoin protocol, which allows it to maintain a wallet and send/receive transactions without needing a local copy of Bitcoin Core. It comes with full documentation and some example apps showing how to use it.

Technologies

  • Java 7+ and Gradle 4.4+ for the core module
  • Java 8+ and Gradle 4.4 - Gradle 6.9 for tools and examples
  • Java 11+ and Gradle 4.10 - Gradle 6.9 for the JavaFX-based wallettemplate
  • Gradle - for building the project
  • Google Protocol Buffers - for use with serialization and hardware communications

Note: Building with Gradle 7 is currently unsupported. See Issue #2112 and Issue #2119.

Getting started

To get started, it is best to have the latest JDK and Gradle installed. The HEAD of the master branch contains the latest development code and various production releases are provided on feature branches.

Building from the command line

Official builds are currently using with JDK 8, even though the core module is compatible with JDK 7 and later.

To perform a full build (including JavaDocs and unit/integration tests) use JDK 8+

gradle clean build

If you are running JDK 11 or later and Gradle 4.10 or later, the build will automatically include the JavaFX-based wallettemplate module. The outputs are under the build directory.

To perform a full build without unit/integration tests use:

gradle clean assemble

Building from an IDE

Alternatively, just import the project using your IDE. IntelliJ has Gradle integration built-in and has a free Community Edition. Simply use File | New | Project from Existing Sources and locate the build.gradle in the root of the cloned project source tree.

Building and Using the Wallet Tool

The bitcoinj tools subproject includes a command-line Wallet Tool (wallet-tool) that can be used to create and manage bitcoinj-based wallets (both the HD keychain and SPV blockchain state.) Using wallet-tool on Bitcoin's test net is a great way to learn about Bitcoin and bitcoinj.

To build an executable shell script that runs the command-line Wallet Tool, use:

gradle bitcoinj-tools:installDist

You can now run the wallet-tool without parameters to get help on its operation:

./tools/build/install/wallet-tool/bin/wallet-tool

To create a test net wallet file in ~/bitcoinj/bitcoinj-test.wallet, you would use:

mkdir ~/bitcoinj
./tools/build/install/wallet-tool/bin/wallet-tool --net=TEST --wallet=$HOME/bitcoinj/bitcoinj-test.wallet create

To sync the newly created wallet in ~/bitcoinj/bitcoinj-test.wallet with the test net, you would use:

./tools/build/install/wallet-tool/bin/wallet-tool --net=TEST --wallet=$HOME/bitcoinj/bitcoinj-test.wallet sync

To dump the state of the wallet in ~/bitcoinj/bitcoinj-test.wallet with the test net, you would use:

./tools/build/install/wallet-tool/bin/wallet-tool --net=TEST --wallet=$HOME/bitcoinj/bitcoinj-test.wallet dump

Note: These instructions are for macOS/Linux, for Windows use the tools/build/install/wallet-tool/bin/wallet-tool.bat batch file with the equivalent Windows command-line commands and options.

Example applications

These are found in the examples module.

Where next?

Now you are ready to follow the tutorial.

Testing a SNAPSHOT build

Building apps with official releases of bitcoinj is covered in the tutorial.

If you want to develop or test your app with a Jitpack-powered build of the latest master or release-0.15 branch of bitcoinj follow the dynamically-generated instructions for that branch by following the correct link.

bitcoinj's People

Contributors

amichair avatar c-otto avatar cyberzac avatar devrandom avatar devrandom1 avatar erasmospunk avatar haraldh avatar hashengineering avatar jarlfr avatar jim618 avatar kirill-vlasov avatar kmels avatar kparmar1 avatar ksedgwic avatar manfredkarrer avatar matthewleon avatar mikehearn avatar mjjbell avatar mruddy avatar msgilligan avatar natzei avatar oscarguindzberg avatar peterdettman avatar ragmondo avatar ripcurlx avatar schildbach avatar thebluematt avatar troggy avatar w-shackleton avatar wlk avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bitcoinj's Issues

new branch for 0.15.3

I have rebased the bisq specific commits on to 0.15.3 https://github.com/bodymindarts/bitcoinj/tree/bisq_0.15.3 and would like to contribute it back to this repo.

As I do not have push access I cannot create a new branch and cannot open a PR against a branch that doesn't exist.

Perhaps someone with access could do the following:

git clone https://github.com/bitcoinj/bitcoinj.git && cd bitcoinj
git remote add bisq https://github.com/bisq-network/bitcoinj.git
git checkout v0.15.3
git checkout -b bisq_0.15.3
git push -u bisq bisq_0.15.3

Then I can open a PR against that branch to add the bisq specific commits.

bisq's bitcoinj status

In progress

  • Merge to bisq project PR for using the latest bitcoinj 0.14.7 version with improvements on connection handling based on audit (useless if we migrate to bitcoinj 0.15)
  • Migration to bitcoinj 0.15
  • Deprecated transaction confidence / broadcasting problem research
  • PeerGroup.handlePeerDeath() + BlockingClient.socket.close() called twice
    • Upstream bug bitcoinj#1765
    • Someone has to look at the bug again and confirm whether the bug described matches reality and then implement a solution.
    • I guess this is not causing big problems for Bisq.
    • The analysis is based on bitcoinj 0.14 but I guess it is still valid for bitcoinj 0.15.
  • "Restore from seed" on "wallet template" upstream bitcoinj project is broken.
    • Upstream bug: bitcoinj#1704
    • Waiting for someone to fix the bug.
    • It is important for Bisq that "restore from seed" works on "wallet template" because it is a benchmark on how bisq "restore from seed" should work.
  • Path issue with serialized wallets migrated from pre-0.15 to 0.15
    • Upstream bug bitcoinj#1812
    • A workaround was implemented on bisq by using BisqKeyChainFactory and isqKeyChainGroupStructure, but a generic upstream solution would be better.

TODO (i.e. not started yet)

Bisq's bitcoinj "addr" message support audit

  • Commits to audit
  • How it works now
    • PeerGroup: When "addr" msg is received, add a new peer to the "inactives" collection.
    • PeerGroup property addPeersFromAddressMessage can be used to disable the feature.
    • bisq/bitcoinj node never sends “getaddr” msgs.
    • bisq/bitcoinj node receives from time to time unsolicited “addr” msgs from peers.
    • bisq sets addPeersFromAddressMessage to false (disables the feature) if preferences.getBitcoinNodes() is not empty.
  • upstream bitcoinj support for addr/getaddr messages
  • Bitcoin core support: https://en.bitcoin.it/wiki/Satoshi_Client_Node_Discovery
  • Comments by Dan (commit author)
    • I modified bitcoinj to listen to the addr p2p messages and add nodes dynamically. I was rather surprised mainline bitcoinj didn't already do that since it is standard bitcoind functionality.
    • This was in context of enabling bitcoinj p2p traffic to work over Tor, including DNS lookups which was a particularly tricky aspect.
    • btc nodes come and go, so hard-coded seeds become unavailable over time. thus unreliable.
      • Oscar comment: it's true, but same non critical issue is present on every app that uses bitcoinj.
    • DNS provided nodes are a tiny subset of total bitcoin p2p network, and using only them is a point of centralization. dns maintainers become gatekeepers.
      • Oscar comment: but same non critical issue is present on every app that uses bitcoinj.
    • DNS over Tor is slow and some record-sets from bitcoin DNS servers are too large, which result in empty result set. (if memory serves, sipa's dns node was one such.) In prior testing, we would sometimes have issues with few connections being made to p2p network, and then it never corrects itself back up to the target 8 or 10. Using the addr message fixed that issue as I recall.
      • Oscar comment: Dan has a point here. Since Bisq uses by default the curated btc nodes harcoded in the bisq code instead of dns discovery, this is not as critical as it could be.
    • Bitcoin DNS servers do not provide .onion addresses. So addr messages are best way to become informed of nodes running behind Tor.
      • Oscar comment: Why is important to connect to btc nodes running behind tor?.
    • It is a way to bootstrap p2p connections from a local peer, eg bitcoind on same machine or subnet, even if DNS is being flaky/slow over Tor.
      • Oscar comment: If a user is connected to a local bitcoind, she probably does not want to connect to any other bitcoin node for privacy/trust issues.
    • It's a part of the bitcoin p2p protocol. Not implementing it in bitcoinj seems incomplete at best.
      • Oscar comment: agree.
    • It was not hard to do. low hanging fruit.
      • Oscar comment: See "Implementation problems/limitations" section.
  • Implementation problems/limitations.
    • If a user wants to connect just to the Bisq curated node list in BtcNodes.java for privacy issues, this mechanism might have her connected to other nodes. This could probably be solved by improving when PeerGroup.setAddPeersFromAddressMessage() is set to true.
    • getaddr messages are never sent, so bitcoinj depends on the unsolicited "addr" msgs peers send from time to time.
    • An attacker could send a million "addr" msgs with fake peers causing a DoS on the bitcoinj node.
    • Received addresses are not persisted.
    • Full support of peer discovery using getaddr/addr msgs might be difficult to implement. See upstream bitcoinj attempts to do it detailed above.
  • Actions:
    • Disable "add peers from addr message" custom bitcoinj feature when Bisq curated node list in BtcNodes.java is used: bisq-network/bisq#2591
    • Add a link to this audit in bitcoinj source code: #29

Merge upstream support for v2 transactions

Based on log analysis, we believe that bitcoinj's lack of support for v2 Bitcoin transactions [1] may have led to some of the timeout issues we've seen recently.

Apparently, support has been implemented, but has not yet been officially released in the canonical bitcoinj project.

  • Merge this support into our own fork
  • Upgrade to it in bisq-network/exchange
  • Test it
  • Tag v0.14.5.1 of bisq-network/bitcoinj
  • Upgrade to v0.14.5.1 of bisq-network/bitcoinj in v0.6.3 of bisq-network/exchange

[1] https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki

jdk8 vs jdk 10

The problem

bisq has to be built using jdk 10. bitcoinj has to be built using jdk 8.

Details

bitcoinj 0.14.4+ uses sun.misc.Cleaner.
sun.* packages should not be used directly as a rule of thumb.
bitcoinj uses them to fix some windows specific issue.
Using sun.* seemed to be the lesser evil.
The thing is sun.misc.Cleaner was removed in jdk 9.
bitcoinj does not compile if you use jdk 9, 10 or 11.

The problem was discussed on several bitcoinj mailing list msgs and github issues. The most relevant discussion happened here: bitcoinj#1477

Andreas Schildbach eventually removed the windows specific hack, which eliminates the need to use sun.misc.Cleaner (he thinks the latest versions of windows may not reproduce the bug)
Fix:
bitcoinj@1700ec0
Test:
bitcoinj@dbc4c35
Andreas Schildbach did the change on bitcoinj master (not the bitcoinj fork we use)
So, bitcoinj master can be built using jdk 8, 9, 10 or 11.
Nobody confirmed yet that without the hack, bitcoinj does not reproduce the bug on windows.
Even if someones confirm she is running bitcoinj on windows without problems, that does not mean it will run on every windows environment.

The feature that has the windows problem is marked to be refactored in bitcoinj
https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/store/SPVBlockStore.java#L33

How to solve it

Current solution

  • Devs install both jdk8 and jdk10. Tell intellij to use jdk8 for bitcoinj and jdk10 for bisq. Use a bash script to switch jdk versions when the dev needs to run maven/gradle on the command line.
    • Pros
      • Oscar has the "hack" to switch jdks set up, so it is not a big issue for him to use this "hack".
    • Cons
      • Other devs coding both bisq and bitcoinj need to be aware of this hack and adapt it to their OS if they don't run on Mac.

Other possible solutions

  • cherry pick bitcoinj@1700ec0 to bisq’s bitcoinj fork.
    • Pros
      • Easy to implement.
      • Devs coding both bisq and bitcoinj don't have to switch jdks - jdk 10 is ok for both projects.
    • Cons
      • Risk of bisq users having problems when running on windows (Risk could be partially reduced if we test on Windows).
  • Devs cherry pick bitcoinj@1700ec0 fix locally. Devs use git stash and other git hacks to avoid pushing that change.
    • Pros
      • Easy to implement.
      • Devs coding both bisq and bitcoinj don't have to switch jdks - jdk 10 is ok for both projects.
    • Cons
      • It is a nuisance to make sure some local changes are not included every time you commit.
  • Invest some time to refactor the bitcoinj feature that has the windows problem.
    • Pros
      • Solve the problem for good.
      • Devs coding both bisq and bitcoinj don't have to switch jdks - jdk 10 is ok for both projects.
    • Cons
      • May require a considerable amount of time to do the refactor.

Bisq's bitcoinj general changes audit

I reviewed the changes done by bisq's to bitcoinj.

Here are the conclusions:

Already reviewed

Fixes implemented on https://github.com/bisq-network/bitcoinj/tree/bisq_0.14.7

Still to be reviewed

Bisq’s bitcoinj tor/onion changes audit

Overview

  • Changes done on bisq's bitcoinj just support onion v2 addresses.
  • Most of the code needed to support tor/onion services is in https://github.com/bisq-network/bisq (out of scope of this audit)
  • Upstream tor support in bitcoinj was removed because it was not well maintained and caused problems and there were no volunteers to maintain it (bitcoinj#1314)
  • Tor support could be re-enabled upstream by contributing changes done on bisq's bitcoinj and on https://github.com/bisq-network/bisq
  • Tor support upstream could provide
    • Connect to peers running behind an onion address
    • Resolve DNS over tor
    • Connect via tor to clearnet peers
    • A mechanism to start a local tor client (orchid or another tor client)

Commit review

  • PeerGroup.addAddress(): don't increase max connections if address already exist 011fe40
  • Catch PeerDiscoveryException during getPeers call. avoids stack trace when all DNS lookups timeout 834f57a
  • PeerAddress hashCode() and equals(): add hostname and refactor 0988148
    • No changes suggested.
  • Changes to support connecting to .onion addresses 9f09a89

Failing tests

These tests fail on bisq's bitcoinj fork and need to be a) fixed or b) ignored and documented.

PeerTest

  • Fails because of oscarguindzberg@f76a9da
  • writeTarget.writeBytes() is used to check the connection was in fact closed. Implementation before bisq change used to throw an exception if connection was already closed.

DeterministicKeyChainTest

Bisq's bitcoinj connection handling changes audit

  • I found a couple of problems that are not related to changes done by bisq on bitcoinj. Here they are:
    • On Bisq project: Different timeouts should be used for socket connection and version message exchange.
      • Use different timeouts because version message exchange with the peer by definition happens after connecting the socket to the peer.
      • See bisq-network/bisq#2609
    • PeerGroup.handlePeerDeath() is called twice for the same peer.
      • The problem is the peer will be added twice to the inactives collection. See: inactives.offer(address)
      • I could reproduce this by running bisq when bitcoinj fails to connect to an onion peer
      • I see 2 “Peer died” on the log for the same peer, which means handlePeerDeath() has been invoked twice.
      • Action
  • Commit review
    • PeerGroup: Add check to not duplicate peers
      • Commit: 91ab651
      • Manfred’s comment: can be reproduced if u connect to provided bisq nodes and disconnect/reconnect several times. u will get duplicated nodes without that fix.
      • Oscar’s comments:
        • Changes on triggerConnectionsJob
          • addrToTry was just removed from inactives and is being re-added, so there is no chance for duplication here, so no need to check for isAlreadyAdded(). This change probably caused a positive effect because it prevents re-adding a peer if it was duplicated already.
          • Action: Revert change. See oscarguindzberg@b8d1534
        • isAlreadyAdded() method
          • I understand PeerAddress.equals() is not trustworthy since PeerAddress.equals() includes comparison of “time” field which could be different if informed by different peers in “addr” msgs.
          • Problems
            • It just compares by hostname, 2 peers could be run on the same host and different port.
            • PeerAddress could use addr (InetAddress) instead of hostname so the comparison will not work on this case.
          • Actions:
        • Change on addInactive()
          • backoffMap.containsKey(peerAddress) already checks for peer existence, there is no need to add an extra check to see if the peer is part of the inactives collection.
          • Action: Revert change. See oscarguindzberg@26889a8
    • PeerGroup: Limit number of Bitcoin network peers at re-connect after connection loss
      • Commit: 3f1382d
      • I could reproduce the bug running the code without the fix.
      • Not sure what is the reason for the bug, but the fix for sure does not cause any harm.
      • Action: keep the commit, just use wrapper methods, see oscarguindzberg@f36a1ea
    • PeerGroup: Add additional check isRunning() at handlePeerDeath to avoid errors at shutDown when using BlockingClient
      • Commit: f76d341
      • The idea is ok, but the implementation is wrong: When the PeerGroup is stopped, first peers are stopped and then vRunning is set to false, specially in BlockingClientManager case, so isRunning() is going to return true during shutdown.
      • Action: Revert change and use upstream solution instead: PeerGroup.stopAsync() set downloadPeer to null before channels are stoped. So, there is no download peer replacement on handlePeerDeath(). See oscarguindzberg@630662f
    • Do not close socket if it is already closed. (Should be called: Do not write to socket if it is already closed)
      • Commit: b90bce9
        • The “if” to not write the socket if the connection is ok. the catch code kind of duplicates the “if”.
        • Action: Remove the special catch, add an “else” statement printing a warning “writing to a closed socket”. See oscarguindzberg@4baf363
  • Extra
    • Testing Bisq using all the changes suggested on this audit, I still find bitcoinj errors on the log when the Bisq node is being shut down. I think the problem is the Tor node is shutdown before bitcoinj is told to shutdown, then connection to btc peers fail.
  • PR containing the fixes: #31

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.