clostra / newnode Goto Github PK
View Code? Open in Web Editor NEWNewNode decentralized Content Distribution Network
Home Page: http://newnode.com
License: GNU General Public License v2.0
NewNode decentralized Content Distribution Network
Home Page: http://newnode.com
License: GNU General Public License v2.0
Customers use Android Studio Gradle-based build process.
Injector proxy needs to support uTP on the client side.
E/AndroidRuntime(10651): FATAL EXCEPTION: main
E/AndroidRuntime(10651): Process: com.example.webviewsample, PID: 10651
E/AndroidRuntime(10651): java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "__cmsg_nxthdr" referenced by "libdcdn.so"...
E/AndroidRuntime(10651): at java.lang.Runtime.loadLibrary(Runtime.java:364)
E/AndroidRuntime(10651): at java.lang.System.loadLibrary(System.java:526)
E/AndroidRuntime(10651): at com.clostra.dcdn.Dcdn.<clinit>(Dcdn.java:5)
E/AndroidRuntime(10651): at com.example.webviewsample.MainActivity.onCreate(MainActivity.java:48)
E/AndroidRuntime(10651): at android.app.Activity.performCreate(Activity.java:5231)
E/AndroidRuntime(10651): at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1087)
E/AndroidRuntime(10651): at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2148)
E/AndroidRuntime(10651): at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2233)
E/AndroidRuntime(10651): at android.app.ActivityThread.access$800(ActivityThread.java:135)
E/AndroidRuntime(10651): at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1196)
E/AndroidRuntime(10651): at android.os.Handler.dispatchMessage(Handler.java:102)
E/AndroidRuntime(10651): at android.os.Looper.loop(Looper.java:136)
E/AndroidRuntime(10651): at android.app.ActivityThread.main(ActivityThread.java:5001)
E/AndroidRuntime(10651): at java.lang.reflect.Method.invokeNative(Native Method)
E/AndroidRuntime(10651): at java.lang.reflect.Method.invoke(Method.java:515)
E/AndroidRuntime(10651): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
E/AndroidRuntime(10651): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
E/AndroidRuntime(10651): at dalvik.system.NativeStart.main(Native Method)
W/ActivityManager( 477): Force finishing activity com.example.webviewsample/.MainActivity
I also saw rand
missing.
Seems to me that KitKat is still quite common to be ignored, especially in Asia and Middle East.
In file included from dht.cpp:7:
In file included from libbtdht/src/dht.h:26:
libbtdht/btutils/src/utypes.h:40:9: fatal error: 'INT64_MAX' macro redefined [-Wmacro-redefined]
#define INT64_MAX 0x7fffffffffffffffLL
^
~/dcdn/android-toolchain-armv7-a/bin/../sysroot/usr/include/stdint.h:188:9: note: previous definition is here
#define INT64_MAX (INT64_C(9223372036854775807))
^
1 error generated.
(Also I can't test with build anymore as there is no easy way of cleaning the android built libraries).
After either the injector
or injector_helper
exits, there is a huge list of memory leaks reported from ASan
inside the libbtdht
library. This makes it very hard to spot memory leaks inside the dcdn
if we were to introduce some.
So far I've found two causes of this, first one being that blocks of memory allocated here are not freed in the destructor of the BlockAllocator struct (nor anywhere else).
The second problem is that the main object of type DhtImpl is never destroyed (Note: objects of type BlockAllocator
are also members of this class). We do try to destroy it here but reference count of that smart_ptr
has been incremented a few times inside libbtdht (e.g. here) and thus the destruction doesn't take place.
It is not clear to me whether this destruction can be forced cleanly and so we should investigate that further. If not, then we could try to suppress these errors as this problem only appears on app exit (or switch to another library).
https://docs.travis-ci.com/user/osx-ci-environment/
Right now, the Linux build has made master very hard to work with from Mac OS X.
Having a Mac OS X build on Travis will make this much easier.
To better understand what we'll need to do to get good test coverage, let's start with injector helper.
I'm having problems which can be resolve we had a build automation instead of home brewed ./build.sh. For example I get this:
Building dht.o...
In file included from dht.cpp:1:
In file included from /usr/include/sys/types.h:25:
/usr/include/features.h:180:3: fatal error: "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [-W#warnings]
# warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
^
So if I drop -D_BSD_SOURCE in the source it will goes away and then I'll get
Building dht.o...
In file included from dht.cpp:14:
./dht.h:5:10: fatal error: 'Block.h' file not found
#include <Block.h>
^~~~~~~~~
1 error generated.
this is my clang version
clang version 4.0.0 (tags/RELEASE_400/final)
I know I can hack my way out but probably these are problems that others will face sooner and later, and build automation system has been invented to exactly sort out these kind of problems without human intervention.
In Content-Location
, which we add.
Separate environments so the keys can be different in dev and in prod. Prod keys obviously should not be in the repo.
We already have -fsanitize=address
.
Should we use the output of the leak sanitizer?
Are there any ASAN_OPTIONS or LSAN_OPTIONS we should use?
Should we use -fsanitize-address-use-after-scope
?
We need a minimal app that shows how to integrate the SDK. This issue is just that.
In `examples/', create a very simple Android app that uses Android Studio Gradle-based build.
The example app could load and display the Bing image of the day of maybe of one of Flickr "interesting" images, whichever is more convenient. A button to load a new image of the same kind would be nice.
http://bbc.com
through injector helper, which goes to injector@inetic says:
[..B]oth injector and injector_helper no longer enter the error handling code. Additionally, the test_server only sends back HTTP OK header and no body, so injector doesn't enter the body handling code. I'll add that while working on something else as that's an easy fix.
Aborted in libnewnode.v1.5.1.so:244060
Fatal signal from native code: 6 (Aborted)
libnewnode.v1.5.1.so:244060 - bufferevent_free_checked
libnewnode.v1.5.1.so:305532 - ubev_bev_close
libnewnode.v1.5.1.so:307688 - ubev_write_cb
libnewnode.v1.5.1.so:400972 - bufferevent_writecb
libnewnode.v1.5.1.so:434864 - event_process_active_single_queue
libnewnode.v1.5.1.so:417536 - event_base_loop
libnewnode.v1.5.1.so:247968 - network_loop
libnewnode.v1.5.1.so:178732 - client_run
libnewnode.v1.5.1.so:178816 - client_thread
Created by Greg Hazel via Bugsnag
Instead of fetching net.dns, a bunch of Java code is required:
There's a stub in .travis.yml, but I haven't looked at what is required to get that to work. You might need something from me besides merging the pull request.
https://github.com/clostra/dcdn/blob/81dae45e4a1f53b945eafa406d8f898894e4eb9f/build.sh
The master
branch has injector but doesn't have the client code and the ghazel
branch builds the client.c file but doesn't build injector.c.
As mentioned in #45, I also got a build failure when running build.sh
on a pristine clone of the repo (git clone --recursive URL
) in a Debian Sid system:
In file included from dht.cpp:1:
In file included from /usr/include/x86_64-linux-gnu/sys/types.h:25:
/usr/include/features.h:148:3: fatal error: "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [-W#warnings]
# warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
^
1 error generated.
Simply replacing -D_BSD_SOURCE
by -D_DEFAULT_SOURCE
in build.sh
makes the issue go away.
I guess that newer versions of GNU libc (mine is 2.24) are more picky with obsolete definitions, so upgrading to the new definition should be an easy and safe fix.
Even when everything is set up correctly, there is a good chance that the client will respond with failure to the browser with the "Bad Gateway (502)" HTTP status code.
This is due to the fact that each one of us (testers) uses a different set of public/private signing keys yet we all register at the same place in the DHT. Thus when the client contacts a random injector from the DHT it'll probably pick one with an incompatible key.
Given that anyone can fork this repository and join the injector swarm with a different set of signing keys, this is an easy way to deny service to clients.
As per the specification, a client (named peer in the spec) requesting and successfully fetching a URL from an injector or injector_helper, should result in that URL content being shared on the BitTorrent network.
However, testing this behavior results with failure using the current HEAD of the ghazel branch.
In particular, this is what's being tested:
C1
, wait for it to find the injector, thenbbc.com
using C1
as a proxy, thenC2
making sure it won't find any injector nor injector helper through the DHT. Thenbbc.com
using C2
as a proxy.It is expected that the step number 5 will succeed, because C2
should download the content from BitTorrent. Instead C2
return the Bad Gateway
HTTP error code.
I would like to point out that identifiers like “__BEV_SPLICE_H__
” and “__NETWORK_H__
” do not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?
Some tests that check for aggregate ability of the system to perform its intended function.
http://bbc.com/?base64-of-128bits-of-random
, perhaps?)We have not customized it in any way and it's one of the things that slows down Travis the most.
Perhaps we can just install binaries?
In particular, if coverage drops 0.01%, Codecov will fail the build. That's a bit too much.
To reproduce block the origin using a firewall rule such as
sudo iptables -t filter -I FORWARD -d wadiya.wikia.com -p tcp -j REJECT
Request the orign you'll get a
Bad Gateway
page.
I wonder how the dcdn decides if the origin is blocked.
Would you like to add more error handling for return values from functions like the following?
@inetic says:
[..W]hen the origin is down, it takes ~40 seconds for the test to finish. I.e. when the origin is down it takes ~40 seconds between the injector_helper creating a request and having the request callback executed.
Not sure whether this is a bug in the injector helper on in the injector.
It looks to me like we could add it to the Travis build. Is there a better way in terms of GitHub integration—a separate existing service, maybe with a bot?
Can be reproduced when AddressSanitizer env options are set like this:
export ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-4.0/bin/llvm-symbolizer
export MSAN_SYMBOLIZER_PATH=/usr/lib/llvm-4.0/bin/llvm-symbolizer
export ASAN_OPTIONS=strict_string_checks=1:detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1
=================================================================
==24533==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6250000027ee at pc 0x000000449d74 bp 0x7ffeaa98bfa0 sp 0x7ffeaa98b750
READ of size 5 at 0x6250000027ee thread T0
#0 0x449d73 in StrtolFixAndCheck(void*, char const*, char**, char*, int) (/home/peter/work/dcdn-ghazel/injector+0x449d73)
#1 0x44a2a1 in __interceptor_strtoll (/home/peter/work/dcdn-ghazel/injector+0x44a2a1)
#2 0x58054f in BencEntity::ParseNum(unsigned char const*) /home/peter/work/dcdn-ghazel/libbtdht/btutils/src/bencoding.cpp:544:11
#3 0x580624 in BencEntity::SetParsed(IBencParser::PARSE_T, unsigned char const*, unsigned long, BencEntity::AllocRegime*) /home/peter/work/dcdn-ghazel/libbtdht/btutils/src/bencoding.cpp:558:5
#4 0x580d7b in BencodedDict::ResumeDict(IBencParser*, BencEntity**, BencEntity::AllocRegime*) /home/peter/work/dcdn-ghazel/libbtdht/btutils/src/bencoding.cpp:703:11
#5 0x5810ca in BencEntity::DoParse(BencEntity&, IBencParser*, BencEntity::AllocRegime*) /home/peter/work/dcdn-ghazel/libbtdht/btutils/src/bencoding.cpp:811:39
#6 0x580472 in BencEntity::Parse(unsigned char const*, BencEntity&, unsigned char const*) /home/peter/work/dcdn-ghazel/libbtdht/btutils/src/bencoding.cpp:759:7
#7 0x514ff1 in load_dht_state(BencEntity*) /home/peter/work/dcdn-ghazel/dht.cpp:71:5
#8 0x5496a1 in DhtImpl::LoadState() /home/peter/work/dcdn-ghazel/libbtdht/src/DhtImpl.cpp:3827:2
#9 0x549501 in DhtImpl::Initialize(UDPSocketInterface*, UDPSocketInterface*) /home/peter/work/dcdn-ghazel/libbtdht/src/DhtImpl.cpp:359:2
#10 0x548ce9 in DhtImpl::DhtImpl(UDPSocketInterface*, UDPSocketInterface*, void ()(unsigned char const, int), void ()(BencEntity), ExternalIPCounter*) /home/peter/work/dcdn-ghazel/libbtdht/src/Dht
Impl.cpp:246:2
#11 0x545c45 in create_dht(UDPSocketInterface*, UDPSocketInterface*, void ()(unsigned char const, int), void ()(BencEntity), ExternalIPCounter*) /home/peter/work/dcdn-ghazel/libbtdht/src/dht.cpp:3
0:29
#12 0x515a61 in dht_setup /home/peter/work/dcdn-ghazel/dht.cpp:125:15
#13 0x52580a in network_setup /home/peter/work/dcdn-ghazel/network.c:163:14
#14 0x5231bc in main /home/peter/work/dcdn-ghazel/injector.c:388:18
#15 0x7f28255f182f in __libc_start_main /build/glibc-bfm8X4/glibc-2.23/csu/../csu/libc-start.c:291
#16 0x41cec8 in _start (/home/peter/work/dcdn-ghazel/injector+0x41cec8)
0x6250000027ee is located 0 bytes to the right of 9966-byte region [0x625000000100,0x6250000027ee)
allocated by thread T0 here:
#0 0x4d5548 in __interceptor_malloc (/home/peter/work/dcdn-ghazel/injector+0x4d5548)
#1 0x514faf in load_dht_state(BencEntity*) /home/peter/work/dcdn-ghazel/dht.cpp:67:24
#2 0x5496a1 in DhtImpl::LoadState() /home/peter/work/dcdn-ghazel/libbtdht/src/DhtImpl.cpp:3827:2
#3 0x549501 in DhtImpl::Initialize(UDPSocketInterface*, UDPSocketInterface*) /home/peter/work/dcdn-ghazel/libbtdht/src/DhtImpl.cpp:359:2
#4 0x548ce9 in DhtImpl::DhtImpl(UDPSocketInterface*, UDPSocketInterface*, void ()(unsigned char const, int), void ()(BencEntity), ExternalIPCounter*) /home/peter/work/dcdn-ghazel/libbtdht/src/DhtI
mpl.cpp:246:2
#5 0x545c45 in create_dht(UDPSocketInterface*, UDPSocketInterface*, void ()(unsigned char const, int), void ()(BencEntity), ExternalIPCounter*) /home/peter/work/dcdn-ghazel/libbtdht/src/dht.cpp:30
:29
#6 0x515a61 in dht_setup /home/peter/work/dcdn-ghazel/dht.cpp:125:15
#7 0x52580a in network_setup /home/peter/work/dcdn-ghazel/network.c:163:14
#8 0x5231bc in main /home/peter/work/dcdn-ghazel/injector.c:388:18
#9 0x7f28255f182f in __libc_start_main /build/glibc-bfm8X4/glibc-2.23/csu/../csu/libc-start.c:291
SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/peter/work/dcdn-ghazel/injector+0x449d73) in StrtolFixAndCheck(void*, char const*, char**, char*, int)
Shadow bytes around the buggy address:
0x0c4a7fff84a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff84b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff84c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff84d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff84e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a7fff84f0: 00 00 00 00 00 00 00 00 00 00 00 00 00[06]fa fa
0x0c4a7fff8500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff8510: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff8520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff8530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff8540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==24533==ABORTING
Injector proxy does some work on startup/resume/disconnect and some on each request. Establishing a connection to an injector should be done on startup/resume/disconnect. Indeed, the success of that is what determines whether this even is a functional injector proxy and should advertise itself as such.
https://travis-ci.org/clostra/dcdn/jobs/240955416
Any idea what's going on? Tests pass fine on the Mac, both locally and in Travis.
Rare but does occur: bugsnag breadcrumbs are accessed from both the Java thread and the NewNode thread.
Segmentation fault in libnewnode.so:298140
Fatal signal from native code: 11 (Segmentation fault)
libnewnode.so:298140 - (null)
libnewnode.so:297920 - json_value_free
libnewnode.so:322700 - bugsnag_event_add_breadcrumb
libnewnode.so:158128 - bugsnag_add_breadcrumb
libnewnode.so:147336 - bugsnag_leave_breadcrumb_log
libnewnode.so:278112 - bugsnag_log
libnewnode.so:210900 - http_request_cb
libnewnode.so:572644 - (null)
Created by Greg Hazel via Bugsnag
This one is similar to the "40 seconds to fail" issue, but while before it took too long for injector_helper
to realize the origin
was down, now it takes it too long to realize the injector
is down.
Here is a failing test showing the problem.
@inetic Can you help with the Linux build?
I got it to dealing with libsodium; probably should at this point use the submodule, rather than an installed library.
Each build attempt takes me a few minutes because I don't have Ubuntu. Can you take a look and transfer whatever you're doing locally to make it build to .travis.yml
?
You can view the Travis CI build status by clicking on the badge in README, which takes to https://travis-ci.org/clostra/dcdn
Currently, testing dCDN in machines connected to the Internet is hampered by the fact that there is just one dCDN network, since the base swarms are hardwired into the code (constants.h
). The potential presence of other participants makes it difficult to e.g. ensure that just a given set of clients or injectors are connected to experiment the reactions of the system to injectors and clients going up or down etc.
One simple way to create different dCDN networks for testing would be to support having an optional salt string shared by all participants in the network which would be used in all swarm SHA1 operations (injector swarm, injector proxy swarm, URL swarm). The salt may be received as an optional command line argument to injector
and client
, empty if missing.
Of course this kind of "shared secret" feature may not be used for production networks, but for testing it should be very useful and not difficult to add.
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.