oftc / toripscanner Goto Github PK
View Code? Open in Web Editor NEWPeriodically scan Tor exits for what IPs they use and produce the results in a plain text file
Periodically scan Tor exits for what IPs they use and produce the results in a plain text file
Apparently the exit policy can indicate accepting ipv6 traffic, but the exit doesn't actually support ipv6.
Be smarter about when to try ipv6.
Tons of exits don't support it, so even with a relatively short 15 second timeout, we're looking at up to 4 hours to scan the entire network. Before closing #1, we were at 90 minutes.
For example:
[2021-07-29 16:36:51,376] [INFO] [MainThread] [scan] [[email protected]:214] Measuring 1DBACC31486FC670FBD403FAE877342EC696D598. 707 relays remain
[2021-07-29 16:36:51,378] [DEBUG] [MainThread] [test_utils] [get_good_relays@test_utils.py:154] 7 good-relays seem to be running.
[2021-07-29 16:36:52,621] [DEBUG] [MainThread] [scan] [[email protected]:154] Will measure 1DBACC31486FC670FBD403FAE877342EC696D598 on 8 [Piratenpartei00 (1CD48F4E) -> hviv128 (1DBACC31)]
[2021-07-29 16:36:52,633] [DEBUG] [Event notifier] [test_utils] [closure_stream_event_listener@test_utils.py:166] Attaching stream 16 to circ 8 [Piratenpartei00 (1CD48F4E) -> hviv128 (1DBACC31)]
[2021-07-29 16:36:52,633] [WARNING] [Event notifier] [test_utils] [closure_stream_event_listener@test_utils.py:171] Couldn't attach stream to circ 8: Connection is not managed by controller.
Idk why stem doesn't think circuit 8 is managed by us. We just built it!
I think the confusion/bug must come from Tor still building preemptive circuits at this point.
A hacky but easy fix would be to sleep for like 10 seconds after bootstrapping but before doing anything real.
ISTR a torrc option to disable preemptive circuits. Try it. But make sure Tor can still bootstrap with and without an existing data directory.
If we need to measure 90% of the relays in the network, we schedule ourselves to do so ASAP. Thus they will all need measuring again at roughly the same time. So every few days we'll do a ton of work, and otherwise do very little.
We should spread out the load we create.
As an easy hack for my relayscanner, I didn't let more than 200 relays get queued up for testing at a time. Assuming you can do >=200 relays before the next consensus, this spread out the measurements across ~half a day. This still means there are busy days and not busy days.
Perhaps a better idea would be to have each relay's measurement expire at a random time X..Y days from now. After the first huge all-at-once scan, the ~entire network will next be measured slowly over the course of a few days.
[2021-08-02 06:48:45,940] [INFO] [MainThread] [scan] [[email protected]:198] Looking for any new relays that need measured.
Some of the bracketed info is repetitive or unnecessary.
Repetitive:
Unnecessary:
Instead of just QUIT, Libera requires at least:
USER foo * * :foo
QUIT
"foo" should be something that identifies the scanner, in order for us to be good netizens.
A wild guess of mine is that it takes a lot of time to keep checking exit policies. If the relay doesn't allow exiting to, say, 6667, we have to go through all the servers w/port 6667 to conclude that.
It would be nice if we could check *:6667 and if it is not allowed, just skip that relay. But it needs to be checked that we won't miss a relay that allows 6667 to only a small part of the internet. I bet we would miss it. So idk if this will actually work.
(The above description oversimplifies because there are multiple ports we check)
Anyway, look into why the func takes so long and look into what is taking so long. Maybe my guess is wrong.
I skimmed by #tor backlog, found an obvious exit (~[email protected]), found the/a relay that is related to that address.
There's >50 IPv6 address we're missing with just this one hostname:
$ dig tor-exit-anonymizer-v6.appliedprivacy.net AAAA | grep AAAA | wc -l
59
The scanner missed this relay. It doesn't allow IRC ports 6667/6697, but it does allow 443.
We should also scan exits that allow 443 to our web IRC client.
I think this would work in practice 95-100% of the time. I don't think it's likely an exit would allow exiting to the IP checking service but not to our web IRC client (or vice versa).
This sucks because we now have another scan type to implement/maintain. Not hard, but not ideal.
For best results it should be on the same IP(s) as the web IRC client.
I know it is dead simple to setup with plain ngnix. Don't need anything else: not even files.
Again, this sucks because we now have another scan type to implement/maintain. Not hard, but not ideal.
Just take the IPs from the server descriptor. Stay with me here as I move on to the next section. I think we'll still get reasonable results.
We can discover more IPs by doing a couple DNS requests. Watch:
>>> socket.gethostbyaddr('2a03:e600:100::9')
('tor-exit-anonymizer-v6.appliedprivacy.net', [], ['2a03:e600:100::9'])
>>> len(socket.getaddrinfo('tor-exit-anonymizer-v6.appliedprivacy.net', 443, proto=socket.IPPROTO_TCP))
57
We only test one destination on one port port 6667. Perhaps this will be taken care of by other work, but if not:
Look for relays that allow exiting to 6697. Make sure we scan them too.
Test all the publicly documented ports on https://oftc.net/
ircs://irc.oftc.net:6697 for SSL (alternative port: 9999), IPv4 and IPv6.
irc://irc.oftc.net:6667 for non-SSL (alternative ports: 6668-6670, 7000), IPv4 and IPv6.
I worry about the impact on the time it takes to run schedule_new_relays()
. Measure the before/after and judge whether the extra coverage, if any, is worth it. This function is run every 1-2 hours, and takes 5s for me on my desktop.
We're including 127.0.0.1, and I'm going to deploy a hack to exclude it, but all private IPs should be excluded.
The good-relays.txt file is re-read for every measurement.
Right now we connect directly to a hardcoded IP of one of OFTC's servers.
We should try connecting to both irc4.oftc.net and irc6.oftc.net.
When doing this, we need to be careful we don't do it back-to-back, as the IRC will complain at you that you're reconnecting too quickly.
This relay somehow resulted in both ::84.209.139.0 and 84.209.139.0
https://metrics.torproject.org/rs.html#details/AC717A01B8E3C00E7617EF65117A4E99C02DC7A0
This script shows me the only way I can get the '::' prefix is by supplying it myself.
#!/usr/bin/env python3
import socket
def f(host: str):
for ret in socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP):
print(ret)
if __name__ == '__main__':
f('cm-84.209.139.0.getinternet.no')
print('---')
f('84.209.139.0')
print('---')
f('::84.209.139.0')
$ python3 a.py
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('84.209.139.0', 0))
---
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('84.209.139.0', 0))
---
(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::84.209.139.0', 0, 0, 0))
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.