I've made a couple of attempts to get Flamethrower working against DoH servers in a test harness environment running Ubuntu 20.04.1 LTS, and so far I'm not having any luck.
My first attempt was to deploy Apache 2.4.41 as the HTTP/2 service handler/muxer and DoH Server desribed here: https://github.com/m13253/dns-over-https
The only major differences in my deplpyment are the DoH server and Apache are not on the same box (mostly for tcpdump and troubleshooting reasons), and the fact I'm not using OSCP stapling or a real CA signed cert, which as far as I know should be perfectly fine for an isolated test harness.
My Apache 2.4.41 config is very simple, and just terminates the H2 connection and proxies the query to the DoH server:
<IfModule mod_ssl.c>
SSLProtocol TLSv1.2
SSLHonorCipherOrder On
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+3DES:!aNULL:!MD5:!DSS:!eNULL:!EXP:!LOW:!MD5
# SSLUseStapling on
# SSLStaplingCache shmcb:/var/lib/apache2/stapling_cache(512000)
<VirtualHost _default_:443>
ServerName kurochan
Protocols h2 http/1.
ProxyPass /dns-query http://10.10.20.20:8053/dns-query
ProxyPassReverse /dns-query http://10.10.20.20:8053/dns-query
# SSL Engine Switch:
# Enable/Disable SSL for this virtual host.
SSLEngine on
# A self-signed (snakeoil) certificate can be created by installing
# the ssl-cert package. See
# /usr/share/doc/apache2.2-common/README.Debian.gz for more info.
# If both key and certificate are stored in the same file, only the
# SSLCertificateFile directive is needed.
SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
ErrorLog /var/log/apache2/error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel debug
CustomLog /var/log/apache2/ssl_access.log combined
</VirtualHost>
</IfModule>
When I use a DoH test client such as the doh-client that comes with facebookexperimental's doh-proxy (https://github.com/facebookexperimental/doh-proxy), it works just fine, as shown here:
root@planck2:~# doh-client-fb --qname www.doh1.com --qtype A --domain 10.10.20.21 --remote-address 10.10.20.21 --insecure
2020-12-23 11:35:53,462: DEBUG: Opening connection to 10.10.20.21
2020-12-23 11:35:53,472: DEBUG: Query parameters: {'dns': 'AAABAAABAAAAAAAAA3d3dwRkb2gxA2NvbQAAAQAB'}
2020-12-23 11:35:53,473: DEBUG: Stream ID: 1 / Total streams: 0
2020-12-23 11:35:53,478: DEBUG: Response headers: [(':status', '200'), ('date', 'Wed, 23 Dec 2020 19:35:53 GMT'), ('server', 'DNS-over-HTTPS/2.2.5 (+https://github.com/m13253/dns-over-https)'), ('access-control-allow-headers', 'Content-Type'), ('access-control-allow-methods', 'GET, HEAD, OPTIONS, POST'), ('access-control-allow-origin', '*'), ('access-control-max-age', '3600'), ('cache-control', 'private, max-age=500'), ('content-type', 'application/dns-message'), ('expires', 'Wed, 23 Dec 2020 19:44:13 GMT'), ('last-modified', 'Wed, 23 Dec 2020 19:35:53 GMT'), ('vary', 'Accept'), ('x-powered-by', 'DNS-over-HTTPS/2.2.5 (+https://github.com/m13253/dns-over-https)'), ('content-length', '234')]
id 0
opcode QUERY
rcode NOERROR
flags QR AA RD RA
edns 0
payload 4096
option ECS 10.10.20.0/24 scope/0
;QUESTION
www.doh1.com. IN A
;ANSWER
www.doh1.com. 1800 IN A 10.10.10.12
www.doh1.com. 1800 IN A 10.10.10.10
www.doh1.com. 1800 IN A 10.10.10.13
www.doh1.com. 1800 IN A 10.10.10.14
www.doh1.com. 1800 IN A 10.10.10.11
;AUTHORITY
doh1.com. 1800 IN NS ns.
;ADDITIONAL
ns. 500 IN A 1.1.1.1
2020-12-23 11:35:53,485: DEBUG: Response trailers: {}
Apache ssl_access.log:
10.10.20.20 - - [23/Dec/2020:11:35:53 -0800] "GET /dns-query?dns=AAABAAABAAAAAAAAA3d3dwRkb2gxA2NvbQAAAQAB HTTP/2.0" 200 587 "-" "-"
However when I try to run Flamethrower with extremely light serial queries pointed at the Apache server, Apache rejects the HTTP/2. It appears it does not like something about the HEADERS frame, but I'm not entirely sure what:
root@planck2:~# docker run ns1labs/flame -P doh https://10.10.20.21/dns-query -r www.doh1.com -q 1 -c 1 -d 100
binding to 0.0.0.0
flaming target(s) [10.10.20.21] on port 443 with 1 concurrent generators, each sending 1 queries every 100ms on protocol doh
query generator [static] contains 1 record(s)
0.995918s: send: 1, avg send: 1, recv: 0, avg recv: 0, min/avg/max resp: 0/-nan/0ms, in flight: 1, timeouts: 0
^C1.18096s: send: 0, avg send: 1, recv: 0, avg recv: 0, min/avg/max resp: 0/-nan/0ms, in flight: 1, timeouts: 0
stopping, waiting up to 3s for in flight to finish...
------
run id : 7ffd9bda0320
run start : 2020-12-23T19:50:19Z
runtime : 4.18091 s
total sent : 1
total rcvd : 0
min resp : 0 ms
avg resp : -nan ms
max resp : 0 ms
avg r qps : 0
avg s qps : 1
avg pkt : 55 bytes
tcp conn. : 1
timeouts : 1 (100%)
bad recv : 0
net errors : 0
Apache error log:
[Wed Dec 23 11:50:19.539183 2020] [ssl:info] [pid 13986:tid 140707071219456] [client 10.10.20.20:37529] AH01964: Connection to child 13 established (server kurochan:443)
[Wed Dec 23 11:50:19.539398 2020] [ssl:debug] [pid 13986:tid 140707071219456] ssl_engine_kernel.c(2372): [client 10.10.20.20:37529] AH02645: Server name not provided via TLS extension (using default/first virtual host)
[Wed Dec 23 11:50:19.539464 2020] [ssl:debug] [pid 13986:tid 140707071219456] ssl_engine_kernel.c(2372): [client 10.10.20.20:37529] AH02645: Server name not provided via TLS extension (using default/first virtual host)
[Wed Dec 23 11:50:19.539591 2020] [core:debug] [pid 13986:tid 140707071219456] protocol.c(2257): [client 10.10.20.20:37529] AH03155: select protocol from h2,http/1., choices=h2 for server kurochan
[Wed Dec 23 11:50:19.539709 2020] [core:debug] [pid 13986:tid 140707071219456] protocol.c(2302): [client 10.10.20.20:37529] AH03156: select protocol, proposals=h2 preferences=h2,http/1. configured=h2,http/1.
[Wed Dec 23 11:50:19.539743 2020] [core:debug] [pid 13986:tid 140707071219456] protocol.c(2320): [client 10.10.20.20:37529] AH03157: selected protocol=h2
[Wed Dec 23 11:50:19.585575 2020] [ssl:debug] [pid 13986:tid 140707071219456] ssl_engine_kernel.c(2233): [client 10.10.20.20:37529] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
[Wed Dec 23 11:50:19.585748 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(958): [client 10.10.20.20:37529] AH03200: h2_session(13,INIT,0): created, max_streams=100, stream_mem=32768, workers_limit=6, workers_max=37, push_diary(type=1,N=256)
[Wed Dec 23 11:50:19.585811 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1040): [client 10.10.20.20:37529] AH03201: h2_session(13,INIT,0): start, INITIAL_WINDOW_SIZE=65535, MAX_CONCURRENT_STREAMS=100
[Wed Dec 23 11:50:19.585846 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(2134): [client 10.10.20.20:37529] AH03079: h2_session(13,INIT,0): started on kurochan:443
[Wed Dec 23 11:50:19.585869 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,BUSY,0): transit [INIT] -- init --> [BUSY]
[Wed Dec 23 11:50:19.585913 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,BUSY,0): sent FRAME[SETTINGS[length=6, stream=0]], frames=0/1 (r/s)
[Wed Dec 23 11:50:19.585966 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,BUSY,0): sent FRAME[WINDOW_UPDATE[stream=0, incr=2147418112]], frames=0/2 (r/s)
[Wed Dec 23 11:50:19.586062 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,IDLE,0): transit [BUSY] -- no io (keepalive) --> [IDLE]
[Wed Dec 23 11:50:19.629428 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(341): [client 10.10.20.20:37529] AH03066: h2_session(13,IDLE,0): recv FRAME[SETTINGS[length=6, stream=0]], frames=0/2 (r/s)
[Wed Dec 23 11:50:19.629545 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(341): [client 10.10.20.20:37529] AH03066: h2_session(13,IDLE,0): recv FRAME[SETTINGS[ack=1, stream=0]], frames=1/2 (r/s)
[Wed Dec 23 11:50:19.629616 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_stream.c(543): [client 10.10.20.20:37529] AH03082: h2_stream(13-1,IDLE): created
[Wed Dec 23 11:50:19.629650 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,BUSY,1): transit [IDLE] -- stream change --> [BUSY]
[Wed Dec 23 11:50:19.629697 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(221): [client 10.10.20.20:37529] AH03063: h2_session(13,BUSY,1): recv invalid FRAME[HEADERS[length=71, hend=1, stream=1, eos=1]], frames=2/2 (r/s)
[Wed Dec 23 11:50:19.629738 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,BUSY,1): sent FRAME[SETTINGS[ack=1, stream=0]], frames=2/3 (r/s)
[Wed Dec 23 11:50:19.629762 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,BUSY,1): sent FRAME[RST_STREAM[length=4, flags=0, stream=1]], frames=2/4 (r/s)
[Wed Dec 23 11:50:19.629798 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(267): [client 10.10.20.20:37529] AH03065: h2_stream(13-1,CLOSED): closing with err=1 protocol error
[Wed Dec 23 11:50:19.629895 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,IDLE,0): transit [BUSY] -- no io (keepalive) --> [IDLE]
[Wed Dec 23 11:50:22.548659 2020] [ssl:info] [pid 13986:tid 140707062826752] (70014)End of file found: [client 10.10.20.20:37529] AH01991: SSL input filter read failed.
[Wed Dec 23 11:50:22.548889 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(2192): (70014)End of file found: [client 10.10.20.20:37529] AH03403: h2_session(13,IDLE,0): no data, error
[Wed Dec 23 11:50:22.548927 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(1788): [client 10.10.20.20:37529] AH03401: h2_session(13,IDLE,0): conn error -> shutdown
[Wed Dec 23 11:50:22.548966 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,IDLE,0): sent FRAME[GOAWAY[error=0, reason='timeout', last_stream=1]], frames=2/5 (r/s)
[Wed Dec 23 11:50:22.549066 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(753): [client 10.10.20.20:37529] AH03069: h2_session(13,IDLE,0): sent GOAWAY, err=0, msg=timeout
[Wed Dec 23 11:50:22.549109 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,DONE,0): transit [IDLE] -- local goaway --> [DONE]
[Wed Dec 23 11:50:22.549166 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_conn.c(213): (70014)End of file found: [client 10.10.20.20:37529] AH03045: h2_session(13,DONE,0): process, closing conn
[Wed Dec 23 11:50:22.549205 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,CLEANUP,0): transit [DONE] -- pre_close --> [CLEANUP]
[Wed Dec 23 11:50:22.549305 2020] [ssl:debug] [pid 13986:tid 140707062826752] ssl_engine_io.c(1106): [client 10.10.20.20:37529] AH02001: Connection closed to child 14 with standard shutdown (server kurochan:443)
Apache really doesn't like something about the HTTP/2, claiming it received an invalid frame.
My final attempt was to give up on Apache and just spin up the facebookexperimental doh-proxy, but pointing Flamethrower to that also fails.
Launching doh-proxy:
NOTE: doh-proxy was slightly modified to print out the headers so I could see what doh-proxy was getting from the Flamethrower client.
root@planck2:~# doh-proxy --certfile /etc/ssl/certs/ssl-cert-snakeoil.pem --keyfile /etc/ssl/private/ssl-cert-snakeoil.key --listen-address 10.10.20.20 --upstream-resolver localhost
2020-12-23 13:45:03,403: INFO: Serving on <Server sockets=[<socket.socket fd=8, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('10.10.20.20', 443)>]>
Running Flamethrower:
root@planck2:~# docker run ns1labs/flame -P doh https://10.10.20.20/dns-query -r www.doh1.com -q 1 -c 1 -d 100
binding to 0.0.0.0
flaming target(s) [10.10.20.20] on port 443 with 1 concurrent generators, each sending 1 queries every 100ms on protocol doh
query generator [static] contains 1 record(s)
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
Output from doh-proxy (printing the headers list):
[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
As you can see above, Flamethrower is claiming it's getting back malformed data which could be true, but it also looks like the headers being sent by Flamethrower might be an issue. The path header is supposed to contain the complete URI, including the /dns-query part, but it's missing for some reason. Scheme and authority should probably also be set. Any ideas why this might be the case?
Am I doing something wrong here in how I'm executing Flamethrower from Docker? Any help/guidance would be super cool.
Thanks in advance, and happy holidays! :)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent