Comments (4)
The proxy never returns a 502 response code. But it enforces timeouts, so that clients don't wait forever. There is also a maximum number of simultaneous connections (512, by default).
When one of these limits is reached, the proxy will close the connection, probably leading to a 502 response code generated by Nginx.
Try increasing these limits to see if it helps. But then you also need to make sure that you are not going to hit system limits, such as the file descriptor limits.
I also run the DoH servers behind Nginx, but I don't log.
from doh-server.
I've a slightly better understanding now but further digging is needed and help is welcome. :)
nginx config
Let's start with some excerpts from the nginx config to better understand the logs:
log_format minimaldns_upstream '$time_iso8601 $status $upstream_status $upstream_addr $geoip_country_code';
upstream rust_doh_backend {
server 127.0.0.1:3000;
server 127.0.0.1:3001;
}
two types of HTTP 502 log events
We observed two type of log entries related to HTTP 502:
a)
2019-05-26T16:04:06+02:00 502 502 127.0.0.1:3000 -
same for the instance on port 3001.
They occur rarely but consistently.
b)
2019-05-26T15:57:14+02:00 502 502 rust_doh_backend -
They occurred even less frequently than (a) but when they happen they came at a high number at once.
https://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_addr explains the difference of the upstream_addr
value:
If a server cannot be selected, the variable keeps the name of the server group.
conclusion about type (b) log events
That means during type (b) events, nginx considered all two backends unavailable.
"failed" is defined in: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream
So by increasing max_fails
to a value higher than 1 we can prevent type (b) events to tell nginx "try harder, don't give up so easily".
Now we need to find out more about type (a) log events. According to nginx, doh-proxy actually returned HTTP 502.
Since there are two doh-proxy instances running we are not anywhere near the default limit of concurrent connections since nginx gets less than that limit (512).
HTTP 502 consistent across instances
The following logs suggest that the error is consistent across the two instances of doh-proxy:
2019-05-26T16:44:04+02:00 502 502, 502 127.0.0.1:3000, 127.0.0.1:3001 -
2019-05-26T16:44:06+02:00 502 502, 502 127.0.0.1:3000, 127.0.0.1:3001 -
2019-05-26T16:44:38+02:00 502 502, 502 127.0.0.1:3000, 127.0.0.1:3001 -
2019-05-26T16:44:46+02:00 502 502, 502 127.0.0.1:3001, 127.0.0.1:3000 -
2019-05-26T16:44:53+02:00 502 502, 502 127.0.0.1:3001, 127.0.0.1:3000 -
from doh-server.
I assume I understand the issue better now.
This is expected:
curl http://127.0.0.1:3000/query -v
* Expire in 0 ms for 6 (transfer 0x55664e118d00)
* Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55664e118d00)
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /query HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< content-length: 0
< date: Fri, 31 May 2019 22:03:45 GMT
<
* Connection #0 to host 127.0.0.1 left intact
This is unexpected (no response at all):
curl -d "" http://127.0.0.1:3000/query -v
* Expire in 0 ms for 6 (transfer 0x562d8ddb4d00)
* Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x562d8ddb4d00)
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> POST /query HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.64.0
> Accept: */*
> Content-Length: 0
> Content-Type: application/x-www-form-urlencoded
>
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server
What do you think about returning 400 Bad Request
in such cases of malformed requests instead of closing the connection directly?
For the upstream timeout case that currently also results in a closed connection:
What do you think about returning 504 Gateway Timeout
for cases where it reaches the 10 seconds timeout?
https://tools.ietf.org/html/rfc7231#section-6.6.5
This would be nice so we could tell these cases apart in the error stats.
from doh-server.
de99e6a returns a 400
error code for POST
queries (this was already the case for GET
queries).
Returning something on a timeout seems to be too complicated for my knowledge of Rust and Tokio. The timeout is on the socket itself, and once it fires, the HTTP layer is not accessible any more.
Something I may revisit once async I/O in Rust stabilizes and becomes a bit more friendly.
from doh-server.
Related Issues (20)
- Stop working while doh-proxy received the first request. HOT 12
- (rust-doh 0.2.1) return HTTP Error 400 HOT 3
- SSL_ERROR_RX_MALFORMED_CERTIFICATE with Let's Encrypt certificates HOT 1
- Cannot download binaries if not logged into Github HOT 4
- HTTPS/SSL with self signed failed HOT 3
- SSL Cert decode issue? HOT 1
- HTTP/2 Termination Question HOT 5
- doh-proxy 0.3.4 build error HOT 3
- TLS certificates error: No private keys found HOT 8
- try to listen on both ipv4 and ipv6 HOT 4
- odohrelay-crypto-sx: Malformed response HOT 2
- Retry over TCP HOT 6
- prometheus metrics HOT 1
- [Feature Request] Support more cipher suites like in dnscrypt-proxy HOT 2
- howto tutorial HOT 3
- Can we have Windows precompiled binaries? HOT 1
- doh-proxy 0.9.6 deb package incompatible with latest Debian libc6 HOT 5
- Crash when using certs (clap?) HOT 1
- Which hash to use in the sdns address for DoH HOT 5
- Some domains cannot be resolved (timeout). Behavior different depending on the DoH client used HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from doh-server.