projectdiscovery / fastdialer Goto Github PK
View Code? Open in Web Editor NEWDialer with DNS Cache + Dial History
License: MIT License
Dialer with DNS Cache + Dial History
License: MIT License
net/http
has recover statement with defer in every goroutine it creates to catch panics[3:44:50] | Templates: 7384 | Hosts: 253 | RPS: 141 | Matched: 3255 | Errors: 50463 | Requests: 1911514/2797674 (68%)
panic: runtime error: slice bounds out of range [1721:49]
goroutine 26744483 [running]:
internal/poll.(*FD).Write(0x14016273600, {0x1403009ce00, 0x31, 0x32})
internal/poll/fd_unix.go:380 +0x3ac
net.(*netFD).Write(0x14016273600, {0x1403009ce00?, 0x104f0f0e0?, 0x10512e220?})
net/fd_posix.go:96 +0x28
net.(*conn).Write(0x14011ed7d48, {0x1403009ce00?, 0x14042d90fd8?, 0x10289b44c?})
net/net.go:191 +0x34
github.com/miekg/dns.(*Conn).Write(0x1400dce5400, {0x1403009ce00, 0x31, 0x32})
github.com/miekg/[email protected]/client.go:359 +0x108
github.com/miekg/dns.(*Conn).WriteMsg(0x1400dce5400, 0xc47f431c4ed?)
github.com/miekg/[email protected]/client.go:348 +0xe4
github.com/miekg/dns.(*Client).ExchangeWithConnContext(0x140009faa80, {0x1051e4b78, 0x106722380}, 0x14015f11440, 0x1400dce5400)
github.com/miekg/[email protected]/client.go:220 +0x2a8
github.com/miekg/dns.(*Client).ExchangeWithConn(...)
github.com/miekg/[email protected]/client.go:187
github.com/miekg/dns.(*Client).Exchange(0x140451584e0?, 0x140451584e0?, {0x14013099540?, 0x104a08320?})
github.com/miekg/[email protected]/client.go:170 +0xe0
github.com/projectdiscovery/retryabledns.(*Client).queryMultiple(0x14000b6d5f0, {0x140451584e0, 0x14}, {0x14042d913c4, 0x2, 0x1031545f0?}, {0x0, 0x0})
github.com/projectdiscovery/[email protected]/client.go:361 +0x718
github.com/projectdiscovery/retryabledns.(*Client).QueryMultiple(...)
github.com/projectdiscovery/[email protected]/client.go:264
github.com/projectdiscovery/retryabledns.(*Client).Resolve(0x1400039c6c0?, {0x140451584e0?, 0x140451584e0?})
github.com/projectdiscovery/[email protected]/client.go:144 +0x44
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).GetDNSData(0x1400039c6c0, {0x140451584e0?, 0x1051c1700?})
github.com/projectdiscovery/[email protected]/fastdialer/dialer.go:489 +0x390
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).dial(0x1400039c6c0, {0x1051e4da8, 0x14003ed2e00}, {0x103fcdf09, 0x3}, {0x140451584e0, 0x17}, 0x0, 0x0, 0x0, ...)
github.com/projectdiscovery/[email protected]/fastdialer/dialer.go:216 +0x29c
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).Dial(...)
github.com/projectdiscovery/[email protected]/fastdialer/dialer.go:136
github.com/projectdiscovery/retryablehttp-go.DefaultReusePooledTransport.func1({0x1051e4da8?, 0x14003ed2e00?}, {0x103fcdf09?, 0x104d9dfe0?}, {0x140451584e0?, 0x1027f249c?})
github.com/projectdiscovery/[email protected]/http.go:52 +0x64
net/http.(*Transport).dial(0x1400f4bb5c0?, {0x1051e4da8?, 0x14003ed2e00?}, {0x103fcdf09?, 0x14042d91a58?}, {0x140451584e0?, 0x1?})
net/http/transport.go:1183 +0xdc
net/http.(*Transport).dialConn(0x14000c39180, {0x1051e4da8, 0x14003ed2e00}, {{}, 0x0, {0x14008fad780, 0x4}, {0x140451584e0, 0x17}, 0x0})
net/http/transport.go:1625 +0x61c
net/http.(*Transport).dialConnFor(0x140253bf260?, 0x1401def9550)
net/http/transport.go:1467 +0x7c
created by net/http.(*Transport).queueForDial in goroutine 26738384
net/http/transport.go:1436 +0x380
this seems mostly due to open fd etc but i think maybe panic could have been avoided if we have added recover and caught it in fastdialer
Some bug fixes since v1.5.3
Replace deepcopier with custom clone of only meaningful fields
Hello,
I found that when I try using this library in my own projects, my stderr gets flooded with multiple messages. I'm unsure if I'm doing something wrong to cause these errors messages, but no error seems to be returned during initialization and everything runs fine.
stderr
2021/06/12 01:36:09 /tmp/hm928258121
2021/06/12 01:36:09 /tmp/hm341994260
2021/06/12 01:36:09 /tmp/hm957686115
2021/06/12 01:36:09 /tmp/hm497662310
2021/06/12 01:36:09 /tmp/hm236611981
2021/06/12 01:36:09 /tmp/hm279069576
2021/06/12 01:36:09 /tmp/hm406964039
2021/06/12 01:36:09 /tmp/hm423496186
2021/06/12 01:36:09 /tmp/hm279549201
2021/06/12 01:36:09 /tmp/hm890850108
2021/06/12 01:36:09 /tmp/hm536011627
2021/06/12 01:36:09 /tmp/hm160537038
How I use fastdialer
func buildHttpClient() (c *http.Client) {
fastdialerOpts := fastdialer.DefaultOptions
fastdialerOpts.EnableFallback = true
dialer, err := fastdialer.NewDialer(fastdialerOpts)
if err != nil {
fmt.Printf("Error initializing dialer: %s\n", err)
return
}
transport := &http.Transport{
MaxIdleConns: -1,
IdleConnTimeout: time.Second,
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
DisableKeepAlives: true,
DialContext: dialer.Dial,
}
re := func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
}
client := &http.Client{
Transport: transport,
CheckRedirect: re,
Timeout: time.Second * 10,
}
return client
}
I should note, this only happens when using fastdialer. I switched over a couple of projects to use the library and had these messages pop up on all of them.
Multiple tools over time are showing the necessity of allowing/blocking traffic upon specific patterns. This has been implemented in dishomogeneous ways in multiple apps at different OSI layers.
Most allowing/exclusion logic happens at layer 7 (ex. HTTP):
In other tools such as naabu
this happens at layer 4 via ipranger
The task is about implementing a generic cross-tool solution capable to plug allow/block capabilities with maximum coverage for all tools. At current time fastdialer
seems a good candidate for most of the tools performing network activities as it sits between layer 7 and layer 4. The component should expose helper/callbacks to add items to allow/block list covering the same cases mentioned in projectdiscovery/httpx#1427.
Note: The logic should also work over socks5 proxy
examples
folderInvestigate introducing parallel dialing in fastdialer/fastdialer/dialer.go.dial(...)
and returning the first successful connection (with optional reuse/load balancing of other connections) similarly to https://go.dev/src/net/dial.go.dialParallel(...)
to reduce failure time and cumulative timeout
If a domain, such as hackerone.com, resolves to multiple IPs, Fastdialer previously used sequential iteration. This was inefficient because if a port on the address was closed, Fastdialer would iterate over all IPs and each failed iteration would add to the total time. This issue was exacerbated when Nuclei called this function on 1000 goroutines, creating a bottleneck and delay in execution.
Add resolvers from /etc/resolv/conf
on unix systems
When custom ip is provided via the ip
context argument, no other ips should be attempted. Actually we put the custom ip as first to try in the list, and then append all ipv4 and ipv6.
This behavior might not be the expected one, as for those tools using scan all ips functionalities it can cause multiple connections to the same target.
DialTLSConfig
due to client certificates . hence fallback logic is not applied at all now with fastdialer it is preserved and available by defaultrenegotiation
was not set which caused projectdiscovery/nuclei#3553 in nucleiI want to know if the Socks5 proxy is used when querying DNS?
Currently, the default state for HTTPX/Naabu/Nuclei is to use a set of default resolvers for DNS as defined in the DefaultResolvers
variable within options.go
in fastdialer.
// DefaultResolvers trusted
var DefaultResolvers = []string{
"1.1.1.1:53",
"1.0.0.1:53",
"8.8.8.8:53",
"8.8.4.4:53",
}
This is a deviation from the expected behavior, which is to use the host systems DNS configuration as a default. This is, for example, the way curl works.
There are a few significant drawbacks to doing this.
Use utils.
i know it has supported utls in tls handshake,but it does not support utls with socks5 proxy support,so iadvise to add this feature
newest release and source code are different, can you update?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.