haveibeensquatted / twistrs Goto Github PK
View Code? Open in Web Editor NEWA domain name permutation and enumeration library powered by Rust.
Home Page: https://haveibeensquatted.com/
License: MIT License
A domain name permutation and enumeration library powered by Rust.
Home Page: https://haveibeensquatted.com/
License: MIT License
Details originated from this thread.
We should add word mappings such as:
m -> nn
cl -> d
w -> vv
b -> l3
More can be included down the line.
When generating permutations for domains such as ox.ac.uk
, we are entering into a run-off state where the number of permutations keeps cycling -- which is a bug.
If we generate permutations for cool.com
, we should expect to find çool.com
(i.e., https://xn--ool-1la.com/) in the list of domains.
Current tracing into the internals of the library is non-existent, making it very difficult to understand what is happening internally when using the library. This should change and include some instrumentation to help with this.
Currently the permutation library does not perform any sort of IDNA domain filtering which hinders performance, particularly for lookups. This method is implemented by dnstwist
and should be included into all permutation methods.
Currently when performing DNS resolution for linkedin.com
, twistrs is yielding 158 domains, whereas dnstwist.py is yielding 178.
python3 dnstwist.py linkedin.com --registered
cd examples/twistrs-cli
cargo r -- linkedin.com
twistrs yields as many, or more, domains as dnstwist (currently 178).
twistrs yields 158 domains behind dnstwist's 158 domains.
Using the rust-geopip
library (or something similar), use the cached GeoIP lookup table to perform lookups on resolved IPs.
Note that message signature may vary as IPs need to pushed as opposed to FQDNs.
Enhance the current example suite by including a new client implementation using WebSockets.
For .cz TLDs there are widely used .cy TLDs as another type of typosquat attack.
Because people here in CZ very often make a mistake to type "y" instead of "z" (Czech keyboard has switched position of these two letters in comparison to ENG keyboard.)
For example: seznam.cz (legit) vs seznam.cy (fake)
Currently the library is a mix between async and non-async modules. The permutation engine is trivial to use and integrate, making it pleasant to work with. The enrichment module however locks into a specified implementation which is (a) inflexible and (b) a pain to maintain.
After some thoughts, it seems that enrichment does not necessarily belong in the library at all, besides perhaps some traits to help library users extend the functionality to their liking.
Wanted to import and run
let domain = Domain::new("google.com").unwrap();
let permutations = domain.all().unwrap();
Getting errors at bottom.
❯ cargo --version
cargo 1.72.0 (103a7ff2e 2023-08-15)
great-squat on main [!] is 📦 v0.1.0 via 🦀 v1.72.0 on ☁️ (us-east-1)
❯ rustc --version
rustc 1.72.0 (5680fa18f 2023-08-23)
errors:
❯ cargo build
Compiling twistrs v0.5.2 (https://github.com/haveibeensquatted/twistrs.git?branch=main#5ebc1d17)
error[E0277]: the trait bound `String: Serialize` is not satisfied
--> /home/tscanlan/.cargo/git/checkouts/twistrs-ab64395ec7e7a9bf/5ebc1d1/twistrs/src/enrich.rs:90:24
|
90 | #[derive(Debug, Clone, Serialize, Default)]
| ^^^^^^^^^ the trait `Serialize` is not implemented for `String`
91 | pub struct DomainMetadata {
92 | /// The domain that is being enriched.
| -------------------------------------- required by a bound introduced by this call
|
= help: the trait `Serialize` is implemented for `str`
note: required by a bound in `enrich::_::_serde::ser::SerializeStruct::serialize_field`
--> /home/tscanlan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.188/src/ser/mod.rs:1865:12
|
1859 | fn serialize_field<T: ?Sized>(
| --------------- required by a bound in this associated function
...
1865 | T: Serialize;
| ^^^^^^^^^ required by this bound in `SerializeStruct::serialize_field`
error[E0277]: the trait bound `Vec<std::net::IpAddr>: Serialize` is not satisfied
--> /home/tscanlan/.cargo/git/checkouts/twistrs-ab64395ec7e7a9bf/5ebc1d1/twistrs/src/enrich.rs:90:24
|
90 | #[derive(Debug, Clone, Serialize, Default)]
| ^^^^^^^^^ the trait `Serialize` is not implemented for `Vec<std::net::IpAddr>`
...
95 | /// Any IPv4 and IPv6 ips that were discovered during
| ----------------------------------------------------- required by a bound introduced by this call
|
= help: the following other types implement trait `Serialize`:
bool
char
isize
i8
i16
i32
i64
i128
and 93 others
= note: required for `std::option::Option<Vec<std::net::IpAddr>>` to implement `Serialize`
note: required by a bound in `enrich::_::_serde::ser::SerializeStruct::serialize_field`
--> /home/tscanlan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.188/src/ser/mod.rs:1865:12
|
1859 | fn serialize_field<T: ?Sized>(
| --------------- required by a bound in this associated function
...
1865 | T: Serialize;
| ^^^^^^^^^ required by this bound in `SerializeStruct::serialize_field`
error[E0277]: the trait bound `Vec<(std::net::IpAddr, String)>: Serialize` is not satisfied
--> /home/tscanlan/.cargo/git/checkouts/twistrs-ab64395ec7e7a9bf/5ebc1d1/twistrs/src/enrich.rs:90:24
|
90 | #[derive(Debug, Clone, Serialize, Default)]
| ^^^^^^^^^ the trait `Serialize` is not implemented for `Vec<(std::net::IpAddr, String)>`
...
106 | /// IP addresses resolved through GeoIP lookup to City, Country, Continent.
| --------------------------------------------------------------------------- required by a bound introduced by this call
|
= help: the following other types implement trait `Serialize`:
bool
char
isize
i8
i16
i32
i64
i128
and 93 others
= note: required for `std::option::Option<Vec<(std::net::IpAddr, String)>>` to implement `Serialize`
note: required by a bound in `enrich::_::_serde::ser::SerializeStruct::serialize_field`
--> /home/tscanlan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.188/src/ser/mod.rs:1865:12
|
1859 | fn serialize_field<T: ?Sized>(
| --------------- required by a bound in this associated function
...
1865 | T: Serialize;
| ^^^^^^^^^ required by this bound in `SerializeStruct::serialize_field`
error[E0277]: the trait bound `String: Serialize` is not satisfied
--> /home/tscanlan/.cargo/git/checkouts/twistrs-ab64395ec7e7a9bf/5ebc1d1/twistrs/src/enrich.rs:115:24
|
115 | #[derive(Debug, Clone, Serialize)]
| ^^^^^^^^^ the trait `Serialize` is not implemented for `String`
...
120 | /// Message received back from the SMTP server
| ---------------------------------------------- required by a bound introduced by this call
|
= help: the trait `Serialize` is implemented for `str`
note: required by a bound in `enrich::_::_serde::ser::SerializeStruct::serialize_field`
--> /home/tscanlan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.188/src/ser/mod.rs:1865:12
|
1859 | fn serialize_field<T: ?Sized>(
| --------------- required by a bound in this associated function
...
1865 | T: Serialize;
| ^^^^^^^^^ required by this bound in `SerializeStruct::serialize_field`
error[E0277]: the trait bound `String: Serialize` is not satisfied
--> /home/tscanlan/.cargo/git/checkouts/twistrs-ab64395ec7e7a9bf/5ebc1d1/twistrs/src/permutate.rs:36:26
|
36 | #[derive(Default, Debug, Serialize)]
| ^^^^^^^^^ the trait `Serialize` is not implemented for `String`
...
41 | /// The top-level domain of the FQDN (e.g. `.com`).
| --------------------------------------------------- required by a bound introduced by this call
|
= help: the trait `Serialize` is implemented for `str`
note: required by a bound in `enrich::_::_serde::ser::SerializeStruct::serialize_field`
--> /home/tscanlan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.188/src/ser/mod.rs:1865:12
|
1859 | fn serialize_field<T: ?Sized>(
| --------------- required by a bound in this associated function
...
1865 | T: Serialize;
| ^^^^^^^^^ required by this bound in `SerializeStruct::serialize_field`
For more information about this error, try `rustc --explain E0277`.
error: could not compile `twistrs` (lib) due to 5 previous errors
Currently we are embedding a GeoIP2 Lite mmdb to be able to run a single small GeoIP test. This is an overkill so we either need to think about git-lfs, or stripping down the sample binary to an acceptable size.
Enhance the current example suite by including a new client implementation using gRPC server.
Currently both PermutationError
and EnrichmentError
are marked as deprecated and are not expected to be used. Further research needs to be done to better understand error handling and propagation and improve it in the core library.
All documentation must be updated to reflect this.
When generating results for lobste.rs
we receive lobste.com
as a potential typosquat domain. However when we search for lobste.com
we do not get lobste.rs
as a result.
Currently missing cached WhoIs lookup implementation which should be feature-gated.
Currently the core enrichment library is functional in terms of permutation methods, however is missing key enrichment methods such as HTTP Banner lookups.
Currently WhoIs lookup is blocking which causes problems when trying to scale the functionality to hundreds of concurrent requests.
Should take a look at the whois_rust
crate and see if it's possible to refactor this to an async call - https://docs.rs/whois-rust/latest/whois_rust/#asynchronous-apis
Since release of the library, there have been a couple of permutations modes that need to be implemented. These currently include:
These need to be implemented and benchmarked to make sure they operate properly
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.