s0md3v / uro Goto Github PK
View Code? Open in Web Editor NEWdeclutters url lists for crawling/pentesting
License: Apache License 2.0
declutters url lists for crawling/pentesting
License: Apache License 2.0
i just can't get this to work
have cloned the repo and run the install command, bur when i try "cat file.txt | uro" it dosen't work.
do i have to do any additional commands?
any installation video??:)
cat urls.txt | uro
Traceback (most recent call last):
File "/usr/local/bin/uro", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/uro/uro.py", line 155, in main
if re.search(pattern, path):
File "/usr/lib/python3.8/re.py", line 201, in search
return _compile(pattern, flags).search(string)
File "/usr/lib/python3.8/re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib/python3.8/sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "/usr/lib/python3.8/sre_parse.py", line 948, in parse
p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
File "/usr/lib/python3.8/sre_parse.py", line 443, in _parse_sub
itemsappend(_parse(source, state, verbose, nested + 1,
File "/usr/lib/python3.8/sre_parse.py", line 598, in _parse
raise source.error(msg, len(this) + 1 + len(that))
re.error: bad character range P-C at position 31
└─# cat params.csv | uro | wc -l
Traceback (most recent call last):
File "/usr/local/bin/uro", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/dist-packages/uro/uro.py", line 155, in main
if re.search(pattern, path):
File "/usr/lib/python3.9/re.py", line 201, in search
return _compile(pattern, flags).search(string)
File "/usr/lib/python3.9/re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib/python3.9/sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "/usr/lib/python3.9/sre_parse.py", line 962, in parse
raise source.error("unbalanced parenthesis")
re.error: unbalanced parenthesis at position 68
6547
Traceback (most recent call last):
File "/usr/local/bin/uro", line 11, in
load_entry_point('uro==0.0.4', 'console_scripts', 'uro')()
File "/usr/local/lib/python3.10/dist-packages/uro-0.0.4-py3.10.egg/uro/uro.py", line 124, in main
File "/usr/lib/python3.10/urllib/parse.py", line 392, in urlparse
splitresult = urlsplit(url, scheme, allow_fragments)
File "/usr/lib/python3.10/urllib/parse.py", line 483, in urlsplit
raise ValueError("Invalid IPv6 URL")
ValueError: Invalid IPv6 URL
tried downloading URO on windows machine, is it possible to use uro on Wndows machine
D:\uro>uro
Traceback (most recent call last):
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\Scripts\uro-script.py", line 33, in
sys.exit(load_entry_point('uro==0.0.2', 'console_scripts', 'uro')())
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\Scripts\uro-script.py", line 25, in importlib_load_entry_point
return next(matches).load()
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\importlib\metadata.py", line 77, in load
module = import_module(match.group('module'))
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\importlib_init_.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 655, in _load_unlocked
File "", line 618, in _load_backward_compatible
File "", line 259, in load_module
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\site-packages\uro-0.0.2-py3.8.egg\uro\uro.py", line 4, in
ImportError: cannot import name 'SIGPIPE' from 'signal' (C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\signal.py)
when i run this command (python3 setup.py install) it install but not working. OS=kali
When using uro on subdomains it leaves :// in front
example:
cat subs.txt | uro
subs.txt example:
site.com
sub.site.com
sub123.site.com
anything without http:// or https:// in front it leaves the :// in front.
When we check the paths, we see that 43935989 and 43935976 are used differently.
root@localhost:~# cat urls.txt
https://news.mail.ru/politics/43935976/?social=tw
https://news.mail.ru/politics/43935989/?social=tw
it should delete one of them but it doesn't.
root@localhost:~# cat urls.txt | uro
https://news.mail.ru/politics/43935976/?social=tw
https://news.mail.ru/politics/43935989/?social=tw
So I added uro to my workflow and after a while I got this error:
Traceback (most recent call last):
File "/usr/local/bin/uro", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/uro/uro.py", line 139, in main
if matches_patterns(path):
File "/usr/local/lib/python3.8/dist-packages/uro/uro.py", line 107, in matches_patterns
if re.search(pattern, path):
File "/usr/lib/python3.8/re.py", line 201, in search
return _compile(pattern, flags).search(string)
File "/usr/lib/python3.8/re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib/python3.8/sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "/usr/lib/python3.8/sre_parse.py", line 948, in parse
p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
File "/usr/lib/python3.8/sre_parse.py", line 443, in _parse_sub
itemsappend(_parse(source, state, verbose, nested + 1,
File "/usr/lib/python3.8/sre_parse.py", line 836, in _parse
raise source.error("missing ), unterminated subpattern",
re.error: missing ), unterminated subpattern at position 369
It is happening to me with different inputs so seems to be something that happens often
Maybe it can be useful to add this extension to the one to declutter, at least, it's what I usually do:
.doc
.docx
.mp3
.mp4
.exe
.tif
.ttf
.woff
.woff2
.ico
.zip
How Can I Get The Result In A txt File?
Multiple URL either have parameters with base64 encoding or any other encodings
My suggestion is to get the output in json format, to tell if a particular url was decoded, and what encoding was present.
extend ext filter css|jpg|png|gif|svg|ico|woff|woff2|pdf|ttf|otf|jpeg|JPG|js|mp4|mp3|avi|eot
Hi Somdev,
'svg','img','gif','mp4','flv','ogv','webm','webp','mov','mp3','m4a','m4p','ppt','pptx','pdf','scss','tif','tiff','ttf','otf','woff','woff2','eot','htc','swf','rtf','image'
js
extension as there are lots of interesting features/endpoints to be found on them and I don't think if they are considered "useless".Thanks!
Kind Regards,
HolyBugx
Remove "js" file filtering
cat urls.txt | uro > test
Traceback (most recent call last):
File "/usr/local/bin/uro", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/uro/uro.py", line 123, in main
for line in sys.stdin:
File "/usr/lib/python3.10/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
suya has the error...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behavior with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
I've done the steps above but haven't found a bright spot :(
can anyone help me???
Traceback (most recent call last):
File "/usr/bin/uro", line 8, in
sys.exit(main())
^^^^^^
File "/home/greyhound/.local/lib/python3.11/site-packages/uro/uro.py", line 163, in main
process_url(parsed_url)
File "/home/greyhound/.local/lib/python3.11/site-packages/uro/uro.py", line 136, in process_url
new_params = [param for param in params.keys() if param not in params_seen]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/greyhound/.local/lib/python3.11/site-packages/uro/uro.py", line 136, in
new_params = [param for param in params.keys() if param not in params_seen]
^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
Ive tried tried setting aliases, changing paths etc wont work..
I am using "uro" with other tools to filter duplicate urls, jpegs, etc., and so on. However, after feeding a large wordlist to "uro" with a lot of url paths ending with /678XXXXXXXXXXXXXXXXXXXXXXX, it crashed. Later on, with other normal urls, it no longer works as usual. I installed the uro by pipx-
pipx install uro
Steps to reproduce:-
cat link_results.txt | uro | tee -a output_uro.txt
Traceback (most recent call last):
File "/usr/local/bin/uro", line 33, in <module>
sys.exit(load_entry_point('uro==0.0.5', 'console_scripts', 'uro')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/uro-0.0.5-py3.11.egg/uro/uro.py", line 123, in main
File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 7841: invalid continuation byte
In the test environment I have -
uro 1.0.0-beta
Python3 Version - 3.11.9
kali 6.8.11-amd64 kernel
pipx version 1.6.0
It might fix if improve or update in the code regarding UnicodeDecode.
like i want to filter "/A/embed?url=" or "/B/embed?url=" which return similar data
like i want to filter "/A.php" or "/A.php/" which return similar data
I have been encountering this issue:
File "/usr/local/bin/uro", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/uro/uro.py", line 151, in main
print(host + path + dict_to_params(param))
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/local/bin/uro", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/uro/uro.py", line 161, in main
print(host + path)
BrokenPipeError: [Errno 32] Broken pipe
Any idea why would it be?
λ cat newfile222.txt | uro
Traceback (most recent call last):
File "C:\Users\Yaseen\AppData\Local\Programs\Python\Python39\Scripts\uro-script.py", line 33, in
sys.exit(load_entry_point('uro==0.0.1', 'console_scripts', 'uro')())
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\site-packages\uro\uro.py", line 139, in main
if matches_patterns(path):
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\site-packages\uro\uro.py", line 107, in matches_patterns
if re.search(pattern, path):
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\re.py", line 201, in search
return _compile(pattern, flags).search(string)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_parse.py", line 948, in parse
p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_parse.py", line 443, in _parse_sub
itemsappend(_parse(source, state, verbose, nested + 1,
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_parse.py", line 836, in _parse
raise source.error("missing ), unterminated subpattern",
re.error: missing ), unterminated subpattern at position 379
cat: write error: No space left on device
Can you help, it saying space issue, i have alot of space
Hi
I've installed the old version and how can I update to new version?
After installing the new version how can I know about the version? There isn't any help menu!
Please add more info about the update and help menu in readme file.
Thanks for sharing your code
Traceback (most recent call last):
File "/usr/local/bin/uro", line 33, in
sys.exit(load_entry_point('uro==0.0.4', 'console_scripts', 'uro')())
File "/usr/local/bin/uro", line 25, in importlib_load_entry_point
return next(matches).load()
StopIteration
I used pip command but not installed.
It shows :
Requirement already satisfied: uro in ./.local/lib/python3.11/site-packages (1.0.0)
Can i use URO for decluttering post request to use fetch unique requests using ffuf
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.