Comments (9)
Thanks for following up and providing such great detail!!! Wonderful!
Re: The context regex, you can drop the square brackets the question mark will only apply to the preceding token (the s). https://www.regexplanet.com/share/index.html?share=yyyyfpma03r (Click 'Java')
Ah yes - nice idea, thanks!
from zaproxy.
Did you try the weekly? That one has a newer Selenium version.
from zaproxy.
@thc202 I hadn't - I've just tried it now. I'm no longer receiving any errors, but it also doesn't seem to the spidering correctly either. Polling for AJAX results returns 0
, and when it eventually completes and I try to run an Active scan (after running the Traditional + AJAX spider) I get the following crash error:
2024-01-17 19:38:29 59953 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Running Crawljax (with chrome-headless): API - Context: default
2024-01-17 19:38:29 59969 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Starting proxy...
2024-01-17 19:38:29 59984 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Proxy started, listening at port [35473].
2024-01-17 19:38:30 60283 [ZAP-AjaxSpiderApi] INFO com.crawljax.core.plugin.Plugins - Loaded org.zaproxy.zap.extension.spiderAjax.SpiderThread$DummyPlugin@6fd3216e as a OnBrowserCreatedPlugin
2024-01-17 19:39:46 136260 [ZAP-AjaxSpiderApi] INFO com.crawljax.core.CrawlController - Received shutdown notice. Reason is Exausted
2024-01-17 19:39:46 136382 [ZAP-AjaxSpiderApi] INFO com.crawljax.core.CrawlController - Shutdown process complete
2024-01-17 19:39:46 136382 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Stopping proxy...
2024-01-17 19:39:46 136399 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Proxy stopped.
2024-01-17 19:39:46 136408 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Finished Crawljax: API - Context: default
2024-01-17 19:39:51 141656 [ZAP-IO-Server-1-2] INFO org.parosproxy.paros.core.scanner.Scanner - scanner started
2024-01-17 19:39:51 141789 [ZAP-IO-Server-1-2] ERROR org.zaproxy.zap.extension.api.API - Exception while handling API request:
2024-01-17 19:39:51 java.util.ConcurrentModificationException: null
2024-01-17 19:39:51 at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) ~[?:?]
2024-01-17 19:39:51 at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.zap.extension.ascan.ActiveScanAPI.handleApiView(ActiveScanAPI.java:1035) ~[zap-D-2024-01-16.jar:D-2024-01-16]
2024-01-17 19:39:51 at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:557) ~[zap-D-2024-01-16.jar:D-2024-01-16]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleApiRequest(ZapApiHandler.java:111) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleRequest(ZapApiHandler.java:85) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleMessage(ZapApiHandler.java:70) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.MainServerHandler.notifyMessageHandlers(MainServerHandler.java:151) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.MainServerHandler.processMessage(MainServerHandler.java:131) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.LocalServerHandler.processMessage(LocalServerHandler.java:67) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.MainServerHandler.process(MainServerHandler.java:94) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.MainServerHandler.lambda$channelRead0$0(MainServerHandler.java:82) ~[?:?]
2024-01-17 19:39:51 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2024-01-17 19:39:51 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2024-01-17 19:39:51 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [network-beta-0.14.0.zap:?]
2024-01-17 19:39:51 at java.lang.Thread.run(Thread.java:829) [?:?]
from zaproxy.
Which API call are you using for the AJAX Spider results?
For the Active Scan call that's a bug but calling the endpoint again it would work.
from zaproxy.
@thc202 - I'm polling the ajaxSpider.number_of_results
endpoint, which always returns 0
. I just double-checked in the Python console as well and got the following for a ZAP instance after running the AJAX spider with Chrome:
scanner.zap.ajaxSpider.number_of_results
Out[5]: '0'
scanner.zap.ajaxSpider.results()
Out[6]: []
The Active Scan seems to work this morning 🤷
Edit: Just realised there is an extra couple of error messages in the ZAP logs this morning:
2024-01-18 09:06:04 58454 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Running Crawljax (with chrome-headless): API - Context: default
2024-01-18 09:06:04 58510 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Starting proxy...
2024-01-18 09:06:04 58535 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Proxy started, listening at port [38749].
2024-01-18 09:06:05 59392 [ZAP-AjaxSpiderApi] INFO com.crawljax.core.plugin.Plugins - Loaded org.zaproxy.zap.extension.spiderAjax.SpiderThread$DummyPlugin@2be6d0f0 as a OnBrowserCreatedPlugin
2024-01-18 09:06:25 79574 [ZAP-IO-EventExecutor-4-2] WARN org.zaproxy.addon.network.internal.server.http.MainServerHandler - Failed to write/forward the HTTP response to the client: java.io.IOException: Broken pipe
2024-01-18 09:06:45 99547 [ZAP-IO-EventExecutor-4-2] WARN org.zaproxy.addon.network.internal.server.http.MainServerHandler - Failed to write/forward the HTTP response to the client: java.io.IOException: Broken pipe
2024-01-18 09:07:34 148562 [ZAP-AjaxSpiderApi] INFO com.crawljax.core.CrawlController - Received shutdown notice. Reason is Exausted
2024-01-18 09:07:34 148918 [ZAP-AjaxSpiderApi] INFO com.crawljax.core.CrawlController - Shutdown process complete
2024-01-18 09:07:34 148920 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Stopping proxy...
2024-01-18 09:07:34 148978 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Proxy stopped.
2024-01-18 09:07:34 149017 [ZAP-AjaxSpiderApi] INFO org.zaproxy.zap.extension.spiderAjax.SpiderThread - Finished Crawljax: API - Context: default
2024-01-18 09:07:42 156861 [ZAP-IO-Server-1-2] INFO org.parosproxy.paros.core.scanner.Scanner - scanner started
org.zaproxy.addon.network.internal.server.http.MainServerHandler - Failed to write/forward the HTTP response to the client: java.io.IOException: Broken pipe
?
Also, if it helps, I've tried with and without running the traditional spider before the AJAX spider - it makes no difference but for reference the traditional spider works just fine and returns results.
from zaproxy.
@thc202 - I think I've identified what's causing the 0
results above. I tested this with the GUI against http://testphp.vulnweb.com
and saw the same thing. It seems to be something to do with Chrome and the site being http
rather than https
.
I ran the Automation Framework test as suggested in the FAQ with https://demo.owasp-juice.shop
and the test passed with Chrome, but when I switched to http://testphp.vulnweb.com
it failed.
So I tried the same thing in the GUI and found that Chrome seems to crash when scanning http://testphp.vulnweb.com
:
Screen.Recording.2024-01-31.at.11.15.12.copy.mov
(Note: I'll attached a screen recording for FireFox separately because it's too big for one comment)
tl;dr - it works correctly with Firefox but exits / crashes(?) with Chrome.
This seems like a bug to me unless there is a setting I'm missing somewhere. Is Chrome short-circuiting on a certificate error perhaps?
from zaproxy.
And here's the Firefox recording:
Screen.Recording.2024-01-31.at.11.15.12.copy2.mov
(It jumps a bit in a few places because I had to cut it a bit to get it under 10MB).
from zaproxy.
Right I think I finally have this working more or less. I thought I'd detail the issues I hit and how I fixed them incase it is of use to anyone in the future.
Chrome + HTTP redirect
This is the one I faced above. It seems like Chrome automatically redirects HTTP to HTTPS, which was out-of-scope for the ZAP context configuration I had, and thus ended the session. Firefox doesn't do this. By updating the front of the inclusion regex to http[s]?
and making https
accepted but optional, everything seems to work.
I still think this is a bug because it forces me to allow a scheme I didn't want / need, but happy to continue with the workaround for now.
Multiple Chrome instances failing to start
The GUI and Automation Framework test both set Number of Browsers
to 1
, however, the API via Docker defaults to 16
. When trying to run the AJAX spider with Chrome, I found I kept getting this error:
org.openqa.selenium.NoSuchSessionException: invalid session id
I fixed this by following this advice and adding -config selenium.chromeArgs.arg.argument=--disable-dev-shm-usage
to my startup command.
Sandbox issue
Finally, I also had an issue with Chrome and Docker permissions, it looked a little like this:
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
This is an issue with Chrome trying to create a sandbox in Docker and not having the necessary permissions. You can fix this by adding -config selenium.chromeArgs.arg.argument=--no-sandbox
; however, this isn't advised for security reasons and instead I added my own security policy configuration (something akin to --security-opt seccomp=$(pwd)/chrome.json
) as advised here.
from zaproxy.
Thanks for following up and providing such great detail!!! Wonderful!
Re: The context regex, you can drop the square brackets the question mark will only apply to the preceding token (the s).
https://www.regexplanet.com/share/index.html?share=yyyyfpma03r
(Click 'Java')
from zaproxy.
Related Issues (20)
- OpenAPI import from URL timeout verbose/stacktrace HOT 4
- Error while running scan on ZAP Docker HOT 4
- zap-api-scan.py allow to use a URL when overriding hostname (`-O`) HOT 3
- MIssing icons HOT 19
- Check for updates failed HOT 2
- Webpages distinguished by URL fragment (site.com/#/abc) are neither visible nor recorded by ZAP HOT 2
- A Dev Container for ZAP based on the ZAP Stable Dockerfile HOT 3
- SSE add-on should handle empty "id" field more gracefully HOT 14
- Can ZAP docker scan multiple IPs and Ports? HOT 2
- Handle lack of disk space better HOT 4
- cannot close message box "failed to start Chrome browser" (Chrome is installed) HOT 2
- owasp/zap2docker-stable:latest - No manifest file causing build error on Jenkins HOT 7
- Fail to generate pdf report dues to SAXParseException HOT 3
- When i start the zap proxy pods without doing any hit i can see that zap proxy tries hitting some random sites HOT 2
- ZAP gui hanged when opening previous session on Winwows 10 x64 HOT 1
- ZAP 2.14 is not working with Oracle JDK 17 or 21, or with microsoft-openJDK11 but it is working fine with Oracle JDK 11 HOT 13
- API to View URLs should return past history HOT 4
- Internal error appears in ZAP when starting new autoscan after creating new session HOT 2
- Allow for modern and plus type of reports HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zaproxy.