Giter Club home page Giter Club logo

Comments (9)

acardnell-intruder avatar acardnell-intruder commented on June 18, 2024 1

Thanks for following up and providing such great detail!!! Wonderful!

Re: The context regex, you can drop the square brackets the question mark will only apply to the preceding token (the s). https://www.regexplanet.com/share/index.html?share=yyyyfpma03r (Click 'Java')

Ah yes - nice idea, thanks!

from zaproxy.

thc202 avatar thc202 commented on June 18, 2024

Did you try the weekly? That one has a newer Selenium version.

from zaproxy.

acardnell-intruder avatar acardnell-intruder commented on June 18, 2024

@thc202 I hadn't - I've just tried it now. I'm no longer receiving any errors, but it also doesn't seem to the spidering correctly either. Polling for AJAX results returns 0, and when it eventually completes and I try to run an Active scan (after running the Traditional + AJAX spider) I get the following crash error:

2024-01-17 19:38:29 59953 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Running Crawljax (with chrome-headless): API - Context: default
2024-01-17 19:38:29 59969 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Starting proxy...
2024-01-17 19:38:29 59984 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Proxy started, listening at port [35473].
2024-01-17 19:38:30 60283 [ZAP-AjaxSpiderApi] INFO  com.crawljax.core.plugin.Plugins - Loaded org.zaproxy.zap.extension.spiderAjax.SpiderThread$DummyPlugin@6fd3216e as a OnBrowserCreatedPlugin
2024-01-17 19:39:46 136260 [ZAP-AjaxSpiderApi] INFO  com.crawljax.core.CrawlController - Received shutdown notice. Reason is Exausted
2024-01-17 19:39:46 136382 [ZAP-AjaxSpiderApi] INFO  com.crawljax.core.CrawlController - Shutdown process complete
2024-01-17 19:39:46 136382 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Stopping proxy...
2024-01-17 19:39:46 136399 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Proxy stopped.
2024-01-17 19:39:46 136408 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Finished Crawljax: API - Context: default
2024-01-17 19:39:51 141656 [ZAP-IO-Server-1-2] INFO  org.parosproxy.paros.core.scanner.Scanner - scanner started
2024-01-17 19:39:51 141789 [ZAP-IO-Server-1-2] ERROR org.zaproxy.zap.extension.api.API - Exception while handling API request:
2024-01-17 19:39:51 java.util.ConcurrentModificationException: null
2024-01-17 19:39:51 at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) ~[?:?]
2024-01-17 19:39:51 at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.zap.extension.ascan.ActiveScanAPI.handleApiView(ActiveScanAPI.java:1035) ~[zap-D-2024-01-16.jar:D-2024-01-16]
2024-01-17 19:39:51 at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:557) ~[zap-D-2024-01-16.jar:D-2024-01-16]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleApiRequest(ZapApiHandler.java:111) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleRequest(ZapApiHandler.java:85) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleMessage(ZapApiHandler.java:70) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.MainServerHandler.notifyMessageHandlers(MainServerHandler.java:151) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.MainServerHandler.processMessage(MainServerHandler.java:131) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.LocalServerHandler.processMessage(LocalServerHandler.java:67) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.MainServerHandler.process(MainServerHandler.java:94) ~[?:?]
2024-01-17 19:39:51 at org.zaproxy.addon.network.internal.server.http.MainServerHandler.lambda$channelRead0$0(MainServerHandler.java:82) ~[?:?]
2024-01-17 19:39:51 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2024-01-17 19:39:51 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2024-01-17 19:39:51 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [network-beta-0.14.0.zap:?]
2024-01-17 19:39:51 at java.lang.Thread.run(Thread.java:829) [?:?]

from zaproxy.

thc202 avatar thc202 commented on June 18, 2024

Which API call are you using for the AJAX Spider results?

For the Active Scan call that's a bug but calling the endpoint again it would work.

from zaproxy.

acardnell-intruder avatar acardnell-intruder commented on June 18, 2024

@thc202 - I'm polling the ajaxSpider.number_of_results endpoint, which always returns 0. I just double-checked in the Python console as well and got the following for a ZAP instance after running the AJAX spider with Chrome:

scanner.zap.ajaxSpider.number_of_results
Out[5]: '0'
scanner.zap.ajaxSpider.results()
Out[6]: []

The Active Scan seems to work this morning 🤷

Edit: Just realised there is an extra couple of error messages in the ZAP logs this morning:

2024-01-18 09:06:04 58454 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Running Crawljax (with chrome-headless): API - Context: default
2024-01-18 09:06:04 58510 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Starting proxy...
2024-01-18 09:06:04 58535 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Proxy started, listening at port [38749].
2024-01-18 09:06:05 59392 [ZAP-AjaxSpiderApi] INFO  com.crawljax.core.plugin.Plugins - Loaded org.zaproxy.zap.extension.spiderAjax.SpiderThread$DummyPlugin@2be6d0f0 as a OnBrowserCreatedPlugin
2024-01-18 09:06:25 79574 [ZAP-IO-EventExecutor-4-2] WARN  org.zaproxy.addon.network.internal.server.http.MainServerHandler - Failed to write/forward the HTTP response to the client: java.io.IOException: Broken pipe
2024-01-18 09:06:45 99547 [ZAP-IO-EventExecutor-4-2] WARN  org.zaproxy.addon.network.internal.server.http.MainServerHandler - Failed to write/forward the HTTP response to the client: java.io.IOException: Broken pipe
2024-01-18 09:07:34 148562 [ZAP-AjaxSpiderApi] INFO  com.crawljax.core.CrawlController - Received shutdown notice. Reason is Exausted
2024-01-18 09:07:34 148918 [ZAP-AjaxSpiderApi] INFO  com.crawljax.core.CrawlController - Shutdown process complete
2024-01-18 09:07:34 148920 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Stopping proxy...
2024-01-18 09:07:34 148978 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Proxy stopped.
2024-01-18 09:07:34 149017 [ZAP-AjaxSpiderApi] INFO  org.zaproxy.zap.extension.spiderAjax.SpiderThread - Finished Crawljax: API - Context: default
2024-01-18 09:07:42 156861 [ZAP-IO-Server-1-2] INFO  org.parosproxy.paros.core.scanner.Scanner - scanner started

org.zaproxy.addon.network.internal.server.http.MainServerHandler - Failed to write/forward the HTTP response to the client: java.io.IOException: Broken pipe?

Also, if it helps, I've tried with and without running the traditional spider before the AJAX spider - it makes no difference but for reference the traditional spider works just fine and returns results.

from zaproxy.

acardnell-intruder avatar acardnell-intruder commented on June 18, 2024

@thc202 - I think I've identified what's causing the 0 results above. I tested this with the GUI against http://testphp.vulnweb.com and saw the same thing. It seems to be something to do with Chrome and the site being http rather than https.

I ran the Automation Framework test as suggested in the FAQ with https://demo.owasp-juice.shop and the test passed with Chrome, but when I switched to http://testphp.vulnweb.com it failed.

So I tried the same thing in the GUI and found that Chrome seems to crash when scanning http://testphp.vulnweb.com:

Screen.Recording.2024-01-31.at.11.15.12.copy.mov

(Note: I'll attached a screen recording for FireFox separately because it's too big for one comment)

tl;dr - it works correctly with Firefox but exits / crashes(?) with Chrome.

This seems like a bug to me unless there is a setting I'm missing somewhere. Is Chrome short-circuiting on a certificate error perhaps?

from zaproxy.

acardnell-intruder avatar acardnell-intruder commented on June 18, 2024

And here's the Firefox recording:

Screen.Recording.2024-01-31.at.11.15.12.copy2.mov

(It jumps a bit in a few places because I had to cut it a bit to get it under 10MB).

from zaproxy.

acardnell-intruder avatar acardnell-intruder commented on June 18, 2024

Right I think I finally have this working more or less. I thought I'd detail the issues I hit and how I fixed them incase it is of use to anyone in the future.

Chrome + HTTP redirect

This is the one I faced above. It seems like Chrome automatically redirects HTTP to HTTPS, which was out-of-scope for the ZAP context configuration I had, and thus ended the session. Firefox doesn't do this. By updating the front of the inclusion regex to http[s]? and making https accepted but optional, everything seems to work.

I still think this is a bug because it forces me to allow a scheme I didn't want / need, but happy to continue with the workaround for now.

Multiple Chrome instances failing to start

The GUI and Automation Framework test both set Number of Browsers to 1, however, the API via Docker defaults to 16. When trying to run the AJAX spider with Chrome, I found I kept getting this error:

org.openqa.selenium.NoSuchSessionException: invalid session id

I fixed this by following this advice and adding -config selenium.chromeArgs.arg.argument=--disable-dev-shm-usage to my startup command.

Sandbox issue

Finally, I also had an issue with Chrome and Docker permissions, it looked a little like this:

Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted

This is an issue with Chrome trying to create a sandbox in Docker and not having the necessary permissions. You can fix this by adding -config selenium.chromeArgs.arg.argument=--no-sandbox; however, this isn't advised for security reasons and instead I added my own security policy configuration (something akin to --security-opt seccomp=$(pwd)/chrome.json) as advised here.

from zaproxy.

kingthorin avatar kingthorin commented on June 18, 2024

Thanks for following up and providing such great detail!!! Wonderful!

Re: The context regex, you can drop the square brackets the question mark will only apply to the preceding token (the s).
https://www.regexplanet.com/share/index.html?share=yyyyfpma03r
(Click 'Java')

from zaproxy.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.