.DS_Store parsing about zaproxy HOT 54 CLOSED

Released and announced: https://twitter.com/zaproxy/status/1628803269362429959

from zaproxy.

That looks promising! I look forward to testing it once ready.

By the way, I meant to write a blog post about this new feature, to explain what a .DS_Store file is, its internal structure, the Kaitai Struct project, etc. Is it something you would be interested in as a guest post on the ZAP blog?

from zaproxy.

That's awesome! 🥳

I'll try to have the blog post ready shortly.

from zaproxy.

This can be implemented as an add-on, the libraries would not be a big problem IMO. Have you checked their size?

from zaproxy.

@SkypLabs Thanks for reaching out, I had been looking into using that model but encountered a few build issues. I'll revive the branch and add some notes/questions here.

from zaproxy.

Line 662 of my compiled Java class is exactly the same as the one on the Format Gallery 😕 It seems that the problem comes from the Java compiler. I've already used the result of the Python compiler and it worked perfectly.

It would be worth reporting this issue upstream. I can take care of it if you want. For this purpose, could you please share the Java code you used for your tests?

from zaproxy.

I didn’t test, the warning comes from the IDE. I don’t recall what I changed previously to work around it. Because I didn’t get familiar enough with the code to get a useful result out of it previously.

This is my only exposure/use of anything kaitai related so probably better if someone more familiar reports it.

from zaproxy.

Hi @kingthorin,

if you have or do open a ticket for Kaitai can you let us know the link/details?

Of course. I haven't done it yet. I wanted to write a Java PoC to demonstrate the issue in my ticket but I have been busy with work and personal matters.

Also if you have a python PoC working that extracts content and builds URLs that'd help.

Sure, I can easily do that 👍

from zaproxy.

@SkypLabs any idea how to get some attention/movement on that? 😉

I wish, but it seems the ticket finally got some traction from project members! 🤞

from zaproxy.

Thanks for sticking with it. I appreciate all your help!!

from zaproxy.

There you go: https://gist.github.com/SkypLabs/bc5510838acf7d6a7e3562999091f4c0#file-app-java

It's a quick-and-dirty first implementation but it does the job.

from zaproxy.

That's wonderful! I just skimmed it quickly and it seems to hit all the high points, so hopefully from there I can implement or implement and tweak any edge cases.

Thank you very much!

from zaproxy.

It probably won't be an add-on itself, I plan to just add it as a new component to the spider. But I'll definitely let you know when I have something together (hopefully in the next week or so).

It'll probably be somewhat like the SvgHrefParser and add a seed similar to the robots.txt and sitemap.xml handling (Might also need to add it to the options/params, I'll have to discuss that with the team.)

Edit: Actually it'll be a different seed handling than robots and sitemap since they're root and this is more "everywhere". https://github.com/zaproxy/zap-extensions/blob/534f8a8da38d364e43cbd5c921d212feda117836/addOns/spider/src/main/java/org/zaproxy/addon/spider/Spider.java#L298-L313

from zaproxy.

I’ve got the functional bits well underway: https://github.com/kingthorin/zap-extensions/tree/ds-store

Just need to tackle the the user interface bits: options dialog/params, api support.

from zaproxy.

Definitely!

from zaproxy.

Analyzing the robots.txt file has been on my list for a while, but not thought about
analyzing .DS_Store files - that would be really neat.

Original issue reported on code.google.com by psiinon on 2010-12-18 10:19:38

• Labels added: Type-Enhancement
• Labels removed: Type-Defect

from zaproxy.

Hi fitblip,

robots.txt and .DS_Store files are sometimes a good source for hidden direcories and
such. So, i like your idea. Fact is, that there is already an extension for Andiparos,
but we haven't managed to integrate all Andiparos features into ZAP. Anyway, you could
copy those extensions and improve it if you feel like doing it... ;)

Thanks for feeding the issue list with some nice ideas

Cheers,
Axel

Original issue reported on code.google.com by a.c.neumann on 2010-12-18 10:21:38

from zaproxy.

Hi there,

Can this be split out so that this one is .DS Store, and create a new one for robots.txt
similar to the sitemap.xml I created earlier?

Original issue reported on code.google.com by vanderaj on 2012-07-20 12:37:15

from zaproxy.

Yes, that makes sense.
Changed this to just cover .DS_Store parsing, as it includes more info about it.
Issue 330 (nit of a coincidence;) now covers robots.txt

Original issue reported on code.google.com by psiinon on 2012-07-20 12:50:49

from zaproxy.

(No text was entered with this change)

Original issue reported on code.google.com by psiinon on 2012-07-20 12:51:04

from zaproxy.

Hi

Digininja discuss .DS_Store parsing too and he releases his tools FDB:
http://www.digininja.org/projects/fdb.php
Maybe this PERL module could be useful and added to ZAP.

Original issue reported on code.google.com by segu.info on 2013-07-24 14:01:07

from zaproxy.

Simon, I'd like to take a crack at this. I've looked at the passive parsers, and at
the very least, I reckon I could get the filenames out into a list that could then
be added to the sitemap. 

Original issue reported on code.google.com by vanderaj on 2014-04-11 03:22:49

from zaproxy.

Go for it :)

Anything you need from me?

Original issue reported on code.google.com by psiinon on 2014-04-11 15:27:45

from zaproxy.

http://doc.kaitai.io/lang_java.html

http://formats.kaitai.io/ds_store/java.html

from zaproxy.

I started to look into this based on the kaitai library and model. My first attempt wasn't successful, thinking maybe I just had a particularly fickle example file I ran it through an online parser which had no problem with it. (I did get the pieces setup, adjusted the spider filter, etc. so the ground work is there 👍).

Bringing in the kaitai library(ies) just for this might be overkill, I may look at just strings'ing it as originally suggested (something like http://www.java2s.com/Code/Java/Data-Type/Stringsextractprintablestringsfrombinaryfile.htm) or reimplementing one of the Open Source python etc based parsers in Java?

from zaproxy.

I had started with it in core, but I can look at re-doing things as an add-on. I wasn't as much concerned about the size of the libraries as their usefulness, since they didn't work as advertised 'out of the box' I figured it might be more simple to build something from scratch vs. trying to debug and revise whatever was broken with kaitai.

from zaproxy.

If that's faster/easier, sounds good.

from zaproxy.

Hi,

I am the author of the .DS_Store file description that you can find on the Kaitai Struct Format Gallery (http://formats.kaitai.io/ds_store/java.html). Let me know if I can be of any help here.

I've actually been thinking about writing a ZAP extension for this very purpose for a while now but I didn't realise that there was already a thread about it.

from zaproxy.

@SkypLabs in the generated format, line 662:
io.seek(_root.buddyAllocatorBody().blockAddresses().get((int) blockId()).offset());

Cannot cast from Long to int

Specifically on the blockId part.

I think I had previously worked around this but I had to re-implement things this morning (I guess I'd done away with the previous feature branch.)

from zaproxy.

Hey @kingthorin, sorry for my late answer.

I just compiled the ds_store.ksy description into a Java class without any issue:

$ cat /etc/debian_version
10.10

$ java --version
openjdk 11.0.12 2021-07-20
OpenJDK Runtime Environment (build 11.0.12+7-post-Debian-2deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.12+7-post-Debian-2deb10u1, mixed mode, sharing)

$ sudo apt install kaitai-struct-compiler
...

$ ksc --version
kaitai-struct-compiler 0.9

# Download the latest version of my DS_Store description.
$ wget https://raw.githubusercontent.com/kaitai-io/kaitai_struct_formats/master/macos/ds_store.ksy

$ ksc -t java ds_store.ksy

$ head DsStore.java
// This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild

import io.kaitai.struct.ByteBufferKaitaiStream;
import io.kaitai.struct.KaitaiStruct;
import io.kaitai.struct.KaitaiStream;
import java.io.IOException;
import java.util.Arrays;
import java.util.ArrayList;
import java.nio.charset.Charset;

from zaproxy.

Then, to parse a .DS_Store file in Java, you can follow the instructions on this page: https://formats.kaitai.io/ds_store/java.html.

I have published samples to parse in the PR that I submitted to the Kaitai Struct project: https://github.com/kaitai-io/kaitai_struct_formats/files/3015783/ds_store_samples.zip.

from zaproxy.

Could you make the created code available or replace the code on: http://formats.kaitai.io/ds_store/java.html (which is what I used and had the cast issue)

from zaproxy.

Could you make the created code available or replace the code on: http://formats.kaitai.io/ds_store/java.html (which is what I used and had the cast issue)

Right, I thought you couldn't compile the KSY file into a Java class. I misunderstood your issue.

Here is the Java class that I compiled: DsStore.zip. It might be the same as the one on the Format Gallery in the end.

from zaproxy.

@SkypLabs if you have or do open a ticket for Kaitai can you let us know the link/details?

Also if you have a python PoC working that extracts content and builds URLs that'd help.

from zaproxy.

@SkypLabs any news?

from zaproxy.

Ping @SkypLabs any news in this?

from zaproxy.

Hi @kingthorin,

Sorry about my delayed answer. I'm in the middle of a house move so I couldn't make progress on it. I am still willing to work on it though and I should have free time next week.

I'll keep you posted.

from zaproxy.

Thanks!

from zaproxy.

Hi @kingthorin,

I have written a small Java test project and I could reproduce your issue. I got the same error message as the one from your IDE.

I submitted a ticket to the Kaitai Struct project: kaitai-io/kaitai_struct#956

from zaproxy.

@SkypLabs thank you!

from zaproxy.

@SkypLabs any idea how to get some attention/movement on that? 😉

from zaproxy.

Also if you have a python PoC working that extracts content and builds URLs that'd help.

Sure, I can easily do that 👍

@SkypLabs it seems that the katai team has provided a fix/work around. I know you said earlier that the existing java class page provides instructions but it's still really unclear to me. If there was a working PoC of some sort I feel I'd be much better positioned to sort it out. (Getting the structure is clear'ish, I think I can turn an HTTP response into a byte array to be loaded, but interacting with it in useful ways isn't: I don't want to manipulate the file I just want to extract paths/filenames and use them.)

from zaproxy.

Hey @kingthorin. Sorry again for the late answer.

Yes I've seen their suggestion. I'll submit a PR to update my ds_store.ksy. Then I'll provide you a PoC to extract the file and directory paths from a .DS_Store in Java 👍

from zaproxy.

PR created: kaitai-io/kaitai_struct_formats#632

from zaproxy.

Wonderful, thanks very much!

from zaproxy.

FYI, the PR has been merged.

from zaproxy.

@SkypLabs sorry to be a pain, but any news on this?

Then I'll provide you a PoC to extract the file and directory paths from a .DS_Store in Java

Thanks in advance!

from zaproxy.

@SkypLabs sorry to be a pain, but any news on this?

Hi @kingthorin,

Sorry, I hadn't forgotten but I had difficulties finding the time to work on it. Furthermore, I hit a nasty bug when I tried to use the compiled KSY file. I submitted a patch to fix it: kaitai-io/kaitai_struct_formats#637.

I've started to write small code examples in both Python and Java. You can them here. I will try to finish them in the next coming days.

from zaproxy.

Thanks for sticking with it. I appreciate all your help!!

I'm as eager as you to see this feature land! I just struggle with time.

My fix has been merged: kaitai-io/kaitai_struct_formats#637

For your information, you can use Kaitai Struct's online IDE to interactively parse a .DS_Store file to get more familiar with its internal structure:

If you want to try it out yourself, you can find sample files in my original PR. And don't forget to manually apply my fix as the macos/ds_store.ksy file in the online IDE is not up to date.

from zaproxy.

Cool, thanks

from zaproxy.

Hi @kingthorin,

The Python implementation is complete: https://gist.github.com/SkypLabs/bc5510838acf7d6a7e3562999091f4c0#file-parser-py

I am now implementing the same logic in Java.

from zaproxy.

Cool, thanks!

from zaproxy.

Great! Keep me posted. I'd like to participate in the add-on creation.

from zaproxy.

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

from zaproxy.