Example parsers about nom HOT 97 OPEN

I wrote a Python parser: https://docs.rs/python-parser/

from nom.

A parser for Turtle. It passes the test suite in 15ms.

https://github.com/vandenoever/rome/tree/master/src/io/turtle

from nom.

I made a GameBoy ROM parser with nom5!
https://github.com/MarkMcCaskey/gameboy-rom-parser
https://crates.io/crates/gameboy-rom

It's extremely simple and doesn't do much, but the crate provides a useful abstraction over the metadata of GameBoy ROMs.

I'll add more optional validation functions to it and refactor my emulator's ROM code to use it soon.

edit:
this post is what inspired me to make this

from nom.

FYI I used nom to parse the linux perf data format (https://github.com/gz/rust-perfcnt/blob/master/src/linux/parser.rs) in case you want to add it. In comparison to most examples listed here it parses binary data.

Also, it's roughly 25x faster than an equivalent parser written in python ;)

from nom.

Yeah, I'm aware of the scale problem of Rust. I don't want to write that one, but I think it's a good holy grail for any parser library written in Rust.

As of version 0.10.0, syn is now able to parse practically all of Rust syntax. One of my test cases is to parse the entire github.com/rust-lang/rust repo into an AST and print it back out, asserting that the output is identical to the original.

I am technically not using nom but instead a fork which removes the IResult::Incomplete variant. I found that the extra macro code generated to handle Incomplete was more than doubling the compile time for something that I didn't even want. Nevertheless, the code is enough like nom that I think we can check off the box.

Example snippet to parse one arm of a match expression:

named!(match_arm -> Arm, do_parse!(
    attrs: many0!(outer_attr) >>
    pats: separated_nonempty_list!(punct!("|"), pat) >>
    guard: option!(preceded!(keyword!("if"), expr)) >>
    punct!("=>") >>
    body: alt!(
        map!(block, |blk| ExprKind::Block(BlockCheckMode::Default, blk).into())
        |
        expr
    ) >>
    (Arm {
        attrs: attrs,
        pats: pats,
        guard: guard.map(Box::new),
        body: Box::new(body),
    })
));

from nom.

A parser for the Mediawiki format would be quite useful.

from nom.

Correct me if I'm wrong, but the linked Redis project doesn't seem to use nom.

from nom.

I've started working on a parser for IP, TCP, UDP, and Ethernet headers. It is located at https://github.com/moosingin3space/pktparse-rs.
Warning: there is little to no documentation right now!

from nom.

Boxcars is an example of a Rocket League replay parser with serde serialization. Let boxcars be a good example of Rust code using nom, and serde as extensive examples are hard to come by. While lacking user friendly error message -- among other issues, tests and documentation strive to be thorough.

from nom.

@Geal thanks for an awesome library! I wrote a wavefront obj/mtl 3d mesh parser using it nom-obj, which I published to crates.io

from nom.

The BitTorrent example has been deleted, it seems.

from nom.

Nothing at all wrong with asking, and I'm sure someone might want to implement one at some point, but this is a list of example parsers written using nom, rather than a list of formats "supported" by nom. An xml parser would be an excellent idea for learning nom, imo.

from nom.

@Geal - I wanted to post my public suffix domain list parser that I wrote a few months back. I couldn't find a performant library that did what I needed, so I grabbed nom and went to work. https://github.com/dwerner/nom-psl

from nom.

Please add OpenCypher to the list... a nice way to parse Graph DB queries could enable a wave of innovation in databases. There are zero legit serverless / autoscaling or decentralized graph databases (like you'd get with a CRDT/ORDT backend for an OpenCypher parser). GunJS is fairly close but JavaScript is not ideal for storage IMHO

from nom.

@LikeLakers2

Hey there! I was wondering why the Rust parser on this list is syn? From what I can tell, syn does not use nom (although it might have in the past).

Since this is a list of examples of parsers built with nom, I don't see why we should be linking to syn here.

dtolnay/syn#476

syn was using nom until v0.15, this issue was created 3 years before syn dropped its usage of nom. That's why it's still linked here.

You're absolutely correct that it should be removed though.

from nom.

I'm writing a Thrift library for Rust that'll use Nom for both their IDL and the network protocol, so that can be another example (although in a different repo).

from nom.

Nice idea, that will be useful! Please notify me when it is done, I will add a link in this list.

from nom.

This looks interesting. Is anyone actively working on any of these parsers? I'd like to work on a few of these.

from nom.

I have some code for a GIF one at https://github.com/Geal/gif.rs but it is hard to test, since the graphical tools in Piston change a lot.

You can pick any of them. Network packets may be the easiest, since they don't require a decompression phase.

I am using the gif example to see what kind of API can be built over nom. Most of the parsing example are done as one pass over the data, but often there is some logic on the side, and it is not easy to encode correctly.

from nom.

I've started a fastq (http://en.wikipedia.org/wiki/FASTQ_format) parser https://github.com/elij/fastq.rs

from nom.

@elij this is a great idea! Was it easy to do?

from nom.

yup it's a great framework -- though I struggled a bit with eof so I borrowed some code from rust-config (https://github.com/elij/fastq.rs/blob/master/src/parser.rs#L69) -- is there a better solution?

from nom.

yes, eof should be a parser provided by nom, I am just waiting for @filipegoncalves to send a PR 😉

from nom.

Hah, sorry for my silence. I've been busy lately. I just sent a PR (#31).

I will be working on one of these example parsers as soon as I get some spare time. There are some great ideas in here!

from nom.

I might give tar a try

from nom.

Does this check off PCAP?

https://github.com/richo/pcapng-rs

from nom.

pcap-ng and pcap are two different formats, right? It seems the consensus now is to move everything to pcap-ng, though.

from nom.

I will try a FLAC parser, need to add quite a few things for it though.

from nom.

ISO8601 is done in https://github.com/badboy/iso8601 (I hope it's mostly correct.)

from nom.

ok, it should be up to date. More to come 😄

from nom.

WARC file format released. https://crates.io/crates/warc_parser

from nom.

@sbeckeriv great, thanks!

from nom.

It might be informative to try parsing the rust grammar with nom, if nobody has yet. In any case, I'd like to see a few programming languages on that list, since that's my use case.

from nom.

@porglezomp programming languages examples would definitely be useful, but the Rust grammar might be a bit too much for the first attempt. Which other languages would you like to handle?

from nom.

Yeah, I'm aware of the scale problem of Rust. I don't want to write that one, but I think it's a good holy grail for any parser library written in Rust. I'd like to try parsing the Lua grammar first, I think.

I recommend adding to the list:

• Programming Languages
  • Rust
  • Lua (I'll do this)
  • Python (or some other whitespace significant language)
  • C

from nom.

ok, I added them to the list :)

from nom.

You have INI marked as done; do you have a link to it? (I'd love to use this for some tooling I'm hoping to build in 2016; need a good non-trivial example for it, though.)

from nom.

@chriskrycho: https://github.com/Geal/nom/blob/master/tests/ini.rs

from nom.

Thanks very much, @badboy!

from nom.

I'll try to make the TOML parser very soon.

from nom.

Actually, I think I should rewrite that INI parser, now that more convenient combinators are available.
Also, I should really work on that combinator for space separated stuff

from nom.

@fbernier great! Please keep me posted!

from nom.

Maybe add a simple example for trailing commas in lists? Python has those, but is quite complex. Can't think of a simple example though.

from nom.

That IRC example is no longer using nom. The parser was moved into its own repository: https://github.com/Detegr/RBot-parser

from nom.

@l0calh05t to parse something like [a,b,c,] or [a,b,c] ?
@johshoff fixed, thanks

from nom.

@Geal yes, exactly

from nom.

@l0calh05t for [a,b,c], you can parse with delimited!(char!('['), separated_list!( char!(','), alphabetic), char!(']')).
For [a,b,c,], you can have delimited!(char!('['), terminated!(alphabetic, char!(',')), char!(']')).

A parser that would handle both cases is much trickier.

from nom.

Both is really the more interesting case. And what is needed for Python for example

from nom.

Could there be something like maybe_char!(',') which would read a char, consume it if it's ',' or backtrack if it isn't?

EDIT: actually that's probably what opt!(char!(',')) would do, so you just have to take the one that parses [a,b,c] and stick that before the ']' or am I missing something?

from nom.

Problem is that wont work unless a look ahead of more than one character is added automatically

from nom.

In fact, it is easier than I thought, but requires some work:

preceded!(
  char!('['),
  terminated!(
    separated_list!(
      char!(','),
      alphabetic
    ),
    terminated!(
      opt!(char!(',')),
      char!(']')
    )
  )
)

opt! will return an option of the result of its child parser (Some if success, None if failure), so it will accept the trailing comma.

from nom.

I wrote a simplistic bencode parser: nom-bencode.

Not sure if it covers everything (yet).

from nom.

I've started a TOML parser as a learning project: https://github.com/passcod/noml

from nom.

At this point I'm ready to share my flac implementation as an example parser.

from nom.

Hey sourrust. Nom looks great way to do this.
I am interested in parsing different video formats with nom. If there is some existing rust kibs in this space that anyone knows then u could start porting some to nom. Worth a crack to see how it goes.

I am very curious about using the streaming capabilities of nom. For my use case I want to stream data between servers, manipulate frames, and then fan it back into the main stream.
I would love to get some feedback on some potential gotchas.
Doing this type of work should ultimately feedback into making nom better.

from nom.

I agree, I checked the history of it's Cargo.toml and at no point was nom listed as a dependency. I'm not sure how it ended up on the list, but it looks like it should be taken off.

from nom.

It does in another branch, which is still not merged because time.

from nom.

@Geal you can remove my Thrift library as an example as I'm no longer using Nom in it.

from nom.

I've released a TOML parser. It doesn't let you modify everything possible in the document or create documents from scratch, but does correctly parse TOML, report errors, allow some modification and then output the document with comments and whitespace intact.

from nom.

Java class file parser! It is part of a larger class project.

The parser uses helper macros based on #160 to get more backtracking support.

from nom.

Not sure if it's worth putting here or not, but I'm using nom to parse strings for the tracery library I am writing for rust: https://github.com/pwoolcoc/tracery-rs

from nom.

A subset of C, namely C literals and expressions: https://crates.io/crates/cexpr

from nom.

@dtolnay syn is an amazing example, thanks for your hard work :)

from nom.

@dtolnay could I get your input on #356? It might fix your issues with compile times, so I'd like to get your thoughts on this.

from nom.

I am writing a PDF library using nom to parse PDF syntax. Released v0.1.0 just now.
https://github.com/J-F-Liu/lopdf

from nom.

So I've implemented a EDI parser for the ANS standard EDI for work with this. Awesome library really useful. Sadly that's owned by my employer.

I've started implementing an x64 assembler with nom. I'm really struggling with writing the parser. The main reason is register names have a lot of overlap, and are very short. For example r8, r8w, r11, and r12d. Ideally I want to map these to an enum. map!() makes this easy, but how can I match those terms in nom?

from nom.

I converted several "keys" to enum values in my brainfuck parser, might or might not be relevant to your needs. See the first parsers defined with "named!" https://github.com/Keruspe/brainfuck.rs/blob/master/src/parser.rs

from nom.

is there a way (or it would be great if it's possible) to generate EBNF from this? Great package BTW ...

from nom.

Hi,
I just pushed a pcap parser : https://github.com/ithinuel/pcap-rs.
It still needs the PR #492 to be merge so it can use official nom crate.

Any feedback is welcome.

from nom.

I wrote a parser for the simple key/value text format .properties, which is a standard for Java configuration files. It uses nom 3.1. Can it be added to the list?

This is the first parser I wrote using a Parser Combinator library. If anyone can review my code I would be delighted. Also, I tried to add error reporting to my code, but I gave up after I tried to insert add_return_error and return_error calls all over the place to no avail (in the branch "error-reporting"). Is there an example of a text parser that reports parsing errors?

Edit: I rewrote my library using Pest instead of Nom, as I find it more suited to parsing a text format. I will definitely use nom if I need to parse a binary format, though.

from nom.

@Geal thanks for this library.
I've implemented a parser for URI's which is
part of a larger side project for RDF (n3, ttl,...) parsers. The full abnf of rfc 3986 is implemented but the pct-encoding is still a bit messy.

from nom.

Here's a parser for ICE candidates SDP (RFC 5245), used for example in WebRTC: https://github.com/dbrgn/candidateparser

from nom.

I wrote a Session Initiation Protocol (RFC3261) low-level push parser with API inspired by seanmonstar/httparse (hyper's HTTP parser):
https://github.com/kamarkiewicz/parsip

from nom.

I'd be interested in something that could parse SNMP MIB and YANG.

https://en.wikipedia.org/wiki/YANG

from nom.

As a beginner in Rust world, I'm quite sure I will say something horribly wrong, but is there any planned support for some XML dialects ? (typically RSS/ATOM) ?

from nom.

@Riduidel if you're specifically interested in just having parsers for those formats, look at https://github.com/rust-syndication. I don't think there's any nom involved there though.

from nom.

HTTP: https://github.com/hjr3/weldr/blob/00481f80ae60bd6b312805245c126c168ab77b36/src/http/parser.rs

from nom.

I think Redis database file format parser is not using nom at all. I couldn't find any reference to nom anywhere.

from nom.

@idursun Maybe it refers to this old branch from a year before the last update to master. https://github.com/badboy/rdb-rs/tree/nom-parser

from nom.

is there any SQL parser？

from nom.

is there any SQL parser？

it'd seem better to me to import it to an sql engine and interact with that data using Diesel. parsing flat sql files seems very limited.

instead of writing a one-off Rust app to do this, you could add diesel bindings to Torchbear, see jazzdotdev/jazz#85 , then make a Speakeasy library for transporting data from your schema using content model in ContentDB.

then, you could develop a lot further beyond.

from nom.

@naturallymitchell maybe @saggit was simply looking for something to extract some data from a raw sql dump. Like a one-off log analysis tool. :D

from nom.

It's extremely simple and doesn't do much, but the crate provides a useful abstraction over the metadata of GameBoy ROMs.

@MarkMcCaskey It could even make sense to refactor it then into a generalized library with config files (like, TOML and YAML, and now SANE). Do you think that'd be too much more work?

from nom.

@naturallymitchell

Do you mean specifying the layout of the bytes as data and creating a dynamic data structure from it? That's an interesting idea, but I don't think it'd be too helpful for my use case -- as I see it, the primary value-add of the gameboy rom parser is the data layer that it exposes, which lets the user get things like the game's title as as string or the exact cartridge type and how much ROM and RAM it has as well-named, plain Rust values.

The parser may be implementable with serde deserialize on a repr(C) struct though, which is kind of the reverse of what you're saying, I think... I'm not familiar enough with how serde-derive handles errors though.

from nom.

Just got a 0.0.1 version of an NMEA-0183 parser using nom 5 https://github.com/YellowInnovation/nmea-0183 . I need to have a look at the docs and guidelines (the code is ugly for now) and refactor it :) I hope to submit a pull request adding a clean version of it to the parsers list soon ! :)

from nom.

This is SIP parser
https://github.com/armatusmiles/sipcore/tree/master/crates/sipmsg

Torture test: kurotych/sipcore@32040e5 ( https://tools.ietf.org/html/rfc4475#section-3.1.1.1 )

from nom.

@armatusmiles thanks, i added it to the list in 2e58a2c

from nom.

Wrote a UBJSON parser w/ nom
Pretty early version with just parsing, but it does the job.

https://github.com/OtaK/ubjson

https://crates.io/crates/ubjson

from nom.

There is a PDF parser here: https://github.com/J-F-Liu/lopdf (it requires using the nom_parser feature).

FWIW, with lopdf, the nom parser is much faster than the default parser

from nom.

I wrote a tool with its own programming language using nom. here is source repo.

from nom.

The gds2-parser released at https://crates.io/crates/gds2_io. BTW, my pull request tag is #1497

from nom.

would it be feasible to write an ecmascript/typescript parser with nom as well? Or would the scope be too big for that?

from nom.

I have written 2 (public) parsers using nom which may be used as examples:

• BitTorrent/bencoding: https://github.com/alexrsagen/rustorrent/blob/4076d0ea689a950021164d2fdd412021519e7c68/src/bencode.rs
• BitTorrent/torrent files (metainfo) parsing using bencode.rs: https://github.com/alexrsagen/rustorrent/blob/4076d0ea689a950021164d2fdd412021519e7c68/src/torrent/metainfo.rs
• MS-CFB: https://github.com/alexrsagen/rs-nomcfb (probably not a perfect implementation, but usable to parse Outlook .msg files and as a basic example for how to write parsers using nom)

from nom.

I made a bencode parser (the format used by .torrent files), https://github.com/edg-l/nom-bencode/

from nom.

Hey there! I was wondering why the Rust parser on this list is syn? From what I can tell, syn does not use nom (although it might have in the past).

Since this is a list of examples of parsers built with nom, I don't see why we should be linking to syn here.

from nom.

mdict-parser is a parser library for .mdx dictionary format file
https://github.com/eatgrass/mdict-parser

from nom.

crussmap is a parser library and tool for .chain file format

https://github.com/wjwei-handsome/crussmap

from nom.