Sometimes, Result::unwrap is used, which may panic if the result is an error. Instead, proper error handling should be implemented.

• Waveform Detail Tag
• Waveform Color Preview Tag
• Waveform Color Detail Tag
• Song Structure Tag

Attempting to parse Tracks with an ISRC string break the parser because their format falls out of the norm. See the following for more details: https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html#long-utf16le

@Swiftb0y is working on a PoC: #39 (comment)

hi holzhaus!
when scrolling the library I noticed that some of the structs members for example PlaylistEntry are only visible in the crate (not pub).
may I suggest making (useful) struct members to public? that would probably make it easier to use the library.
greetings!

We already have a dependency on the modular_bitfield crate, might make sense to use that instead:

#[bitfield]
pub struct PageFlags {
    unknown1: B1,
    no_data: bool,
    unknown2: B6,
}

Originally posted by @Holzhaus in #93 (comment)

Sometimes, we know the exact byte size of a section (especially in anlz.rs). In those cases, it should be check if that exact size is actually parsed (not more or less).

Basic serialization is already supported for analysis/setting files, but not PDB files. When #64 has been merged, we can look into implementing support for this.

Note that the PDB parser uses FilePtr which does not support serialization out of the box yet (see jam1garner/binrw#4), so we'll have to provide a custom implementation.

We should think about proper error handling facilities. Currently we are just passing along the errors that our parser libraries (nom, binrw) generate. IMO these are dependencies that should not appear in public interface, since that forces a hard transitive dependency for the library consumer, which in turn would make changing internals (restructuring parsing/serialization) a breaking change.

https://blog.yoshuawuyts.com/error-handling-survey/
https://nick.groenen.me/posts/rust-error-handling/

Related issue on the crate-digger repo: Deep-Symmetry/crate-digger#22

Unfortunately I'm getting a ParseError when trying to get an ANLZ struct (happens with multiple .dat files).

i attached the file so you can hopefully reproduce this error.

ANLZ0000_error.zip

In the long run, it would be nice if we could have some diagnostics for the Unknown case. This would help catch edge cases of values we don't know.

We need a way to handle partial failure like this. Lots of the other unknown fields aren't arbitrary, but for example just seem to contain a static value most of the time. It would be nice to have a way of (de-)serializing them while still conveying that the contained value was differing from the expected one. This could help tremendously in helping to find databases where the usual assumptions don't apply.

One possible way to deal with this is adding a new method (e.g. checkForUnexpectedData() -> Result<(), Error>) that looks for unexpected data in fields. But that is something for another PR.

Originally posted by @Swiftb0y and @Holzhaus in #34 (comment)

Is there non-code documentation for this format or did you reverse engineer this just now yourself?

maybe change this:
$ cargo run -- dump-pdb data/complete_export/demo-tracks/PIONEER/rekordbox/export.pdb
to this:
$ cargo run -- dump-pdb data/complete_export/demo_tracks/PIONEER/rekordbox/export.pdb

p.s. nice work, nice project!

While nom can result in quite elegant code IMO, it only gives us deserialization. If we want to add serialization, we have to write at least an equal amount of code that does the manual serialization part. For a large part of the code base, this essentially consists of reversing match statements with little but repetitive differences spread throughout. I'm proposing to use the binrw crate to eliminate some of that boilerplate.

use binrw::binrw; 
#[derive(Debug, PartialEq)]
#[binrw]
#[brw(repr = u8)]
enum Test {
    A = 1,
    B = 2,
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn valid() {
        use binrw::{BinRead, io::Cursor};
        let mut cursor = Cursor::new(&[1]);
        let test = Test::read(&mut cursor).unwrap();
        assert_eq!(test, Test::A);
    }
    #[test]
    #[should_panic]
    fn invalid() {
        use binrw::{BinRead, io::Cursor};
        let mut cursor = Cursor::new(&[0]);
        Test::read(&mut cursor).unwrap();
    }
}

for more complex data_structures, binrw still has options for writing manual readers and writers.

Moreover, writing code based on kaitai specs will also be almost as easy since the resulting macro-annotated datastructures look very similar to a kaitai spec file IMO.

I'm seeing this error when trying to parse the export.pdb file from mixxxdj/mixxx#10923 (comment).

Table 3: Albums
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value:
 ╺━━━━━━━━━━━━━━━━━━━━┅ Backtrace ┅━━━━━━━━━━━━━━━━━━━━╸

 0: Error: no variants matched at 0x8f10...
   ╭───────────────────────┄ Album ┄────────────────────┄
   ┆
   ┆ 0: Error: no variants matched at 0x8f25...
   ┆   ╭───────────────────────┄ ShortASCII ┄────────────────────┄
   ┆   ┆
   ┆   ┆header & 0b1 == 1 at 0x8f25
   ┆   ╰─────────────────────────────────────────────────────────┄
   ┆   ╭───────────────────────┄ Long ┄────────────────────┄
   ┆   ┆
   ┆   ┆bad magic at 0x8f28: 111
   ┆   ╰───────────────────────────────────────────────────┄
   ┆    ...While parsing field 'self_0' in DeviceSQLString
   ┆     at src/pdb/string.rs:54
   ┆ 1: While parsing field 'name' in Album
   ┆     at src/pdb/mod.rs:432
   ┆ 2: While parsing field 'self_0' in Row::Album
   ┆     at src/pdb/mod.rs:821
   ┆
   ╰────────────────────────────────────────────────────┄

I stumbled upon this repo, while researching about rekordbox internals. My idea was to write a rekordbox TUI in Rust and your library is an essential part of building such an application, so my questions are:

• Do you work around the documentation about the pdb and anlz files, reverse engineered by the deepsymmetry guys (this)? Skimming over the code, it looks like it.
• What is the current state of the project? What is already implemented, what is missing and do you have some sort of roadmap?
• How can i contribute? I am fairly new to rust.

Edit: Something, that would be important to me is, exposing the internal structs and types in order to build up a database from fresh without parsing an existing database. Do you plan to support something like that?

Broad TODO list item for the near future. This would help discover inconsistencies between the spec and the implementation.

I'm looking for a tool that lets me automate some of my Rekordbox library stuff but I couldn't clearly see whether this library can only read or of it can also write back, at least to edit what song is in what playlist.

Even if it's not possible, thank you for making this!

As per Deep-Symmetry/crate-digger#32

Needs fixing

Apparently the playlist entry mechanism is more complicated than it seems.

Please check if this is correct:
Playlists with track count

🗀 15.07.22
    🗎 riot code (3 tracks)
🗀 jaki_21_05_22
    🗎 jaki_melow (18 tracks)
    🗎 jaki2 (29 tracks)
    🗎 jaki (19 tracks)
🗀 18.05.22
    🗎 plaid - dial p (15 tracks)
    🗎 otherstuff (10 tracks)
    🗎 wettrax (33 tracks)
    🗎 kumo99 (14 tracks)
🗎 breakcore_gabba (3 tracks)
🗎 10.05.22 (23 tracks)
🗀 grime_hardwave
    🗎 maindir (24 tracks)
    🗎 TMSV - Jungle- Vol. 2 (3 tracks)
    🗎 ian shirtman - The Shirt Tape (7 tracks)
    🗎 Deadcrow - Transmission (3 tracks)
    🗎 Deadcrow - Lies (1 tracks)
    🗎 Deadcrow - 2015-2017 (9 tracks)
    🗎 BEAT PVP - BEAT PVP 1ª EDIÇÃO (16 tracks)
    🗎 Vinnie Grapes - Radiohooligans (8 tracks)
🗀 loststuff
    🗎 maindir (23 tracks)
    🗎 Tommy Holohan & Casper Hastings (4 tracks)
    🗎 The Horrorist (11 tracks)
    🗎 peryl (4 tracks)
    🗎 Pavel K. Novalis (1 tracks)
    🗎 Mode In Gliany (7 tracks)
    🗎 Mass-X-Odus (1 tracks)
    🗎 Kas_st (1 tracks)
    🗎 Jokasti & Nek (1 tracks)
    🗎 Jacidorex (1 tracks)
    🗎 Homemade Weapons & Red Army (1 tracks)
    🗎 Homemade Weapons & Gremlinz (1 tracks)
    🗎 Homemade Weapons (10 tracks)
    🗎 Hephaistos (1 tracks)
    🗎 FSOM (23 tracks)
    🗎 FRACTIONS (6 tracks)
    🗎 B2 (1 tracks)
    🗎 Endlec (1 tracks)
    🗎 Danilo Incorvaia (4 tracks)
    🗎 V_A_ Lars Huismann & Chris Koegler, Alignment, T (1 tracks)
    🗎 Ansome (1 tracks)
🗀 oldfolders
    🗎 Houndstooth (11 tracks)
    🗎 Jiri Ceiver (25 tracks)
    🗎 KiNK (15 tracks)
    🗎 halloween (20 tracks)
    🗎 Justin Berkovi (71 tracks)
    🗎 jackwho (18 tracks)
    🗎 HTH vs HTH (7 tracks)
    🗎 maindir (49 tracks)
    🗎 [tiva011] Justin Berkovi - Gravel Heart EP (4 tracks)
    🗎 _IGOR (2 tracks)
    🗎 acephale (26 tracks)
    🗎 15.12.2017 (7 tracks)
    🗎 v-tracks - limits (5 tracks)
    🗎 Vtracks (5 tracks)
    🗎 Vectors 3 (8 tracks)
    🗎 VA-Phuture_Beats_Vol._1-(DCD089-41802)-2CD-FLAC-1995-dL (40 tracks)
    🗎 va - kids (soundtrack) (13 tracks)
    🗎 usbcd (11 tracks)
    🗎 URBAN HOMES - Centres (5 tracks)
    🗎 Tube Jerk (20 tracks)
    🗎 this machine is broken (14 tracks)
    🗎 The Sound of Belgium (60 tracks)
    🗎 Terence_Fixmer-Depth_Charged (10 tracks)
    🗎 telex - remixes (12 tracks)
    🗎 Studio672 (42 tracks)
    🗎 Sound Architecture (10 tracks)
    🗎 schnoerres (5 tracks)
    🗎 RussianTechno (55 tracks)
    🗎 Rush Hour (12 tracks)
    🗎 Quarks - Rehmix (2000) (14 tracks)
    🗎 Physically Sick - Physically Sick (42 tracks)
    🗎 Point B - Suicide Beauty Spot (14 tracks)
    🗎 Physically Sick 2 - Physically Sick 2 (43 tracks)
    🗎 Palette_All-Stars--After_School_Special-(PAL051)-WEB-2008-dh (3 tracks)
    🗎 Mr Ozio - Lambs Anger (17 tracks)
    🗎 nachtigal (26 tracks)
    🗎 newstuff (7 tracks)
    🗎 nachtigal2 (38 tracks)
    🗎 Mr.Oizo-Analog Worms Attack_224kbps_mFY (15 tracks)
    🗎 marcel (6 tracks)
    🗎 mix (10 tracks)
    🗎 MMODEMM (72 tracks)
    🗎 Missile Recordings (16 tracks)
    🗎 Marcus Zils Musik (41 tracks)
    🗎 Lena Willikens - Phantom Dalia Ep (6 tracks)
    🗎 Kate Tempest-Everybody Down-Paul'sBoutique (11 tracks)
    🗎 Magazine 18 DigiFiles (4 tracks)
    🗎 koelncampus (13 tracks)
    🗎 Lorenzo Senni - Persona (2016) (6 tracks)
    🗎 khmparty (8 tracks)
    🗎 Koenigleopold-Eure_Armut_Kotzt_Mich_An-AT-2013-BCC (12 tracks)
    🗎 Koenigleopold-Eure_Armut_Kotzt_Mich_An (12 tracks)
    🗎 Jeandado - Globale Welt (13 tracks)
    🗎 Hotline Miami OST (23 tracks)
    🗎 Helena Hauff - Discreet Desires (10 tracks)
    🗎 hinterhof (19 tracks)
    🗎 Hotline Miami 2 - Soundtrack (2015) 1 (20 tracks)
    🗎 Gold+Beton 02 (10 tracks)
    🗎 group A - INITIATION (9 tracks)
    🗎 hafen (7 tracks)
    🗎 Gewoelbe (18 tracks)
    🗎 Halloween2 (57 tracks)
    🗎 gewoelbe217 (6 tracks)
    🗎 btf_sommerfest2 (44 tracks)
    🗎 DrumnBass (229 tracks)
    🗎 Good Stuff 7 (11 tracks)
    🗎 gold+beton (9 tracks)
    🗎 Comeme (17 tracks)
    🗎 btf_neo (58 tracks)
    🗎 Cristian Vogel (84 tracks)
    🗎 game tracks (1 tracks)
    🗎 btfweihnachts (60 tracks)
    🗎 campinc (8 tracks)
    🗎 Baumusik (5 tracks)
    🗎 Clipping - CLPPNG (14 tracks)
    🗎 baumusik's dozen - 1st anniversary compilation (9 tracks)
    🗎 btfsommerparty (26 tracks)
    🗎 Border Community (58 tracks)
    🗎 AZ (37 tracks)
    🗎 btf grimme (21 tracks)
    🗎 Albrecht Schrader - Leben in der Großstadt (4 tracks)
    🗎 Alex_Smoke-Vaporub__HAH001_-Promo_CDS-2008-HFT-1 (3 tracks)
    🗎 AWeX_-_Its_Our_Future-(SUPER_DJ_2024)-CDM-FLAC-2001 (1 tracks)
    🗎 antwerpen (53 tracks)
    🗎 baumusik - it is a rose- a raisin - a festive compilation (15 tracks)
    🗎 acephale27 (57 tracks)
    🗎 At the Drive-In - Relationship of Command (13 tracks)
    🗎 Atelier (20 tracks)
    🗎 ADA (50 tracks)
    🗎 Alex_Smoke-Blingkered-_HAH005D_-WEB-2009-XXW (4 tracks)
    🗎 A-Musik (23 tracks)
    🗎 Afreuroparemixes and more (10 tracks)
    🗎 Alex_Smoke_-_Paradolia__2006_ (13 tracks)
🗎 easy (50 tracks)
🗎 medium2 (39 tracks)
🗎 hard2 (13 tracks)
🗎 SARAHBD (25 tracks)
🗎 easy2 (32 tracks)
🗎 medium (47 tracks)
🗎 31.03.2022 (155 tracks)
🗎 kbirthday (26 tracks)
🗎 26.03.2022 (4 tracks)
🗎 30.07.2022 (94 tracks)
🗎 05.03.2022 (21 tracks)
🗎 Acephale_22.01.2022 (65 tracks)
🗎 heavy (6 tracks)
🗎 Acephale30 (36 tracks)
🗎 neumix (16 tracks)
🗎 08.0.8.2022 (94 tracks)
🗀 Ace_20_08_22
    🗎 dance (44 tracks)
    🗎 soft (30 tracks)
🗎 Unbetitelte Liste (2) (0 tracks)
🗎 mix (0 tracks)
🗎 skeelo (0 tracks)
🗎 btf lockdown (45 tracks)

i went through every playlist on the drive in rekordbox compared it to the playlist you sent. i hope i made no mistake but it seems there are still some things.

folder: loststuff
mode in gliany playlist: 8 tracks (in the playlist you sent: 7 tracks)

folder: oldfolders
the sound of belgium playlist: 61 tracks (in the playlist you sent: 60 tracks)
physically Sick 2 playlist: 44 tracks (in the playlist you sent: 42 tracks)
kate tempest-everybody down playlist: 12 tracks (in the playlist you sent: 11 tracks)
hinterhof playlist: 20 tracks (in the playlist you sent: 19 tracks)
drumnbass playlist: 230 tracks (in the playlist you sent: 229 tracks)
christian vogel playlist: 85 tracks (in the playlist you sent: 84 tracks)
antwerpen playlist: 54 tracks (in the playlist you sent: 53 tracks)

folder: (mainfolder)
medium2 playlist: 40 tracks (in the playlist you sent: 39 tracks)
mix playlist: 25 tracks (in the playlist you sent: 0 tracks)
skeelo playlist: 5 tracks (in the playlist you sent: 0 tracks)
btf lockdown playlist: 72 tracks (in the playlist you sent: 45 tracks)
neumix playlist: 17 tracks (in the playlist you sent: 16 tracks)
30.07.2022 playlist: 95 tracks (in the playlist you sent: 94 tracks)

folder: Ace_20_08_22
soft playlist: 31 tracks (in the playlist you sent: 30 tracks)

Originally posted by @emmetiray in mixxxdj/mixxx#10955 (comment)

Pages are a singly-linked lists. Their order can be completely arbitrary. Modelling them using a monotonic range within a table is incorrect.

From the documentation:

The version that rekordbox 6 exports is garbled with an XOR mask to make it more difficult to access the data. All bytes after lene (bytes 10-11) are XOR-masked with a pattern that is generated by adding the value of lene to each byte of the following base pattern:

CB E1 EE FA E5 EE AD EE E9 D2 E9 EB E1 E9 F3 E8 E9 F4 E1

Support for this should be added.

Leaving here as a TODO so no one forgets.
Currently Row::parse_album uses Row::parse_string_offset which expects the offset to be a u16, however, in the
case of Album rows, the offset is only a u8. So parsing that as a u16 usually results in bogus values which causes
the parser to break when interpreting data at those offsets as DeviceSQLStrings.