Database files based on the libreto-database.
The libretro project contains a database of several rom files from different sources such as a no-intro. Those dat files however are sometimes on a particular format that is not easy to parse.
I wrote a small ANTLR based parser libretro-dat-parser to parse those files and export them as a csv file.
Inside the data folder there’s a main roms.csv file that contains all the parsed roms from the several dat files. The table bellow shows what files (and checksums)
that are included in the latest release of the data file.
| Id | Name | Files | Roms |
|---|---|---|---|
1 |
Arcade |
metadat/mame/MAME.dat (fbe8e3ee3f807ca47308221fd055313b ) |
30,038 |
2 |
Nintendo Entertainment System |
metadat/no-intro/Nintendo - Nintendo Entertainment System.dat (44ca60db94e63b8782c01989606206e3 ) |
3,154 |
3 |
Nintendo Family Computer Disk System |
metadat/no-intro/Nintendo - Family Computer Disk System.dat (32178aab352cf23ca4d94db25728855d ) |
631 |
4 |
Nintendo Game Boy |
metadat/no-intro/Nintendo - Game Boy.dat (c265a4d561d727bf75f4877656b12879 ) |
1,722 |
5 |
Nintendo Super Nintendo Entertainment System |
metadat/no-intro/Nintendo - Super Nintendo Entertainment System.dat (28debef1673d3e800f333a044e2eb0cd ) |
3,778 |
6 |
Nintendo Game Boy Color |
metadat/no-intro/Nintendo - Game Boy Color.dat (2d728487a4cf4c9c271b13fb2778dea4 ) |
1,528 |
7 |
Nintendo Virtual Boy |
metadat/no-intro/Nintendo - Virtual Boy.dat (156a566cfa7ae65fed838e1601870002 ) |
33 |
8 |
Nintendo Nintendo 64 |
metadat/no-intro/Nintendo - Nintendo 64.dat (c95ba3ff3444445a74389fb8d5f7753c ) |
2,112 |
9 |
Nintendo Game Boy Advance |
metadat/no-intro/Nintendo - Game Boy Advance.dat (7539cba16acb5a0af0af63efa4404863 ) |
3,289 |
10 |
Nintendo Nintendo DS |
metadat/no-intro/Nintendo - Nintendo DS.dat (aca304e78a6e585cfb16ac897d62e22f ) |
7,294 |
11 |
Nintendo GameCube |
dat/Nintendo - GameCube.dat (883715f4efd943e380dc03cefb046181 ) |
1,675 |
12 |
Nintendo Wii |
dat/Nintendo - Wii.dat (a5c8e45dd8477e6ca4df884c91bbfe6d ) |
8,203 |
13 |
Nintendo Nintendo 3DS |
dat/Nintendo - Nintendo 3DS.dat (d13b4f4d2b81a8e872f51b801970799a ) |
3,285 |
14 |
Nintendo Wii U |
dat/Nintendo - Wii U.dat (daf7db8ff382340efe74e0b602c1d3a2 ) |
2,875 |
15 |
Sega SG1000 |
metadat/no-intro/Sega - SG-1000.dat (14d4c7e0c14476f4f731a4ddedb7c5a4 ) |
207 |
16 |
Sega Master System Mark III |
metadat/no-intro/Sega - Master System - Mark III.dat (19d6165c2616b47632970b54c8dd73fb ) |
610 |
17 |
Sega Mega Drive Genesis |
metadat/no-intro/Sega - Mega Drive - Genesis.dat (4405fd3d57189bb3b892fd86338f1ab4 ) |
2,529 |
18 |
Sega MegaCD Sega CD |
metadat/redump/Sega - Mega-CD - Sega CD.dat (e0e33a80eda693541720d27c02b9077d ) |
578 |
19 |
Sega Game Gear |
metadat/no-intro/Sega - Game Gear.dat (32bf748fe1f2ae16959ea0e5c088261e ) |
800 |
20 |
Sega 32X |
metadat/no-intro/Sega - 32X.dat (a882d98b6c70562b0b056eb2b901e5ce ) |
207 |
21 |
Sega Saturn |
metadat/redump/Sega - Saturn.dat (aa1d733c9dd02204ba86ac1e7af0328b ) |
2,689 |
22 |
Sega Dreamcast |
metadat/redump/Sega - Dreamcast.dat (ab997de605a1ac94a05c85c45ec3340a ) |
1,664 |
23 |
Atari 2600 |
metadat/no-intro/Atari - 2600.dat (4088c67ecb6d5a3067b8c334f18dbed0 ) |
716 |
24 |
Atari 5200 |
metadat/no-intro/Atari - 5200.dat (7fc7e067ffde4f9770b87c09ae830b73 ) |
111 |
25 |
Atari ST |
metadat/no-intro/Atari - ST.dat (eb4a886f06db5f7fb9d5c65ac7f09ac9 ) |
404 |
26 |
Atari 7800 |
metadat/no-intro/Atari - 7800.dat (55afe0385050aa713b054b3b422a8773 ) |
125 |
27 |
Atari Lynx |
metadat/no-intro/Atari - Lynx.dat (c96813fec95b28cdae691c37aa3d942c ) |
94 |
28 |
Atari Jaguar |
metadat/no-intro/Atari - Jaguar.dat (e576a7d2cf1e21ede144bb8f3f87ba5e ) |
208 |
29 |
NEC PC Engine TurboGrafx 16 |
metadat/no-intro/NEC - PC Engine - TurboGrafx 16.dat (aa1520b13f8beab23063976a6a922c4a ) |
405 |
30 |
NEC PC Engine CD TurboGrafxCD |
metadat/redump/NEC - PC Engine CD - TurboGrafx-CD.dat (618bcbbf9e36b15458cc336660aab07b ) |
221 |
31 |
NEC PC Engine SuperGrafx |
metadat/no-intro/NEC - PC Engine SuperGrafx.dat (1ba5b4352c015d18636a93cb9adcd29c ) |
5 |
32 |
SNK Neo Geo CD |
metadat/redump/SNK - Neo Geo CD.dat (3e4136d28c00b62d8f2351359e574dc5 ) |
133 |
33 |
SNK Neo Geo Pocket |
metadat/no-intro/SNK - Neo Geo Pocket.dat (a5f394e715ad3f7c895257313d88fb1c ) |
10 |
34 |
SNK Neo Geo Pocket Color |
metadat/no-intro/SNK - Neo Geo Pocket Color.dat (40ca8c2d9e902697701600cd6f0c4317 ) |
118 |
35 |
The 3DO Company 3DO |
metadat/redump/The 3DO Company - 3DO.dat (349decf3953c40abfd072289c10298d0 ) |
604 |
36 |
Sony PlayStation |
metadat/redump/Sony - PlayStation.dat (3068b0ea97b2e3381014fb1fa7af38d1 ) |
13,514 |
37 |
Sony PlayStation Portable |
metadat/redump/Sony - PlayStation Portable.dat (68144e195423705749be32a4dd8a2ba2 ) |
2,589 |
38 |
Sony PlayStation 2 |
metadat/redump/Sony - PlayStation 2.dat (61ce126fcb2a0b3efb0d797f9b7b9a7d ) |
12,197 |
39 |
Sony PlayStation 3 |
metadat/redump/Sony - PlayStation 3.dat (9b96a42ee7782ae5a9f8c8c45e14535a ) |
0 |
40 |
Sony PlayStation Vita |
metadat/no-intro/Sony - PlayStation Vita.dat (1c6e0779950df06e3f39c635be07bffb ) |
240 |
41 |
Microsoft Xbox |
metadat/redump/Microsoft - Xbox.dat (ced2e2db64d6679e99b286a60236aa8a ) |
2,522 |
42 |
Microsoft Xbox 360 |
metadat/redump/Microsoft - Xbox 360.dat (82c81ceef1ebb790a9a4bd059439b8c0 ) |
0 |
43 |
Bandai WonderSwan Color |
metadat/no-intro/Bandai - WonderSwan Color.dat (8527a357c29c159f0ff23bd9e8c05b57 ) |
107 |
44 |
Bandai WonderSwan |
metadat/no-intro/Bandai - WonderSwan.dat (8cb412f94f0ef9e12c6c9d224dd72cd8 ) |
125 |
45 |
Commodore 64 |
metadat/no-intro/Commodore - 64.dat (59a8954c474632e4851985a94b905237 ) |
237 |
46 |
Commodore Amiga |
metadat/no-intro/Commodore - Amiga.dat (274edbb9c42004b4dbed5ad2485a8608 ) |
3,151 |
47 |
Coleco ColecoVision |
metadat/no-intro/Coleco - ColecoVision.dat (8168185dac4baafd93b2e6dc4b8a1f1b ) |
191 |
48 |
Microsoft MSX |
metadat/no-intro/Microsoft - MSX.dat (388e3b67ef787283bc10b378c9c57db9 ) |
887 |
49 |
Microsoft MSX2 |
metadat/no-intro/Microsoft - MSX2.dat (435226f7081ce977ba81f3c5c62d43a0 ) |
191 |
50 |
Neo Geo |
0 |
|
51 |
Mattel Intellivision |
metadat/no-intro/Mattel - Intellivision.dat (493e135ee9a5baa5000b42addc0d1c9c ) |
186 |
52 |
Magnavox Odissey 2 |
metadat/no-intro/Magnavox - Odyssey2.dat (ff1273df6c833888f7e37291e91c6c78 ) |
131 |
Platforms from datfile
The roms database would allow anyone to write their own scrapper to match roms using either their file checksums or headers (Mega Drive, Saturn, Gamecube) for example.
But it would be nice to associate those roms with data from services such as https://thegamesdb.net/(The Games DB), https://www.igdb.com/(IGDB) or https://www.giantbomb.com/(Giantbomb), all of which provide a free public api for use.
So I’ve wrote a small search algorithm to match roms from the roms.csv to each of these services, and the result is in the data folder as game_similarity.csv.
The match is not ideal, a lot of preprocessing was needed in the roms (escaping chars, removing diacritical characters) and sometimes even the best effort would yield mismatches such as Disney’s Aladdin does not exact match Aladdin although
they are the same game.
But it feels this is a good step if someone decides to manually correct the data.
Data is divided in three categories:
-
Exact Match: The cleaned rom name is an exact match of the game name on the target service.
-
Similar Match: The name is not an exact match, but the https://en.wikipedia.org/wiki/Metaphone(Double Metaphone) comparison of both cleaned strings are the same. This is where
Disney AladdinandAladdinwould yield a Match. -
Missing: Not exact match and the metaphone comparison is also false. Some entries in this category are an actual match, so after manual cleansing the number of match results could be even higher.
The following tables represent the status of the database in approximate matching numbers.
| Id | Platform | Exact Match | Similar | Missing |
|---|---|---|---|---|
1 |
Arcade |
45% |
10% |
45% |
2 |
Nintendo Entertainment System |
76% |
10% |
14% |
3 |
Nintendo Family Computer Disk System |
65% |
16% |
19% |
4 |
Nintendo Game Boy |
61% |
11% |
27% |
5 |
Nintendo Super Nintendo Entertainment System |
73% |
13% |
13% |
6 |
Nintendo Game Boy Color |
65% |
16% |
19% |
7 |
Nintendo Virtual Boy |
79% |
12% |
9% |
8 |
Nintendo Nintendo 64 |
76% |
9% |
15% |
9 |
Nintendo Game Boy Advance |
64% |
16% |
20% |
10 |
Nintendo Nintendo DS |
49% |
14% |
38% |
11 |
Nintendo GameCube |
79% |
11% |
10% |
12 |
Nintendo Wii |
42% |
15% |
43% |
13 |
Nintendo Nintendo 3DS |
47% |
9% |
44% |
14 |
Nintendo Wii U |
47% |
9% |
43% |
15 |
Sega SG1000 |
87% |
5% |
7% |
16 |
Sega Master System Mark III |
75% |
8% |
17% |
17 |
Sega Mega Drive Genesis |
82% |
8% |
10% |
18 |
Sega MegaCD Sega CD |
81% |
7% |
12% |
19 |
Sega Game Gear |
82% |
10% |
9% |
20 |
Sega 32X |
82% |
4% |
14% |
21 |
Sega Saturn |
68% |
18% |
15% |
22 |
Sega Dreamcast |
53% |
14% |
33% |
23 |
Atari 2600 |
85% |
5% |
9% |
24 |
Atari 5200 |
86% |
4% |
11% |
25 |
Atari ST |
28% |
7% |
65% |
26 |
Atari 7800 |
95% |
0% |
5% |
27 |
Atari Lynx |
97% |
1% |
2% |
28 |
Atari Jaguar |
78% |
5% |
17% |
29 |
NEC PC Engine TurboGrafx 16 |
80% |
14% |
6% |
30 |
NEC PC Engine CD TurboGrafxCD |
50% |
19% |
31% |
31 |
NEC PC Engine SuperGrafx |
40% |
20% |
40% |
32 |
SNK Neo Geo CD |
32% |
25% |
43% |
33 |
SNK Neo Geo Pocket |
60% |
40% |
0% |
34 |
SNK Neo Geo Pocket Color |
69% |
25% |
6% |
35 |
The 3DO Company 3DO |
58% |
10% |
31% |
36 |
Sony PlayStation |
50% |
15% |
35% |
37 |
Sony PlayStation Portable |
71% |
11% |
18% |
38 |
Sony PlayStation 2 |
59% |
16% |
25% |
40 |
Sony PlayStation Vita |
90% |
8% |
2% |
41 |
Microsoft Xbox |
75% |
14% |
10% |
43 |
Bandai WonderSwan Color |
76% |
19% |
6% |
44 |
Bandai WonderSwan |
79% |
13% |
8% |
45 |
Commodore 64 |
50% |
8% |
43% |
46 |
Commodore Amiga |
49% |
10% |
41% |
47 |
Coleco ColecoVision |
87% |
4% |
9% |
48 |
Microsoft MSX |
69% |
7% |
24% |
49 |
Microsoft MSX2 |
57% |
13% |
31% |
51 |
Mattel Intellivision |
75% |
8% |
17% |
52 |
Magnavox Odissey 2 |
87% |
3% |
10% |
| Id | Platform | Exact Match | Similar | Missing |
|---|---|---|---|---|
1 |
Arcade |
23% |
10% |
67% |
2 |
Nintendo Entertainment System |
68% |
10% |
22% |
3 |
Nintendo Family Computer Disk System |
54% |
21% |
25% |
4 |
Nintendo Game Boy |
48% |
12% |
40% |
5 |
Nintendo Super Nintendo Entertainment System |
80% |
10% |
9% |
6 |
Nintendo Game Boy Color |
41% |
16% |
43% |
7 |
Nintendo Virtual Boy |
64% |
12% |
24% |
8 |
Nintendo Nintendo 64 |
70% |
10% |
20% |
9 |
Nintendo Game Boy Advance |
47% |
16% |
37% |
10 |
Nintendo Nintendo DS |
42% |
16% |
42% |
11 |
Nintendo GameCube |
65% |
11% |
24% |
12 |
Nintendo Wii |
28% |
16% |
56% |
13 |
Nintendo Nintendo 3DS |
34% |
4% |
62% |
14 |
Nintendo Wii U |
66% |
9% |
25% |
15 |
Sega SG1000 |
28% |
10% |
63% |
16 |
Sega Master System Mark III |
51% |
11% |
38% |
17 |
Sega Mega Drive Genesis |
61% |
12% |
27% |
18 |
Sega MegaCD Sega CD |
65% |
8% |
26% |
19 |
Sega Game Gear |
41% |
11% |
48% |
20 |
Sega 32X |
79% |
3% |
17% |
21 |
Sega Saturn |
44% |
14% |
42% |
22 |
Sega Dreamcast |
51% |
11% |
38% |
23 |
Atari 2600 |
41% |
6% |
53% |
24 |
Atari 5200 |
51% |
2% |
47% |
25 |
Atari ST |
24% |
7% |
69% |
26 |
Atari 7800 |
60% |
0% |
40% |
27 |
Atari Lynx |
33% |
2% |
65% |
28 |
Atari Jaguar |
78% |
3% |
19% |
29 |
NEC PC Engine TurboGrafx 16 |
78% |
12% |
10% |
30 |
NEC PC Engine CD TurboGrafxCD |
38% |
8% |
54% |
31 |
NEC PC Engine SuperGrafx |
60% |
20% |
20% |
32 |
SNK Neo Geo CD |
22% |
23% |
55% |
33 |
SNK Neo Geo Pocket |
20% |
50% |
30% |
34 |
SNK Neo Geo Pocket Color |
46% |
36% |
18% |
35 |
The 3DO Company 3DO |
36% |
5% |
59% |
36 |
Sony PlayStation |
45% |
15% |
41% |
37 |
Sony PlayStation Portable |
23% |
20% |
58% |
38 |
Sony PlayStation 2 |
53% |
16% |
31% |
40 |
Sony PlayStation Vita |
70% |
12% |
18% |
41 |
Microsoft Xbox |
59% |
11% |
30% |
43 |
Bandai WonderSwan Color |
21% |
14% |
64% |
44 |
Bandai WonderSwan |
23% |
11% |
66% |
45 |
Commodore 64 |
24% |
7% |
68% |
46 |
Commodore Amiga |
24% |
8% |
67% |
47 |
Coleco ColecoVision |
43% |
6% |
51% |
48 |
Microsoft MSX |
23% |
5% |
72% |
49 |
Microsoft MSX2 |
21% |
9% |
70% |
51 |
Mattel Intellivision |
51% |
4% |
45% |
52 |
Magnavox Odissey 2 |
1% |
0% |
99% |
| Id | Platform | Exact Match | Similar | Missing |
|---|---|---|---|---|
1 |
Arcade |
19% |
12% |
69% |
2 |
Nintendo Entertainment System |
22% |
13% |
65% |
3 |
Nintendo Family Computer Disk System |
33% |
11% |
55% |
4 |
Nintendo Game Boy |
32% |
12% |
56% |
5 |
Nintendo Super Nintendo Entertainment System |
30% |
16% |
54% |
6 |
Nintendo Game Boy Color |
32% |
16% |
52% |
7 |
Nintendo Virtual Boy |
91% |
0% |
9% |
8 |
Nintendo Nintendo 64 |
45% |
14% |
40% |
9 |
Nintendo Game Boy Advance |
37% |
23% |
40% |
10 |
Nintendo Nintendo DS |
25% |
16% |
59% |
11 |
Nintendo GameCube |
18% |
9% |
73% |
12 |
Nintendo Wii |
20% |
14% |
66% |
13 |
Nintendo Nintendo 3DS |
0% |
1% |
99% |
14 |
Nintendo Wii U |
21% |
13% |
66% |
15 |
Sega SG1000 |
24% |
8% |
68% |
16 |
Sega Master System Mark III |
26% |
7% |
67% |
17 |
Sega Mega Drive Genesis |
32% |
11% |
57% |
18 |
Sega MegaCD Sega CD |
28% |
5% |
67% |
19 |
Sega Game Gear |
31% |
6% |
63% |
20 |
Sega 32X |
55% |
5% |
40% |
21 |
Sega Saturn |
17% |
13% |
70% |
22 |
Sega Dreamcast |
21% |
12% |
67% |
23 |
Atari 2600 |
49% |
8% |
43% |
24 |
Atari 5200 |
7% |
2% |
91% |
25 |
Atari ST |
3% |
2% |
95% |
26 |
Atari 7800 |
27% |
2% |
71% |
27 |
Atari Lynx |
49% |
3% |
48% |
28 |
Atari Jaguar |
49% |
2% |
49% |
29 |
NEC PC Engine TurboGrafx 16 |
39% |
18% |
44% |
30 |
NEC PC Engine CD TurboGrafxCD |
12% |
5% |
84% |
31 |
NEC PC Engine SuperGrafx |
20% |
20% |
60% |
32 |
SNK Neo Geo CD |
3% |
18% |
79% |
33 |
SNK Neo Geo Pocket |
10% |
70% |
20% |
34 |
SNK Neo Geo Pocket Color |
58% |
28% |
14% |
35 |
The 3DO Company 3DO |
23% |
7% |
70% |
36 |
Sony PlayStation |
23% |
21% |
56% |
37 |
Sony PlayStation Portable |
34% |
16% |
50% |
38 |
Sony PlayStation 2 |
22% |
20% |
58% |
40 |
Sony PlayStation Vita |
24% |
12% |
64% |
41 |
Microsoft Xbox |
13% |
9% |
78% |
43 |
Bandai WonderSwan Color |
64% |
21% |
15% |
44 |
Bandai WonderSwan |
66% |
18% |
16% |
45 |
Commodore 64 |
5% |
4% |
91% |
46 |
Commodore Amiga |
9% |
6% |
85% |
47 |
Coleco ColecoVision |
25% |
4% |
71% |
48 |
Microsoft MSX |
16% |
6% |
78% |
49 |
Microsoft MSX2 |
6% |
1% |
94% |
51 |
Mattel Intellivision |
41% |
6% |
52% |
52 |
Magnavox Odissey 2 |
12% |
2% |
86% |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent