Add TWL Cart support to NTR Cart Dumping... about decrypt9wip HOT 76 CLOSED

Yeah Normmatt is still cleaning it up. It's nearly there now. But minor corruption in the DSi Secure area and some junk data in the 0x1000 that needs to be removed. For NTR mode only stuff, it's already functional, but for the sake of having proper dumps, it's best to wait till the DSi extended area gets dumped correctly. :D

from decrypt9wip.

How can i dump pokekon black via hourglass9 ?

from decrypt9wip.

Accidentially closed :). Anyways, I somewhat feel the original issue is solved with the most recent commits, thus I can close this. Anyone disagreeing on that?

from decrypt9wip.

Alright... invalid secure area is not an actual bug. and doesn't mean that the dump is invalid I guess I should just remove that line from the source code. @GerbilSoft already sent a p/r that solves this problem, there's just some small stuff that has to be adapted a little bit.

The issue in question (TWL cart dumping) is solved, so I'm finally closing this issue (after almost half a year, woohooo!). If anything else that is cart related pops up, feel free to open a new issue.

Also, thanks a ton to @YodaDaCoda, @thisokra, @ApacheThunder, @idgrepthat, @Normmatt, @osilloscorpion and everyone else involved in solving this issue and to

from decrypt9wip.

I compiled and had someone in Discord test - it failed to dump Pokemon Black 2 with

Error reading secure data
Invalid secure area. (37898C54 34196416)

from decrypt9wip.

Normmatt has updated it. It now dumps TWL carts correctly. This is ready to be added. :D

from decrypt9wip.

The new revision in the gist is the same diff as previous, copied below itself.

from decrypt9wip.

Added in 0d5a05f
Please test!

from decrypt9wip.

Dumped Pokemon Platinium and Pokemon W2 successfully (both work on desmume)
But "Invalid secure area.(F3D59990 687A6856)" for PokemonW2
Also DSTWO (not +) dump failed

Tested with the lastest commit

from decrypt9wip.

Alright, could someone also test dumping a regular (non-DSi) DS cart? Just to make sure nothing is broken

from decrypt9wip.

Using 0d5a05f

Failed to dump Pokemon Black, Pokemon White (TWL).
Successfully dumped Pokemon SoulSilver (NTR).

Pokemon White log

Selected: [Dump NTR/TWL Cart (full)]
Cartridge ID: C001FFC2
Cartridge Type: NTR
Product name: POKEMON W
Product ID: IRAO01
Cartridge data size: 256MB
Cartridge used size: 195MB
Cartridge dump size: 256MB
Invalid SecureCartID. (B6884E00 != C001FFC2)
Error reading secure data
Dump NTR/TWL Cart (full): failed!

Press B to return, START to reboot.

Pokemon Black log:

Selected: [Dump NTR/TWL Cart (full)]
Cartridge ID: C001FFC2
Cartridge Type: NTR
Product name: POKEMON B
Product ID: IRBO01
Cartridge data size: 256MB
Cartridge used size: 195MB
Cartridge dump size: 256MB
Invalid SecureCartID. (B6884E00 != C001FFC2)
Error reading secure data
Dump NTR/TWL Cart (full): failed!

from decrypt9wip.

Just tested Pokemon White, and I have the same error

from decrypt9wip.

The gist is at 4 revisions now.

At two revisions (the current d9 commit), it dumped biggest loser US and system flaw correctly according to dat-o-matic checksums

from decrypt9wip.

It dumped my Sonic Classic Collections cart correctly last night. (it will say there was an error reading DSi Secure area, but will dump anyway. There is a problem with the detection routine and Normmatt decided to leave that as is.

Not sure if he's done more work on it since I went to bed though. The build he sent me was fully functional in terms of dumping the TWL cart I did have. :P

from decrypt9wip.

"the gist is at 4 revisions now" - he did make changes after the d9 release.

from decrypt9wip.

Sorry to be noobiest :),but when saying NTR,what is it? (NDS codename?) Same thing for TWL.

from decrypt9wip.

NTR is the code name used for DS/DS Lite. TWL is the code name for the DSi. ;) Also being that this has been implemented, I will close this issue now. :D

from decrypt9wip.

I can confirm that the lastest gist revision (rev 4 at time of writing) does not dump TWL carts. Neither does revision 0d5a05f in this repository.

Gist revision 3 doesn't apply (git apply gives errors). Revision 2 is revision 1 pasted twice. Revision 1 doesn't dump either.

I have not experienced what @ApacheThunder describes where it gives an error but dumps anyway.

from decrypt9wip.

I can confirm that the lastest gist revision (rev 4 at time of writing) does not dump TWL carts. Neither does revision 0d5a05f in this repository.
Gist revision 3 doesn't apply (git apply gives errors). Revision 2 is revision 1 pasted twice. Revision 1 doesn't dump either.
I have not experienced what @ApacheThunder describes where it gives an error but dumps anyway.

Dumped 4 test carts fine for me, both rev 2 and 4.
System Flaw TWL good
Biggest Loser TWL/NTR good
DQV NTR good
Pokmn PL NTR good

Both revisions gave the same results.

from decrypt9wip.

Could the console itself be a factor? I'm using an AL9H'd o3DSXL.

Perhaps the cart is a factor? Can someone else test with Pokemon B/W?

I have no problems dumping NTR carts, only TWL.

from decrypt9wip.

Could the console itself be a factor? I'm using an AL9H'd o3DSXL.

Perhaps the cart is a factor? Can someone else test with Pokemon B/W?

I have no problems dumping NTR carts, only TWL.

Could you try this guys build? It seems to work for me
http://gbatemp.net/attachments/decrypt9wip-20160811-181557-zip.58748/

from decrypt9wip.

@idgrepthat

As you suggested, I used the Decrypt9WIP.bin from that archive and I got the same result.

Full log from that session:

Initializing SD card... success
Build: Decrypt9WIP (2016/08/11)
Work directory: /files9
Game directory: /files9
sector0x96 Key: loaded, stored
0x03 KeyX & KeyY: automatically set up
0x25 KeyX: loaded, verified, set up
0x18 KeyX: already set up
0x1B KeyX: loaded, verified, set up
0x24 KeyY: loaded, set up
Finalizing Initialization...

Initialization: success!

Selected: [Dump NTR/TWL Cart (full)]
Cartridge ID: C001FFC2
Cartridge Type: NTR
Product name: POKEMON W
Product ID: IRAO01
Cartridge data size: 256MB
Cartridge used size: 195MB
Cartridge dump size: 256MB
Invalid SecureCartID. (B6884E00 != C001FFC2)
Error reading secure data
Dump NTR/TWL Cart (full): failed!

Press B to return, START to reboot.

Selected: [Dump NTR/TWL Cart (trim)]
Cartridge ID: C001FFC2
Cartridge Type: NTR
Product name: POKEMON W
Product ID: IRAO01
Cartridge data size: 256MB
Cartridge used size: 195MB
Cartridge dump size: 195MB
Invalid SecureCartID. (B6884E00 != C001FFC2)
Error reading secure data
Dump NTR/TWL Cart (trim): failed!

Press B to return, START to reboot.

from decrypt9wip.

Perhaps it's not detecting that Secure area valid properly. It had this issue with the DSi Secure Area. Normmatt simply allowed it to dump anyways by bypassing the abort triggered by the error.

Perhaps the same can be done in NTR cart dump section. This would also allow dumping some flashcart roms since they don't have valid secure areas.

from decrypt9wip.

Huh. 2ds 9.2 a9lh here. I suppose it's a problem with those games.
^ and what apache said.

from decrypt9wip.

Following on from what @ApacheThunder said, I applied the following patch to revision 0d5a05f and to rev4 of the gist.

Dumped Pokemon White, but in both cases the sha1sum of the dumped file does not match scene release and does not run in emulator.

diff --git a/source/decryptor/game.c b/source/decryptor/game.c
index bb87ada..8d02dc1 100644
--- a/source/decryptor/game.c
+++ b/source/decryptor/game.c
@@ -1712,7 +1712,7 @@ u32 DumpTwlGameCart(u32 param)

    if (!NTR_Secure_Init (buff, Cart_GetID(), 0)) {
        Debug("Error reading secure data");
        [-return-]{+//return+} 1;
    }

    Debug("");

from decrypt9wip.

Yoda, you have that build? Would like to test and see if it dumps my DS-Xtreme properly now that you disabled the secure area check. :D

from decrypt9wip.

@ApacheThunder You can give it a try if you want...

D9WIP-twltest-201608121138.zip

Edit: that's the modified build from rev4 of the gist.

from decrypt9wip.

Thanks. I tested that build compared to the build Normmatt had me test last night. It still dumps my Sonic Classic Collection game correctly (boots in No$GBA in DSi mode). So not sure how others here aren't getting the same results.

As for the DS-X, it dumps with corruption in the arm9 region and beyond just as it does in wooddumper. :(

Fortunately I've dumped this cart by other means I just wanted to see if this will prove useful for other trouble some flashcarts. But it appears it will run into the same problem. Oddly NDS Backup Tool Wifi is the one that does dump the DS-X correctly. I have a card swapper device hence I'm able to use it. :P

from decrypt9wip.

Alright, as this issue is still not 100% resolved, I'm reopening it now. Once there is some progress and this is reliable enough, please send me a pull request.

from decrypt9wip.

I saw people have issues dumping Pokemon White and Black. I used the latest precompiled version. Pokemon White dumped fine, Black didn't. It showed the same error like in the other comments. Both versions are PAL/German

from decrypt9wip.

I get the feeling that because these games are just DSi enhanced, they have essentially an NTR header with just extra code for the DSi features. Maybe this will require some sort of white list for false negatives of DSi games.

from decrypt9wip.

This issue was open for a while, so, are there some more ideas? A whitelist is okay, but only if this is for just a handful of titles. And I'll still need to see how this works.

from decrypt9wip.

Given there aren't that many DSi-Enhanced games to begin with, I don't see the whitelist getting too big. Sadly, I have no clue where to start with how to make it work. Clearly each game has their own unique header. The only thing that needs to be done is verifying the DSi-only code, if it's encrypted.

from decrypt9wip.

Hoping to get this discussion going again, here's an incomplete list of DSi enhanced titles:
http://www.rfgeneration.com/blogs/pdrydia/DSi-enhanced-titles-an-incomplete-list-1969.php

It's more than you guys think. A whitelist is not really a good solution, I think. Can anyone add information here that would help?

from decrypt9wip.

I have a friend with all of the DS/DSi/3DS Pokemon games. If that helps in any way - I'm assuming if I use the dump private header option, I can then compare the headers and see if there are any obvious differences between them which may assist in identifying the DSi games?

from decrypt9wip.

I'm pretty sure you guys already thought I gave up on this issue, but I'm actually still doing my homework on it. I did some research and I now at least somewhat understand how the NDS and DSi dumping process works.

What would really help me is:

• (1) a bad dump of a DSi enhanced or DSi only cart, as dumped by D9
• (2) the same cart dumped via GM9 (link to test build below)
• (3) a confirmed good dump of the same cart

Now, when I say confirmed good, CRC checking it with no-intro is not enough. You need a dump that can actually run in DSi mode (should be able to confirm via NDStool / no$GBA / certain flashcards), when the D9 / GM9 dump cannot.

Anyone able to provide this?

Link to GodMode9 test build:
https://transfer.sh/nY07U/godmode9-20170114-173114.zip
(just copy the dump from the C: drive)

from decrypt9wip.

I'm using a Pokemon White cart, which is on the list of DSi games linked above.

Using the GM9 build you linked, the first time I open C: I get an error "Invalid SecureCartID (B6884E00 != C001FFC2)" and then no files in the list. If I back out and open C: again, I can see two "0 Byte" files named "POKEMON W.nds" and "POKEMON W.trim.nds". Opening them in the built-in hex editor shows they're empty. Copying them to the SDCard just gives me two 0-byte files of the same names.

from decrypt9wip.

D9 does not dump it at all, right?

from decrypt9wip.

Correct, using the latest release of D9 I get the exact same SecureCartID message and no file dumped.

from decrypt9wip.

Alright... so, is there a known good dump of Pokemon White? You may try this test build:
https://transfer.sh/zqnRj/godmode9-20170116-010022.zip

This will still show that prompt, but will allow you to dump nonetheless. If you can dump it, and if there is a known good one, we'd need to compare the two.

from decrypt9wip.

I have a proper TWL dump of Pokemon White (verified using ndstool - see attached log - I'd appreciate if you could confirm?). I'll dump my cart using the test build linked above when I get home in about 4 hours.

How would you like to go about comparing the two?

Edit: Dumped. Awaiting your instructions.

from decrypt9wip.

Thanks a lot!

I'll basically only need two segments of data from these dumped roms:

• 0x8000 byte @ 0x0000000 (header + secure area, first 0x8000 byte of file)
• 0x4000 byte @ 0xC403000 (modcrypt area)

The only difference between the two files should be in the modcrypt area, everything else should be identical (let me know if there are actually more differences).

As for the NDSTool log, I am not entirely sure. We have: "Segment3 CRC (0x3000-0x3FFF)0x0000 (INVALID)". It would be even better to have this run in no$GBA, in DSi mode. Can you show me the log for the improper dump, too?

from decrypt9wip.

Mind if I email you?

from decrypt9wip.

from decrypt9wip.

Ok I changed to a Pokemon Black cart for reasons.

NDSTool logs:
GM9
TWL

It's interesting to note that ndstool reports the same securearea crc for both, but reports it as being invalid for the gm9 dump.

I can't upload the file chunks, but...

• The header area is the same up until 0x4000 - they differ from 0x4000-0x8000.
• The modcrypt area is entirely different

I can't for the life of me seem to get no$gba to run a rom in DSi mode. I think I need different BIOS files or something.

from decrypt9wip.

@YodaDaCoda you need a NAND from a real DSi to use games in DSi mode...

from decrypt9wip.

@dark-samus from what I understand, that would require a DSi-mode flashcard? I don't think I have the resources to pull that off.

from decrypt9wip.

@YodaDaCoda email is okay, too, ofc.

from decrypt9wip.

@YodaDaCoda - 0x4000....0x8000 is the NTR Secure area. There actually should never be anything wrong with that, and if it is borked, that would mean that not even NTR mode would work with that dump. This may be even worse than I thought... I assume the areas that were different were completely different and also not just something like switched or anything like that?

from decrypt9wip.

@YodaDaCoda nah, gamecarts can't access NAND (or at least no official gamecart can) you'd need a hardmod or DSiwarehax... Someone released a DSi NAND image in chat the other day (not on any forums)

from decrypt9wip.

@YodaDaCoda - thanks for providing the chunks I asked for. The secure area is different because the "TWL" dump has it decrypted, while GM9 left it encrypted (cause the CycloDS does not like decrypted secure areas). The problem may even be identical for the modcrypt area.

Can you try again with this test build?
https://transfer.sh/qXGh5/godmode9-20170117-104102.zip
This will do the decryption that was previously left out.

from decrypt9wip.

Okay, here's a newer test build, including a fix provided by @Normmatt. If you can, please test:
https://transfer.sh/DHXy0/godmode9-20170118-012910.zip

from decrypt9wip.

Using the build from @Normmatt GM9 claims that my copies of Pokemon White 1 and 2, Pokemon Conquest, and interestingly, SMT Strange Journey have Invalid SecureCartIDs and doesn't let me view/dump
Strange Journey is an NTR game
I can see other NTR games like Mario Kart and Final Fantasy IV as normal

from decrypt9wip.

Latest build with Normatt's fix gives the Invalid SecureCartId error once, before listing no files. Previous versions would give the error twice, before listing .nds and .trim.nds.

from decrypt9wip.

The 20170117 build gives the exact same binary result for the 0x0000000 chunk, but the 0xC403000 differs, which I think is the opposite of expected?

from decrypt9wip.

@YodaDaCoda well... I at least know what causes this, but this doesn't bring us any nearer to the solution. Here's a build with another fix @Normmatt suggested:
https://transfer.sh/aaYqg/godmode9-20170118-171337.zip

Also, @thisokra, thanks! This happening with NTR stuff, too, is new.

from decrypt9wip.

Sure thing.
Both 20170117-104102 and 20170118-171337 builds treat Strange Journey normally and can dump it
20170118-171337 still gives the Invalid ID error for Pokemon White but not Pokemon White 2.
I don't have a way to verify if the White 2 dump is valid unfortunately

from decrypt9wip.

I have a DSi nand dump working with No$GBA. If you wish, you can private message me that rom dump and I can see if it boots for you. I presume it won't work if the modcrypt section is bad. Because No$GBA expects modcrypt to be encrypted I think...Or at least valid anyways. :P

You'll have to pm me at GBATemp. Don't know if there is a PM feature in Github. Haven't bothered to check. :P

from decrypt9wip.

We may be getting somewhere, @thisokra and @YodaDaCoda. A previously unable-to-dump cart worked.

Here's a new build that extends on the fix in the previous 171337 had:
https://transfer.sh/PuH6y/godmode9-20170118-191146.zip

Let me know how this works for you. Note: the filename will change if the modcrypt area is bad (= when that invalid ID error happens). One thing that may also help is if you gave me the cart ids (both, from the error message).

from decrypt9wip.

None of the TWL carts I have show an error now (and neither does Strange Journey)
I can still get ID pairs from an older build if that would help

from decrypt9wip.

@thisokra - this is fantastic news! We still need more carts tested, and of course these dumps also need to be checked (in no$gba, f.e., or checked against known good dumps). Other than that, this is a big step forward.

I also only need the cart ids in case something goes wrong, so no need to bother.

from decrypt9wip.

Exciting!

$ md5sum *.nds
37bff1431eda9b3a525737c7f59a432d pokemon_b.gm9.nds
37bff1431eda9b3a525737c7f59a432d pokemon_b.twl.nds

I don't get SecureCartId error anymore and it seems to dump 100%. I'll reach out to @ApacheThunder on GBATemp and see if he has better luck getting past the AP stuff on no$GBA.

from decrypt9wip.

Ok I've managed to get past the save file issue and can confirm that no$GBA is loading the GM9-dumped version of Pokemon Black in TWL mode. I'll test with Pokemon B2 and W2 (the only other TWL games I have) in about 8-10 hours.

To get past the save error, I had to find the correct memory addresses to patch - I don't know if it's because it's running in TWL mode or if it's because it's a proper dump, but the addresses are different to what's posted all over the internet. In order to make it a matter of public record, I'll post what I found here. Only the second address is different - the first still points to the same instruction. All codes are AR raw format.

Original code to get past save screen (this is the one posted all over the internet):
02006F2C 00000000
02180B8C 00000000

Pokemon Black (and possibly White - untested) TWL code to get past save screen:
02006F2C 00000000
02180BA4 00000000

NB: This doesn't bypass the AP checks in the rom - finding those offsets is rather beyond my abilities at the moment.

from decrypt9wip.

I seem to have misplaced my B2/W2 carts, but I dumped Pokemon White, and confirmed it runs in TWL mode in no$GBA, though the code you need to bypass the save screen is the original one. I don't have any more TWL carts to test with.

from decrypt9wip.

Alrigh, thank you!

By now I know of only one title that still has the SecureCart ID error - a somewhat obscure DSi exclusive title called "Face Training". Everything else dumped fine. Does anyone on here own this title? Also, testing with more titles is appreciated!

from decrypt9wip.

If you'd rather test in D9, D9 has its cart dumping code updated as well. Grab your test build from here:
https://transfer.sh/D2cQb/decrypt9wip-20170119-112252.zip

from decrypt9wip.

About DS cart dumping, not really related to this problem, but imo it seems a bit redundant to take the 01 from 0x10-0x11 when dumping (for the file name, with the game code.) Why not use the actual game version at 0x1E instead?
Every DS rom I've checked has 01 there so that information is a bit pointless to have, while it might be more useful to have the dumps marked as (for example) ADAE05 for Pokémon Diamond v1.5, APAJ06 for v1.6, CPUE01 for v1.1, etc
(It's such a small thing I didn't think it would be worth opening a different issue just for that so I figured I'd just put it there with the rest of the DS dumping discussion.)

from decrypt9wip.

Well... you're mistaken. GM9 does not use the data located there. It just uses the title, which is located 0x00...0x0B. I assume you mean D9? The data located at 0x10...0x11 should be the maker code. Are you really sure this is the same in every case?

from decrypt9wip.

I'm not talking about GM9, though... I'm talking about what D9 does. This is an issue thread on the Decrypt9WIP github, is it not?

I looked at what Makercode actually does and it just tells who published the game (hence why everything I looked at was 01, since they're all from Nintendo.)
I just feel like using the rom version rather than the makercode would be better... I can't really think of a reason why knowing the makercode would be more important than knowing the rom version :p (I guess unless someone -really- wants to sort their rom collection by publisher?)

from decrypt9wip.

@Ammako You're very much mistaken. D9/GM9 doesn't read the maker code. While it would be good for D9/GM9 to read the revision and add that to the filename (like what wooddumper does), this is not the thread to request that feature.
https://github.com/d0k3/Decrypt9WIP/blob/master/source/decryptor/game.c#L2129
If you can point out where in this file, or even in the program, where it reads the maker code, then please do so.

EDIT: It has been pointed out to me that D9/GM9 does indeed parse the maker code at 0x10-0x11 at https://github.com/d0k3/Decrypt9WIP/blob/master/source/decryptor/game.c#L2156

from decrypt9wip.

Why do half-measures when you can just parse the banner region for the full game's banner text? TWLoader now does this (when reading roms on SD though. Not directly off slot-1)...I imagine it would not be difficult to do it on arm9 baremetal as well. Though I suppose reading the version field would be useful for differentiating different revisions of the same game. Outside of that though, not many people will care about that. :P

from decrypt9wip.

@ApacheThunder afaik, to do that you'd need to decrypt the header, parse the pointer, then decrypt the area where it is pointing to and then parse the UTF-8 at that point.

from decrypt9wip.

Beyond the usual NTR card protocol encryption, I don't think header and banner region have any additional layer of encryption. Only the NTR/TWL secure areas have an additional layer of crypto. The banner region for TWL carts is pretected by only a hmac hash in the DSi Extended Header which itself is protected at the end of the extended header with the RSA sig. Nintendo never felt the need to encrypt the banner region.

Besides, the basic NTR protocol code is already in D9/GM9. So this should work even with TWL carts that had issues dumping correctly. (those tended to have issues with the DSi Secure area not decrypting right but maybe that's fixed now)

from decrypt9wip.

Alright, I agree that the rom_version should be part of the NDS dump filename. The maker_code is still useful, so I left it in, too.
https://transfer.sh/EpVcc/decrypt9wip-20170120-133100.zip

@ApacheThunder is right - header and banner have no encryption, only the secure area / modcrypt area has. Yes, we could extract the "true" / full name from there. And we could do the same for 3DS carts if we decrypt the icon (smdh) inside the ExeFS. I still won't do it, for two reasons:

• These full names can contain all kinds of special / regional symbols, which would mean I'd either have to handle this or open up the door to all kinds of unforeseen consequences.
• After all, all we need is a unique, identifyable name. Beautifying these names and / or adding info is the job of a rom renamer, not D9 / GM9.

from decrypt9wip.

Did you deliberately put a smileyface in filenames of dumps generated by that test build? Because if so, that's awesome! If not, I might have a bug to report...

I can confirm that the build dumps files correctly (md5sum matches what I dumped before which ran on no$GBA in TWL mode) but I still get an Invalid secure area (3C1119DB C484A706). Also, the filename stops after the Product ID. E.g. Pokemon White dumps as IRA01 - no .nds or product version; I suspect this is because of the aforementioned smileyface ;).

from decrypt9wip.

The commit that added ROM versions to the filename didn't set a maximum width for the field, so the unitcode (0x02 or 0x03) ends up getting appended.

I sent PR #144 that fixes this, along with some other minor improvements. (Version number for CTR, and use the TWL used size for TWL trimmed dumps.)

I also tested the TWL dump updates with System FLAW. The MD5 before and after are the same, and both match no-intro: e45cc21a32b73a9b0e80e11b23e304ce

Decrypt9WIP does complain about an invalid secure area: 99289C58 5A3069D6

from decrypt9wip.