encounter / decomp-toolkit Goto Github PK

As in the title. This stops other functions from running due to it being returned as an error.

I've noticed that dwarf dump are setting classes as structs and structs as classes. Apparently in struct and class members it's defining in the right way. Here's an example:

objdiff joins the battle

I have to add these little tags to track these areas:

	.4byte 0x4C4C2069
	.4byte 0x6E202564
	.2byte 0x0A00
.endobj "@633"
	.2byte 0x0000

.obj "@634", local @ C44
	.4byte 0x4F534368
	.4byte 0x65636B41
	.4byte 0x63746976
	.4byte 0x65546872
	.4byte 0x65616473
	.4byte 0x3A204661
	.4byte 0x696C6564
	.4byte 0x20436865
	.4byte 0x636B5468
	.4byte 0x72656164
	.4byte 0x51756575
	.4byte 0x65287468
	.4byte 0x72656164
	.4byte 0x2D3E7175
	.4byte 0x65756529
	.4byte 0x20696E20
	.4byte 0x25640A00
.endobj "@634"

.obj "@635", local @ C88
	.4byte 0x4F534368
	.4byte 0x65636B41
	.4byte 0x63746976

The @ xxx areas would be nice if automatically emitted by DTK. It makes string lookup awful as I have to make sure I grab the right data for each symbol im adding to my WIP code.

Use case: Large assets embedded in the binary. decomp-toolkit should be able to extract and convert them to C headers that can be included.

Right now it has a few bugs:

• gap_ symbols are added
• Symbols with unknown visibility are set to global

Repository URL

https://github.com/ribbanya/melee/tree/8daa8eb261ccd111d7595bc73405607b2fb4cd5d

Game Name

Super Smash Bros. Melee

Game Version

GALE01

Description

dtk keeps placing a function, whose definition is in .text, but is referenced from .ctors, in .ctors. Well, more accurately, it keeps creating a ctors split where that function is. The function is __sinit_trigf_c.

I'm not sure this is a bug. I would be entirely unsurprised if this is due to us doing something wrong, since as you can see __ctors are not properly implemented in C.

• function asm (which links successfully with make)
• function src, which currently shows 25% in objdiff
• symbols.txt definition
• reference to the function that seems to be causing the issue
• split that dtk insists on making itself, even if it's deleted
• split where ctors is supposed to go (note that it will not successfully deserialize this splits.txt)
• error message when trying to create the above split
• dol info
• excerpt from the mapfile of an OK make build

Game: Bloody Roar: Primal Fury
Revision: 0
Region: NTSC (US)

 INFO Loading config\GBLE52\config.yml
 INFO Loading and analyzing 1 module (using 1 thread)
Failed: While loading object 'main.dol'

Caused by:
    Failed to finalize block @ 3:0x80084054

DTK currently throws a "cryptic" error when the start address of a split is greater than the end of the split.

 INFO Loading config\SB4E01\config.yml
 INFO Loading and analyzing 1 module (using 1 thread)
 INFO Initial analysis completed in 6.283s (found 50349 functions)
 INFO Rebuilding relocations and splitting
thread '<unnamed>' panicked at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce\library\alloc\src\collections\btree\search.rs:120:21:
range start is greater than ninja: error: rebuilding 'build.ninja': subcommand failed  
range end in BTreeMap
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Templates notably break module processing.

Tools parsing splits.txt separately will be able to use this info. Keep it as a comment maybe?

When I try to do a dtk dwarf dump on FIFA_d.elf from FIFA2004 Gamecube I get the error Failed: Unhandled user defined type ClassType. I've tried to create the class type, and that fixed the error, but when i tried the command again i got the error Failed: Unhandled ClassType child Inheritance

https://github.com/encounter/nod-rs

Luigi's Mansion's executable contains YAY0-compressed assets, so support for YAY0 decompression would be appreciated. A C implementation of the decompression algorithm can be found here.

Section alignment is handled haphazardly. Ways to improve:

• Honor splits.txt alignment values when writing ldscript.lcf
• Add alignment values to ldscript_partial.lcf, remove hacky code from rel make
• Guess alignment values in DOL loader

Repository URL

No response

Game Name

Spongebob Squarepants: Creature from the Krusty Krab

Game Version

USA V0

Description

When I run dwarf dump the structures are not populated correctly. Instead of being able to parse the struct, they get hit by and anonymous union at the tail end of it.

I did a simple fix by changing src/util/dwarf.rs line 1838 and changing it from let name = name.ok_or_else(|| anyhow!("Member without name: {:?}", tag))?; to let name = name.unwrap_or("".to_string());

I don't think that is a good fix but it did it for me. I think there might need to be a more specific fix for this.

Right now, a symbol size mismatch will error and halt the diff. This is annoying for symbols where the true size was discovered and symbols.txt hasn't yet been updated to reflect. If the next symbol is still at the correct address (meaning that alignment was not broken), omit a warning instead and continue.

While following dtk-template Getting Started using the OoT JP Wii VC 00000001.app (.app sha1 47546e48467ae14d712b8c207e911821586d1043) as the dol (renaming 00000001.app -> main.dol) I encountered the following crash:

$ ninja
[1/3] TOOL build/tools/dtk
Downloading https://github.com/encounter/decomp-toolkit/releases/download/v0.6.2/dtk-linux-x86_64 to build/tools/dtk
[2/3] SPLIT config/NACJ/config.yml
FAILED: build/NACJ/config.json
build/tools/dtk dol split config/NACJ/config.yml build/NACJ
 INFO Loading config/NACJ/config.yml
 INFO Loading and analyzing 1 module (using 1 thread)
 WARN module{name=oot_vc}: Conflicting size for Control: was 0x190, now 0x18C
 WARN module{name=oot_vc}: Conflicting size for Control: was 0x190, now 0x18C
 WARN module{name=oot_vc}: Conflicting size for InitMetroTRK: was 0x94, now 0x98
 INFO Initial analysis completed in 0.825s (found 4323 functions)
 INFO Rebuilding relocations and splitting
thread '<unnamed>' panicked at src/obj/symbols.rs:308:36:
index out of bounds: the len is 0 but the index is 18446744073709551615
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)
ninja: error: rebuilding 'build.ninja': subcommand failed

The same crash occurs if I manually run (on the 0.6.4 release)

./dtk dol config main.dol -o config.yml
./dtk dol split config.yml output

trying to split tp debug RELs (without using map option), it fails on certain RELs.

Causes issues in build.ninja

 INFO Loading default.elf    
 INFO Splitting 949 objects    
Failed: Range 0x00000000-0x80004380 outside of section .init: 0x80004000-0x80006728

relevant part from the info output:

          .text | 0x800562E4 | 0x4        | sqrt      
          .text | 0x800562E8 | 0x4        | strdup    
          .text | 0x800562EC | 0x54       | strlwr    
          .text | 0x80056340 | 0xA0       | stricmp   
          .init | 0x80004380 | 0x0        | gTRKInterruptVectorTable
          .init | 0x800062B4 | 0x0        | gTRKInterruptVectorTableEnd
          .text | 0x800563E0 | 0x0        | TRKAccessFile
          .text | 0x800563E8 | 0x0        | TRKOpenFile
          .text | 0x800563F0 | 0x0        | TRKCloseFile

Initially I thought this was because the symbol was 0 sized, but then had the same issue with a different ELF

Failed: Range 0x00000000-0x80004300 outside of section .init: 0x80004000-0x800066BC

           .bss | 0x804DF8E0 | 0x30       | __OSExpireAlarm
          .text | 0x80293190 | 0x158      | __OSInitPlayTime
        .rodata | 0x80465688 | 0x40       | @LOCAL@OSCalcCRC32__FPCvUl@crc32_table
          .text | 0x802932F0 | 0x134      | OSCalcCRC32
          .text | 0x80293430 | 0x204      | __OSRelaunchTitle
          .init | 0x80004300 | 0x24       | __init_hardware
          .text | 0x80293640 | 0x20       | __init_user
          .text | 0x802936B0 | 0x4C       | exit      
          .init | 0x80004330 | 0x34       | __flush_cache
          .text | 0x80293700 | 0x2C       | PSMTXIdentity
          .text | 0x80293730 | 0x34       | PSMTXCopy 

dtk fails to detect a relocation in GetMesMaxSizeSub in mario party 4.

 /* 800473F0 00044250  88 7F 00 00 */    lbz r3, 0x0(r31)
 /* 800473F4 00044254  38 63 89 A9 */    subi r3, r3, 0x7657
 /* 800473F8 00044258  7F A3 68 AE */    lbzx r29, r3, r13

the relevant code is this. 0x7657 points to mesWInsert-0x1 which is equivalent to winTabSize
this code translates to

var_r29 = mesWInsert[*var_r31 - 1];

Repository URL

No response

Game Name

F-Zero GX

Game Version

USA, Japan

Description

When attempting to analyze the F-Zero GX dol with the dol info command, control flow errors are returned. Furthermore, when attempting to merge the DOL and REL files together to create an ELF which can be statically analyzed, even more control flow errors are returned, along with a "Tail call analysis" failure.

> on Windows writes UTF-16. An -o option lets us write a UTF-8 file directly.

Repository URL

No response

Game Name

Wii Sports Resort

Game Version

USA v1.1

Description

While trying to analyze the game, I get the error of "Multiple sections with name .rodata". From what I understand, the dol file has a duplicate segment with the same name, but looking at it in IDA it seems like both contain different data. Is there any possible way of specifying the segment it should use?

Currently working on this branch: https://github.com/rjkiv/rb3/tree/binstream

Trying to match binstream.cpp, but am unable to merge BinStream::Cached() and BinStream::GetPlatform() with fn_8077BAA0, the original function these link to in the elf.

I should mention that fn_8077BAA0 hasn't actually been defined, but I'm not sure if defining that would fix this issue.

When running elf disasm on an (unstripped) ELF file (from a Wii game), it fails with the message:

Failed: Range 0x00000000-{e_entry} outside of section .init: {.init->sh_addr}-{.init->sh_addr + .init->sh_size}

It may not happen on every ELF file, but I tried it on at least 10 and they all had the same result, so I think it probably does.

An actual example, from an actual game:
Failed: Range 0x00000000-0x80004134 outside of section .init: 0x80004000-0x800064C4

Seems to be emitted from obj/sections.rs:152:

ensure!(
	self.contains_range(start..end),
	"Range {:#010X}-{:#010X} outside of section {}: {:#010X}-{:#010X}",
	start,
	end,
	self.name,
	self.address,
	self.address + self.size
);

I guess the problem is that it thinks the entry point is a range? No idea where that's determined, though.

i also don't know any rust i just grepped for the string

reproduction

1. get an applicable ELF file
2. dtk elf disasm <path/to/file.elf> .
3. message, maybe

environment

OS: Windows 10 22H2
dtk -V: dtk 0.5.5 36bb5ddcc67dabaf5d00d7335a9e764c4c668ee7

also tried on macOS Catalina with 0.5.4
same stuff but im not on that computer rn so no specs

Lots of elf files in the debugging.games archive fail to disassemble with similar errors. These look like linker-defined symbols which do not actually correspond to source locations. For example:

Megaman.elf:

Section symbol without section: Symbol { name: "_stack_end", address: 2149095232, size: 0, kind: Data, section: Absolute, scope: Dynamic, weak: false, flags: Elf { st_info: 17, st_other: 0 } }

smash.elf:

Section symbol without section: Symbol { name: "__SBSS2_END__", address: 2149515028, size: 0, kind: Data, section: Undefined, scope: Unknown, weak: false, flags: Elf { st_info: 17, st_other: 0 } }

dtk fails to split all versions Inazuma Eleven Strikers when symbols are applied.

With symbols_known turned off the analyzer doesn't seem to know these functions are local

Failed: While loading object 'main.dol'

Caused by:
    Multiple symbols with name longlong2str: 11989 Function 0x801E7D10 and 12018 Function 0x801EAC5C

Enabling symbols_known causes a cyclic dependency error (also on local functions it seems)

Failed: While processing object 'main.dol' (module ID 0)

Caused by:
    Cyclic dependency (involving auto_load_match_tex__Fv_text) encountered while resolving link order

repo can be found here

In the GC mario party games, there is a branch to every .text symbol in a giant list to make the symbols visible to the rels. When dtk encounter this, it throws an error about trying to disassemble data as code

Repository URL

https://github.com/wowjinxy/dtk-brawl

Game Name

Super Smash Brothers Brawl

Game Version

RSBE01

Description

E:\dtk-brawl>ninja
[1/3] TOOL build\tools\dtk.exe
Downloading https://github.com/encounter/decomp-toolkit/releases/download/v0.7.5/dtk-windows-x86_64.exe to build\tools\dtk.exe
[2/3] SPLIT config\RSBE01\config.yml
FAILED: build/RSBE01/config.json
build\tools\dtk.exe dol split config\RSBE01\config.yml build\RSBE01
INFO Loading config\RSBE01\config.yml
INFO Loading and analyzing 127 modules (using 16 threads)
ERROR module{name=sora_scene}: Failed to finalize functions:
ERROR module{name=sora_scene}: 0:0x000221D0
ERROR module{name=sora_scene}: 0:0x00022D3C
ERROR module{name=sora_scene}: 0:0x000234F4
ERROR module{name=sora_scene}: 0:0x0002442C
ERROR module{name=sora_scene}: 0:0x00024BA0
ERROR module{name=sora_scene}: 0:0x00032C3C
ERROR module{name=sora_scene}: 0:0x00036990
ERROR module{name=sora_scene}: 0:0x0003B16C
ERROR module{name=sora_scene}: 0:0x0003C26C
ERROR module{name=sora_scene}: 0:0x0003CFC8
Failed: While loading object 'main.dol'

Caused by:
Overlapping functions 3:0x802F7728-3:0x802F7910 -> 3:0x802F7ninja: error: rebuilding 'build.ninja': subcommand failed
744

Had this happen on multiple dols (eg Wrath of Cortex (GC)), when running dtk dol info sys/main.dol

 WARN Failed to locate _rom_copy_info    
Failed: While processing 1:0x800C4E74: FunctionSlices {
    blocks: {
        1:0x800C4E74: Some(
            1:0x800C4EB4,
        ),
        1:0x800C4EB4: None,
        1:0x800C4ED8: Some(
            1:0x800C4F04,
        ),
        1:0x800C4F04: None,
        1:0x800C4F14: Some(
            1:0x800C4F7C,
        ),
        1:0x800C4F7C: None,
        1:0x800C4F8C: Some(
            1:0x800C4FF4,
        ),
        1:0x800C4FF4: None,
        1:0x800C5004: Some(
            1:0x800C506C,
        ),
        1:0x800C506C: None,
        1:0x800C507C: None,
    },
    branches: {
        1:0x800C4EB0: [
            1:0x800C4EB4,
            1:0x800C4ED8,
        ],
        1:0x800C4F00: [
            1:0x800C4F04,
            1:0x800C4F14,
        ],
        1:0x800C4F78: [
            1:0x800C4F7C,
            1:0x800C4F8C,
        ],
        1:0x800C4FF0: [
            1:0x800C4FF4,
            1:0x800C5004,
        ],
        1:0x800C5068: [
            1:0x800C506C,
            1:0x800C507C,
        ],
    },
    function_references: {},
    jump_table_references: {},
    prologue: Some(
        1:0x800C4E74,
    ),
    epilogue: None,
    possible_blocks: {},
    has_conditional_blr: false,
    has_rfi: false,
    finalized: false,
    has_r1_load: false,
}

Caused by:
    Found duplicate prologue: 1:0x800C4E74 and 1:0x800C517C

I've tried to dump the Dwarf from Gamecube's FIFA 2004 and with the last version it worked really well, but I noticed that the dump missed some functions and global variables, even with their data present in elf. For example, the data from global gPassTri is present here:

But this data and the data around that are not present in the dump

Given a TU with correctly assigned symbol sizes, dtk should be able to infer the unpadded endings of splits, like this:

 melee/ft/chara/ftPurin/ftPr_SpecialHi.c:
     .text       start:0x8013C9C8 end:0x8013CE8C
-     .sdata2     start:0x804D9C10 end:0x804D9C20
+     .sdata2     start:0x804D9C10 end:0x804D9C1C
 
 melee/ft/chara/ftPurin/ftPr_SpecialLw.c:
     .text       start:0x8013CE8C end:0x8013D234
     .sdata2     start:0x804D9C20 end:0x804D9C2C

For MKDD. Could automatically extract binary symbols in this section and include them in ldscript.lcf.

I came across this when I was decomposing MSL; in particular, there's a file named sysenv.GCN.o as specified by some unstripped .elf files, so I added it to my splits

extern/msl/libc/embedded/sysenv.GCN.c:
	.text       start:0x804385FC end:0x80438604

but running dtk dol split (dtk/config.yml disasm) produces the following:

> ls -la disasm/obj/extern/msl/libc/embedded | grep sysenv
-rw-r--r-- 1 muffin None  540 Dec 17 15:15 sysenv.o

reproduction

1. add a file to splits.txt with a name with multiple periods in it like a.b.c
2. split the dol
3. check filename. will probably be missing infixes

environment

OS: Windows 10 22H2
dtk -V: dtk 0.6.4 0cfc5df20b4d03604649afb88897d4e011fc9b26