Giter Club home page Giter Club logo

devilution-comparer's Introduction

Devilution-comparer

Small binary comparison helper tool for devilution.

Generates an orig.asm and a compare.asm in the current directory and can watch the respective *.pdb for changes.

It can also generate a full disassembly of all specified functions in the config file (see the subcommand generate-full) for that.

Use --help for parameter info.

Example call:

devilution-comparer path\to\Diablo_orig.exe devilution\bld\WinRel\Diablo.exe InitMonsterTRN -w

Requirements

This uses Rust in the 2018 edition (so currently nightly only). In order to generate bindings to the Zydis library, you will also need clang/llvm to generate those.

Since the pdb crate doesn't support the old PDB file format generated by VC++ < 7 yet, this tool needs cvdump.exe from Microsoft (MIT License, license found in cvdump-LICENSE) in its directory. If you aren't on windows, this tool tries to run wine cvdump.exe instead.

The config file

The config file contains a mapping from function name/symbol to its offset (and optionally its size). It has to reside in the path of the devilution-comparer, just like cvdump.

It is specified in the TOML format, version 0.5.

The size element in the function definitions is optional, but is needed for some functions like the full export of the original file.

--help

Generates orig.asm and compare.asm in the current working directory. Finds the function specified in the devilution
binary, disassembles it, then disassembles the original binary with the same length at the specified offset. The
disassembled original code will be written into orig.asm, the devilution code into compare.asm.

Note that the disassembler will use the function offset read from the PDB for both decompilations in order to align the
addresses in the output files (including relative jumps).

USAGE:
    devilution-comparer.exe [FLAGS] <DIABLO_FILE> <DEVILUTION_FILE> <DEBUG_SYMBOL>
    devilution-comparer.exe [FLAGS] <SUBCOMMAND>

FLAGS:
    -h, --help           Prints help information
        --no-imms        Hides all immediate values. Use with caution.
        --no-mem-disp    Hide memory displacements and indirect calls. This cleans up the output tremendously, but can
                         cause you to miss wrong stack variables or globals. Use only with caution.
    -i, --show-ip        Shows leading addresses in the output.
    -V, --version        Prints version information
    -w, --watch          Enable watching for changes to the PDB file, updating the output files on change.

ARGS:
    <DIABLO_FILE>        Path to the original Diablo.exe to use
    <DEVILUTION_FILE>    Sets the debug binary file to use. The respective .pdb file needs to exist in the same
                         folder as well. Currently for files generated by VC6 only.
    <DEBUG_SYMBOL>       Function name/debug symbol to compare. This has to be defined for the original binary in
                         the comparer-config.toml. Is the size attribute missing, devilution-comparer will use the
                         size of the devilution function for the original binary as well.

SUBCOMMANDS:
    generate-full    Generates a 'full.asm' file with all functions defined in comparer-config.toml.
    help             Prints this message or the help of the given subcommand(s)

devilution-comparer's People

Contributors

ajenbo avatar dependabot[bot] avatar galaxyhaxz avatar mewmew avatar qndel avatar seritools avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

ajenbo justinhachemeister slo hankbao 9yx admtec chance4us starsurgical

devilution-comparer's Issues

Add all functions with their respective names and offsets/lengths

To make using devilution-comparer even easier, a full mapping from a function name to its respective offset and length in the binary would be awesome. We could remove the DIABLO_OFFSET_START parameter or make it optional then.

This would make using the tool to generate a full diff easier as well.

  • Gather the full function mapping for the 1.09b version and integrate that into devilution-comparer
  • Clean up the function definitions
  • Add size information to all function definitions Tail-call-optimized fucntions are still missing
  • Implement a global check mode for the riivaaja builds

A diff for a specific function

In devilution's #1122 @AJenbo responded about checking bin-exactness of a function in CI:

@sskras https://travis-ci.org/diasurgical/devilution/jobs/528380611 line 595
it's hard to have it any more automatic than that. It has been used to detect issues previously but for something like this, it's not of much help, though it does confirm the fix.

Thanks. The output looks still a little bit too scarse to be sure. I think it can be advanced. Will post details below. Please point me if I am missing something.

Complete the function mapping

You can find the size of the "function chunk" (since IDA doesn't treat that as an actual function) like this:

image

BTW: the rest of the unmapped functions in devilution:

WARN: Function 'FindRep' was not found in the config.
WARN: Function 'Expand' was not found in the config.
WARN: Function 'FlushBuf' was not found in the config.
WARN: Function 'OutputBits' was not found in the config.
WARN: Function '__SETS__' was not found in the config.
WARN: Function 'implode' was not found in the config.
WARN: Function 'GenDecodeTabs' was not found in the config.
WARN: Function 'explode' was not found in the config.
WARN: Function 'GenAscTabs' was not found in the config.
WARN: Function '__OFSUB__' was not found in the config.
WARN: Function 'DecodeDist' was not found in the config.
WARN: Function 'WriteCmpData' was not found in the config.
WARN: Function '$E1' was not found in the config.
WARN: Function '$E4' was not found in the config.
WARN: Function '$E5' was not found in the config.
WARN: Function '__ROL__' was not found in the config.
WARN: Function 'operator delete' was not found in the config.
WARN: Function '$E2' was not found in the config.
WARN: Function 'SortBuffer' was not found in the config.
WARN: Function 'WasteBits' was not found in the config.
WARN: Function 'DecodeLit' was not found in the config.

$E<number> are the c++ struct initializers, __NAME__ are IDA remains, and the rest are from the PKWare explore.cpp and implode.cpp.

Make displacements and indirect calls "local"

--no-mem-disp does a good job of cleaning up the diff and still show you the sizes. But leaves you in a situation where you can't know if you swaped one for another. Still even with out --no-mem-disp enabled it can be hard to track, take this as an exampel:
billede
It's easy to miss that line 32 is actually referencing the wrong value.
If we could instead lable them with a local reference the issue becomes much clear then what we get even with out --no-mem-disp:
billede

Show relative jumps as relative jumps instead of their calculated address

This should make the "diff noise" considerably smaller, for example in cases where there is an additional instruction shifting everything after that a few bytes.
This causes any line with a relative jump to show a diff, since the calculated absolute address is different, even though they all are correct/binary exact.

capstone-rs currently doesn't expose this info, so we'll probably either have to fork it or use the underlying capstone-sys, which are raw bindings to the lib.

Use .MAP file instead of .PDB

Currently we have the awkward situation where the VC6 linker is used to generate builds with a PDB file, since the VC5 linker can't process PDBs produced by the VC6 compiler. This means the builds end up with different layout, different startup stub, different static constructors etc...
But the VC5 linker can produce a .map file which includes the locations of all functions and variables in the final build, just like the PDB information. Could we switch to using it instead so we don't have to make separate builds?

Update function names

Some functions have recently been renamed meaning we get mismatches on full diff. Also part of render.cpp has been split in to inline sub-function to deal with the expected ASM usage in the original src so we might have to think about how to handle that when diffing.


WARN: Function 'encrypt_decrypt_block' was not found in the PDB.
WARN: Function 'encrypt_encrypt_block' was not found in the PDB.
WARN: Function 'encrypt_hash' was not found in the PDB.
WARN: Function 'encrypt_init_lookup_table' was not found in the PDB.
WARN: Function 'encrypt_compress' was not found in the PDB.
WARN: Function 'encrypt_pkware_read' was not found in the PDB.
WARN: Function 'encrypt_pkware_write' was not found in the PDB.
WARN: Function 'encrypt_decompress' was not found in the PDB.
WARN: Function 'mainmenu_action' was not found in the PDB.
WARN: Function 'Decrypt' was not found in the config.
WARN: Function 'GenAscTabs' was not found in the config.
WARN: Function '$E4' was not found in the config.
WARN: Function 'mainmenu_loop' was not found in the config.
WARN: Function '$E2' was not found in the config.
WARN: Function 'PkwareDecompress' was not found in the config.
WARN: Function 'PkwareCompress' was not found in the config.
WARN: Function 'SortBuffer' was not found in the config.
WARN: Function 'Hash' was not found in the config.
WARN: Function 'WriteCmpData' was not found in the config.
WARN: Function 'asm_trans_light_edge_1_3' was not found in the config.
WARN: Function 'DecodeDist' was not found in the config.
WARN: Function '__OFSUB__' was not found in the config.
WARN: Function 'asm_cel_light_edge' was not found in the config.
WARN: Function 'FindRep' was not found in the config.
WARN: Function 'explode' was not found in the config.
WARN: Function '$E5' was not found in the config.
WARN: Function '$E1' was not found in the config.
WARN: Function 'PkwareBufferRead' was not found in the config.
WARN: Function 'GenDecodeTabs' was not found in the config.
WARN: Function 'asm_trans_light_cel_1_3' was not found in the config.
WARN: Function 'PM_DoNewLvl' was not found in the config.
WARN: Function 'PkwareBufferWrite' was not found in the config.
WARN: Function 'implode' was not found in the config.
WARN: Function 'asm_trans_light_edge_0_2' was not found in the config.
WARN: Function 'WasteBits' was not found in the config.
WARN: Function 'OutputBits' was not found in the config.
WARN: Function 'DecodeLit' was not found in the config.
WARN: Function 'Encrypt' was not found in the config.
WARN: Function 'FlushBuf' was not found in the config.
WARN: Function 'InitHash' was not found in the config.
WARN: Function '__ROL__' was not found in the config.
WARN: Function '__SETS__' was not found in the config.
WARN: Function 'asm_cel_light_square' was not found in the config.
WARN: Function 'Expand' was not found in the config.
WARN: Function 'asm_trans_light_cel_0_2' was not found in the config.

Compiler warning about unhandled errors

When compiling with Rust version 1.33.0.

   Compiling devilution-comparer v0.3.5 (devilution-comparer)
warning: unused `std::result::Result` that must be used
  --> src/generate_full.rs:57:25
   |
57 | /                         writeln!(
58 | |                             stdout_lock,
59 | |                             "Note: Skipping '{}' because no size was defined.",
60 | |                             func.name
61 | |                         );
   | |__________________________^
   |
   = note: #[warn(unused_must_use)] on by default
   = note: this `Result` may be an `Err` variant, which should be handled
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

warning: unused `std::result::Result` that must be used
   --> src/generate_full.rs:126:21
    |
126 | /                     writeln!(
127 | |                         stdout_lock,
128 | |                         "WARN: Function '{}' was not found in the PDB.",
129 | |                         func.name
130 | |                     );
    | |______________________^
    |
    = note: this `Result` may be an `Err` variant, which should be handled
    = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

warning: unused `std::result::Result` that must be used
   --> src/generate_full.rs:134:17
    |
134 | /                 writeln!(
135 | |                     stdout_lock,
136 | |                     "WARN: Function '{}' was not found in the config.",
137 | |                     func.1.name
138 | |                 );
    | |__________________^
    |
    = note: this `Result` may be an `Err` variant, which should be handled
    = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

warning: unused `std::result::Result` that must be used
  --> src/hexformat.rs:17:17
   |
17 |                 write!(f, "+");
   |                 ^^^^^^^^^^^^^^^
   |
   = note: this `Result` may be an `Err` variant, which should be handled
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

warning: unused `std::result::Result` that must be used
  --> src/hexformat.rs:20:17
   |
20 |                 write!(f, "0x");
   |                 ^^^^^^^^^^^^^^^^
   |
   = note: this `Result` may be an `Err` variant, which should be handled
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

warning: unused `std::result::Result` that must be used
  --> src/hexformat.rs:24:13
   |
24 |             write!(f, "-");
   |             ^^^^^^^^^^^^^^^
   |
   = note: this `Result` may be an `Err` variant, which should be handled
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

warning: unused `std::result::Result` that must be used
  --> src/hexformat.rs:26:17
   |
26 |                 write!(f, "0x");
   |                 ^^^^^^^^^^^^^^^^
   |
   = note: this `Result` may be an `Err` variant, which should be handled
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

    Finished dev [unoptimized + debuginfo] target(s) in 4m 03s

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.