dga-mi-ssi / yaco Goto Github PK

One temporary workaround I've tried for dealing with #44 is to use git submodules.

Basically having one git repo which contains one folder/submodule per idb. There isn't real value in having that instead of just having a random folder containing the all the git repo (instead of submodules) but it could be just that you want your main repo with lots of stuff and a folder in that repo that contains your idb.

Anyway...

So the first thing when dealing with a git submodule is when you click "Edit > Plugin > YaCo" for the first time to create the "_local.idb" file it prompts for "Specify a remote origin" and fill it with "ssh://username@repository_path/". However if you do it for a clone of the git repo, it won't do it as it will detect successfully the git repo automatically.

Then also if you validate with a valid git repo or leave it blank, in both case it crashes IDA.

My guess is because the .git file is not a folder for a git submodule but it contains text that points to the real path for the git info, YaCo just does not support it.

gitdir: ../../../.git/path/to/git/submodule

Anyway, at the end I won't need that and will just clone several repos but thought I would mention that it crashes IDA in case someone else encounters this problem.

YaCo correctly detects a conflict when 2 users modify a comment on the same line without saving the IDB,
but the saved data when resolving the conflict is random.

Example : user1 sets comment "toto" at line 1 and user2 sets "titi" at the same line.
user1 saves the IDB first. All is OK. user2 saves the IDB and gets a popup telling him there is a conflict, with "toto" and "titi" at line 1. He gets a text input area to enter the comment to be saved. User 2 sets "toto".

Expected : comment at line 1 is "toto".

Result : comment at line 1 is "random" (mix of "?" and alphanum characters).

The result is the same if user2 sets "titi" or "tutu" or whatever input he sets to resolve the conflict.
The same bug exists with repeatable comments.

Tested with IDA Linux 6.95.

This is what I get when running YaCo in IDA (after applying #26 to fix path names)

YaCo: error during run
Traceback (most recent call last):
  File "/Applications/IDA Pro 7.0/ida.app/Contents/MacOS/plugins/yaco_plugin.py", line 88, in run
    import YaCo
  File "/Applications/IDA Pro 7.0/ida.app/Contents/MacOS/plugins/YaTools/YaCo/YaCo.py", line 35, in <module>
    import hooks
  File "/Applications/IDA Pro 7.0/ida.app/Contents/MacOS/plugins/YaTools/YaCo/hooks.py", line 23, in <module>
    import yatools
ImportError: No module named yatools

Obviously, there is no file called yatools in neither the YaCo nor the bin dir

YaCo/YaLibs/YaToolsLib/YaToolReferencedObject.hpp:18:10: fatal error: IModelAccept.hpp: Aucun fichier ou dossier de ce type
 #include "IModelAccept.hpp"
          ^~~~~~~~~~~~~~~~~~

I guess the file is missing ?

Please re-implement the 32-bit build target. I regularly use IDA to analyse 32-bit code and make extensive use of the Hex-Rays decompiler plugin. As you may know, the 64-bit version of the decompiler plugin cannot be used on 32-bit code. I would like to be able to use YaCo to collaborate with other people, but I don't want to have to give up on the decompiler.

yaida32 project failed to compile with the following message:
fatal error C1083: Cannot open include file: 'bytes.hpp': No such file or directory

YaCo fails to build with a format-security error.

Scanning dependencies of target yatools
[ 85%] Building CXX object CMakeFiles/yatools.dir/home/user/git/YaCo/YaLibs/YaToolsLib/XmlAccept.cpp.o
[ 85%] Building CXX object CMakeFiles/yatools.dir/home/user/git/YaCo/YaLibs/YaToolsLib/XmlVisitor.cpp.o
[ 85%] Building CXX object CMakeFiles/yatools.dir/home/user/git/YaCo/YaLibs/YaToolsLib/Configuration.cpp.o
[ 85%] Building CXX object CMakeFiles/yatools.dir/home/user/git/YaCo/YaLibs/YaToolsLib/Git.cpp.o
/home/user/git/YaCo/YaLibs/YaToolsLib/Git.cpp: In instantiation of ‘std::__cxx11::string {anonymous}::format_string(const char*, const Args ...) [with Args = {}; std::__cxx11::string = std::__cxx11::basic_string<char>]’:
/home/user/git/YaCo/YaLibs/YaToolsLib/Git.cpp:135:29:   required from ‘std::__cxx11::string {anonymous}::format_last_git_error(const char*, const Args ...) [with Args = {}; std::__cxx11::string = std::__cxx11::basic_string<char>]’
/home/user/git/YaCo/YaLibs/YaToolsLib/Git.cpp:141:52:   required from ‘void {anonymous}::fail_with({anonymous}::Git&, const char*, const Args ...) [with Args = {}]’
/home/user/git/YaCo/YaLibs/YaToolsLib/Git.cpp:155:13:   required from here
/home/user/git/YaCo/YaLibs/YaToolsLib/Git.cpp:123:40: error: format not a string literal and no format arguments [-Werror=format-security]
         const auto size = std::snprintf(nullptr, 0, fmt, args...);
                           ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/git/YaCo/YaLibs/YaToolsLib/Git.cpp:125:22: error: format not a string literal and no format arguments [-Werror=format-security]
         std::snprintf(&buffer[0], size + 1, fmt, args...);
         ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
CMakeFiles/yatools.dir/build.make:191: recipe for target 'CMakeFiles/yatools.dir/home/user/git/YaCo/YaLibs/YaToolsLib/Git.cpp.o' failed
make[2]: *** [CMakeFiles/yatools.dir/home/user/git/YaCo/YaLibs/YaToolsLib/Git.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:1316: recipe for target 'CMakeFiles/yatools.dir/all' failed
make[1]: *** [CMakeFiles/yatools.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

The build file configure_2017.cmd specifies the target as "Visual Studio 15 2017", which is not a valid target. Changing to "Visual Studio 15" fixes the error.

Hello,
with release 2.6, if I try to create a new repo on linux with ssh://git@server:path.git as repo URL, init fails with:

2018-11-15 16:11:06 repo: Repo created
2018-11-15 16:11:08 git: error unable to upload master to origin:master: malformed URL 'ssh://git@server:path.git'
2018-11-15 16:11:08 git: error unable to get reference from origin/master: no reference found for shorthand 'origin/master'
2018-11-15 16:11:08 repo: warning Master and origin/master does not point to same commit, please update your master
2018-11-15 16:11:08 repo: Current IDB filename is missing _local suffix

if I try with repo url git@server:path.git, I get an SSH agent prompt (so it's working), but then:

2018-11-15 16:05:44 git: error unable to upload master to origin:master: ERROR: Repository not found.

2018-11-15 16:05:44 git: error unable to get reference from origin/master: no reference found for shorthand 'origin/master'
2018-11-15 16:05:44 repo: warning Master and origin/master does not point to same commit, please update your master
2018-11-15 16:05:44 repo: Current IDB filename is missing _local suffix

and a local repository with name git@server:path.git is created.

Any idea ?

I had a _local.idb, after reinstalling my machine I enabled YaCo again by mistake when opening the IDB without _local.

If you do this then your original _local.idb is overwritten and lost forever.

I don't know if this is a limitation or bug so I figured I'd raise it.

My repo is here:

https://github.com/AliveTeam/YaCo

Fork make a change and save to repro. IDA will hang for up to 20+ seconds on save.

Hi,

After the plugin had asked for a remote origin IDA showed Running Python script waitbox. So IDA just freezes and I neither enter it nor close any of dialogs/IDA windows.

This usually happens when a python script asks for some input and the ask box is not closed for some time. This could be solved using execute_ui_requests...

Using yatools-nt_x64-v1.6-0, saving an IDB with a non-repeatable comment creates both non-repeatable and repeatable comments.

I can't find an indication one way or the other in the readme.

Copied the release package to the plugin directory and receiving this error:

Failed while executing plugin_t.init():
module 'idc' has no attribute 'GetIdbPath'
File "C:/Program Files/IDA Pro 7.5 SP3/plugins/yaco_plugin.py", line 63, in init
input_filename = idc.GetIdbPath()

Hi,

Sorry for the dumb question. From my testing, the answer is no and afaict we have to create one git repo per idb? Is that correct?

I tried to add a 2nd idb in the same folder and it looks like it imported the comments from the first idb.

I also tried to create a subfolder in the original git repo and to use an "empty" path when it asks for the git repo and it looks like it created another .git folder in the subfolder as if it wasn't already in a git repo already.

Thanks,

With up-to-date Fedora (gcc-8.1.1) fails to build with following error:

[ 94%] Building CXX object CMakeFiles/yadifflib.dir/build/YaCo/YaDiff/YaDiffLib/Matching.cpp.o
In function ‘yadiff::BinaryInfo_t::BinaryInfo_t(const IModel&, const yadiff::AlgoCfg&)::<lambda(const HVersion&)>::<lambda(const const_string_ref&, const const_string_ref&)>’,
    inlined from ‘static _Res std::_Function_handler<_Res(_ArgTypes ...), _Functor>::_M_invoke(const std::_Any_data&, _ArgTypes&& ...) [with _Res = ContinueWalking_e; _Functor = yadiff::BinaryInfo_t::BinaryInfo_t(const IModel&, const yadiff::AlgoCfg&)::<lambda(const HVersion&)>::<lambda(const const_string_ref&, const const_string_ref&)>; _ArgTypes = {const const_string_ref&, const const_string_ref&}]’ at /usr/include/c++/8/bits/std_function.h:282:44:
/build/YaCo/YaDiff/YaDiffLib/Algo/VectorSign/VectorTypes.cpp:171:20: error: ‘char* strncpy(char*, const char*, size_t)’ specified bound 256 equals destination size [-Werror=stringop-truncation]
             strncpy(format, val.value, FORMAT_MAX_SIZE);
             ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

After setting up a YaCo in the same folder, attempting to attach to a running binary (via debugger->attach to process) causes IDA 6.8 to crash. Verified this is a Yaco specific bug, not an IDA one, since using the non-local database can attach successfully, and renaming the _local database to anything else causes YaCo to alert that "the database doesn't match a YaCo project. YaCo is disabled" after which the database can successfully attach to the process.

I can upload the IDA dmp file if you believe that may be of use, but unfortunately there appear to be no other logs IDA uses to keep information about the crash. Is there some means of debugging YaCo I can use?

Windows 10, IDA 6.8.150423 32 bit with Hexrays, YaCo v1.5-6 (using distributed binaries)

Hello

I'm trying to use YaCo on a private remote git server, I'm using Windows as my git Client/IDA
and managed to connect to the remote repository, however when I try to Ctrl-W or YaCo tools to push
my initial idb, (I'm using the _local.idb), I don't see any changes being sent/committed,

The only error message I get is that something failed and I should look at git_rebase.log but the file doesn't exist, my idb_.log files show no sign of any action being performed and sometimes the "YaCo" toolbar doesn't respond, (it doesn't open the option menubar to show the desired capabilities).

I'm working on IDA 7.0, Windows, and a remote Linux-based Git repo.

Can you help me what I'm doing wrong? It seems there's some operation I'm missing.

Hi,

When a structure StructB is included in a StructA such as:

struct StructA {
	uint64_t	field_0;
	StructB		strb;
	uint64_t	field_x;
	uint64_t	field_y;
};

If the size of the StructB is increased by 8 bytes, the default behavior of IDA is to override the next field in the including structure: field_x in StructA in our example. However when doing this in YaCo it appears to undefine the strb field in the StructA breaking a lot of references.

We did not investigate where this problem come from.

Is there a way to extract the IDA informations stored about a binary and push it to a Polichombr instance?

The other way could be interesting too, as you could benefit from the autoanalysis features of Polichombr.

Unfortunately I don't know cmake well enough to fix this in general because this occurs in a cmake file that is generated by configure_2017.cmd

Basically this is what goes wrong:

CustomBuild:
Building Custom Rule C:/Users/Fridtjof Mund/Desktop/YaCo/build/CMakeLists.txt
CMake does not need to re-run because C:/Users/Fridtjof Mund/Desktop/YaCo/out/x64/CMakeFiles/generate.stamp is up-to-date.
yadb.fbs
Generating yatools_/git_version.dependency
CMake Error at yatools_/git_version.cmake:2 (execute_process):
execute_process given unknown argument "Mund/Desktop/YaCo".

This happens for me because my user directory has a space in it, which is how Windows 10 decided to create it.

I noticed that when a change is made to the function signature (args or return type), either through the type declaration dialogue (Y key) or through the decompilers "rename lvar" on an argument, this change is saved to one of the xml files, but is not applied to any of the other users db's, leading to inconsistencies.

Hi,

First, very good work about this plugin!

When a conflict is detected (below is an example for an anterior comment (INSERT hotkey)), a pop-up asks the user what value (local or remote) he wants to choose.

So if user 2 saved first his IDB hence pushed first this comment.

User 1 would get the following when saving his IDB:

It would be nice if we could copy the value from remote or if we could have buttons to replace local value with remote value (so we can actually copy it).

Also it may be useful if we could merge both comments. One way of doing it is to just allow copying both values so the user can merge them. Another approach would be to have a merge button that would append both comments.

Hello,

It appears that when a constant is set in IDA as being the size of the struct
it is not save in YaCo and when closing and re-opening the database it is
lost.

An example will be:
add eax, 20h becoming add eax, size MyStruct .
But when closing the IDA database and reopening it, it came back to being
add eax, 20h .

We did not investigate where the problem came from.

There is an issue when fetching: there is no TLS stream available.

I've tried to compile my own version, as well as using the released binaries to no avail.

Is there a workaround for this?

When using the configure scripts on msbuild (2015 or 2017) libxml2 will not be configured by cmake, and therefore the build fails.

CMake Error: CMake can not determine linker language for target: libxml2

I'm attempting to use YaCo v2.6 with IDA 7.0 on a 8051 code base that uses banking for code pages. The segment base is corrupted when opening the _local IDB. This appears to be caused by incorrect use of the ida_segment.get_segm_base() and ida_segment.set_segm_base().

The following steps can be used to replicate the issue:

• Create dummy binary file of size 0x20000

• Load into IDA select Intel 8051 leave rest as defaults and click OK until at disassembly view

• Open Program Segmentation and this is what you see:
  

• Modify Program Segmentation to values below
  

• Go to File->Script Command

• Enter following code

import ida_segment
# get segment base linear address
seg = ida_segment.get_segm_by_name("B02")
base = ida_segment.get_segm_base(seg)
print(base)  # this prints 32768 when should be 2048

# now when using set_segm_base(seg) it screws it up the base
ida_segment.set_segm_base(seg, base)

# get segment base linear address
seg = ida_segment.get_segm_by_name("B03")
base = ida_segment.get_segm_base(seg)
print(base)  # this prints 65536 when should be 4096

# now when using set_segm_base(seg) it screws it up the base for segment B03
ida_segment.set_segm_base(seg, base)

• Run script command
• Close and re-open Program Segmentation view and the BASE value has now been changed to incorrect values
  

YaCo is currently relying on get_segm_base() to return the value that should be used when calling set_segm_base(). This is incorrect. I believe you must shift the value obtained by calling get_segm_base() right 4 bits for the value to use when calling set_segm_base().

Ultimately I think IDA may have this screwed up because I would have expected these to be complimentary functions. I am unable to test this on IDA 7.1 so it could be possible that it may be corrected.

Hello,
I've just tried using YaCo using a remote repo which uses ssh keys with a passphrase.
It seems the current binary release can only use keys without passphrase.

I've just managed to use current git HEAD with an ssh-agent on Linux.

Is the ssh-agent auth going to work on Windows too ?

Either way it would be nice to have a small entry in the README to explain that :)

Are there any plans to update this to work with IDA 7.0?

Compiling (at least on Linux) is a bit complex on 64-bits systems:

 python2.7-dev:i386 : Depends: python2.7:i386 (= 2.7.13-2) but it is not going to be installed

And the ABI of IDA is stable. So it would be nice if you could provide precompiled binaries (using Github releases ?).

It seems YaCo does not preserve any changes made in the Local Types view.

Steps to reproduce.

1. Create new local type (e.g. struct my_struct { int a; };) and sync it with the Structures view.
2. Use the new type somewhere (e.g. make some dword_abcdef of type my_struct*).
3. Save.
4. Force discard IDB and pull.
5. Type information is lost (and we have import: error set_type: 0x89abcdef unknown type my_struct * in the output window).

Local Types are very useful and it would be nice to fix (if a bug) or implement this (if a missing feature).

IDA 7.0 Windows version
YaCo 2.2

The plugin seems to not manage agents to manage ssh-keys authentication with git repos.

INFO:YaCo:YaCo.save()
Traceback (most recent call last):
  File "plugins/YaTools/YaCo/ImportExport/YaToolRepoManager.py", line 459, in update_cache
    self.fetch_origin()
  File "plugins/YaTools/YaCo/ImportExport/YaToolRepoManager.py", line 388, in fetch_origin
    self.repo.fetch()
  File "plugins/YaTools/bin/YaToolsPy32.py", line 1956, in fetch
    return _YaToolsPy32.GitRepo_fetch(self, *args)
RuntimeError: Failed to authenticate SSH session: Unable to open public key file

Trying to install YaCo at my local PC, getting an error while running cmake:

$yaco/out/x64> cmake --build . --config RelWithDebInfo

Getting this error log:

InitializeBuildStatus:
Touching "yatools.dir\RelWithDebInfo\yatools.tlog\unsuccessfulbuild".
CustomBuild:
yadb.fbs
Generating yatools_/git_version.dependency
fatal: not a git repository (or any of the parent directories): .git
CMake Error at yatools_/git_version.cmake:9 (message):
C:/Program Files/Git/cmd/git.exe describe --dirty --tags --long: returned
128

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 1. [C:\Users\ami
tr\Documents\IDA\YaCo-master\out\x64\yatools.vcxproj]
Done Building Project "C:\Users\amitr\Documents\IDA\YaCo-master\out\x64\yatools.vcxproj" (default targets) -- FAILED.

Done Building Project "C:\Users\amitr\Documents\IDA\YaCo-master\out\x64\yadifflib.vcxproj" (default targets) -- FAILED.

Done Building Project "C:\Users\amitr\Documents\IDA\YaCo-master\out\x64_yatools_py32.vcxproj" (default targets) -- FAILED.

Project "C:\Users\amitr\Documents\IDA\YaCo-master\out\x64\ALL_BUILD.vcxproj" (1) is building "C:\Users\amitr\Documents\IDA\YaCo-master\out\x64\gtest.vcxproj" (23) on node 1 (default
targets).`

Has anyone encountered this? Can't figure out which repository is not a git repo, and why it should be.
Thanks!

I noticed that the function colors which we use for tracking decompilation progress (like orange = started, green = done) are not synchronized between different users. The colors seem not to be saved to the xml, so it would be nice if that could be added, as it's quite an important feature for us.

initially reported by dlul in issue #10, splitting in 2 bugs

When there is a conflict with a structure (user1 applies "toto.fieldA" and user2 applies "titi.fieldB" on the same line), the conflict is not detected, thus the last one that saves the IDB overwrites the structure.

Tested with IDA Linux 6.95.

yaco: WARNING: Failed to fetch origin, error: malformed URL 'ssh://[email protected]:HIDDEN/HIDDEN/HIDDEN.git'
Sensetive or unimportant stuff has been replaced with HIDDEN
Using the latest build from releases

Hi,

YaCo breaks a lot the arrays defined in structures.

It seems to happen a lot with arrays of structures, and when we have several arrays in the same structure ; when we define one, the other is not considered as an array anymore.

Here is a git diff of a commit that breaks structures:

diff --git a/cache/strucmember/65C3F15F2504B1BF.xml b/cache/strucmember/65C3F15F2504B1BF.xml
index 5217e0c..02e643e 100644
--- a/cache/strucmember/65C3F15F2504B1BF.xml
+++ b/cache/strucmember/65C3F15F2504B1BF.xml
@@ -7,7 +7,7 @@
       <parent_id>1D11CCBAE74D78E8</parent_id>
       <address>C0</address>
       <userdefinedname>internalArray</userdefinedname>
-      <proto>InternalStructArray /*%InternalStructArray#F594934A80AA7F38%*/[8]</proto>
+      <proto>InternalStructArray /*%InternalStructArray#F594934A80AA7F38%*/</proto>
       <flags>0x60000400</flags>
       <signatures/>
       <xrefs>


       
diff --git a/cache/strucmember/9F04AED9FA27A903.xml b/cache/strucmember/9F04AED9FA27A903.xml
index 1795d4c..6db84cd 100644
--- a/cache/strucmember/9F04AED9FA27A903.xml
+++ b/cache/strucmember/9F04AED9FA27A903.xml
@@ -3,7 +3,7 @@
   <strucmember>
     <id>9F04AED9FA27A903</id>
     <version>
-      <size>0x0000000000000020</size>
+      <size>0x0000000000000320</size>
       <parent_id>1D11CCBAE74D78E8</parent_id>
       <address>1930</address>
       <userdefinedname>ArrayConf</userdefinedname>

As you can see, sometimes it breaks the size, sometimes it breaks the prototype.
We investigated a bit in the code, and we think it might come from the simple_tif_to_string() function, that removes the pointer and arrays, and only recovers pointers...

As a result of such a commit, the array is undefined and set as unique element:

00000000 MyStruct        MyStruct ?               
00000010                 db ? ; undefined
00000011                 db ? ; undefined
00000012                 db ? ; undefined
00000013                 db ? ; undefined
00000014                 db ? ; undefined
00000015                 db ? ; undefined

Tested with IDA 7.1

I was attempting to set up YaCo, and tried to push the initialized repo and first commit to github. However, i get the error File -----.idb is 174.09 MB; this exceeds GitHub's file size limit of 100.00 MB. Checking the first commit, it appears YaCo is attempting to push the entire idb file, which for my binary is 174mb (it's an analysis of a large video game).

I've managed to get around this for right now by collecting garbage and deflating the database, but i was wondering if this was a design limit, and if there was a known workaround for overly large ida databases (such as git large file storage), or if a non git-based host should be used to store the repository.

Are there any plans to support macOS at some point?