unity-technologies / unitydatatools Goto Github PK

Experimental tools and libraries for reading and analyzing Unity data files.

License: Other

C# 99.33% ShaderLab 0.67%

unitydatatools's Introduction

UnityDataTools

The UnityDataTool is a set of command line tools showcasing what can be done with the UnityFileSystemApi native dynamic library. The main purpose of these tools is to analyze the content of Unity data files. You can directly jump here if your goal is to understand how to use the UnityDataTool command-line tool.

The UnityFileSystemApi library is distributed in the Tools folder of the Unity editor (starting in version 2022.1.0a14). For simplicity, it is also included in this repository. The library is backward compatible, which means that it can read data files generated by any previous version of Unity.

Note that the UnityFileSystemApi library included in this repository is a custom version containing an additional function that is required to support the SerializeReference attribute. This version of the library will be included in future releases of Unity.

What is the purpose of the UnityFileSystemApi native library?

The purpose of the UnityFileSystemApi is to expose the functionalities of the WebExtract and binary2text tools, but in a more flexible way. To fully understand what it means, let's first discuss how Unity generates the data files in a build. The data referenced by the scenes in a build is called the Player Data and is contained in SerializedFiles. A SerializedFile is the file format used by Unity to store its data. In builds, they contain the serialized assets in the target's platform-specific format.

When using AssetBundles or Addressables, things are slightly different. Firstly, note that Addressables are AssetBundles on disk so we will only use the term AssetBundle in the remaining of this document. AssetBundles are archive files (similar to zip files) that can be mounted at runtime. They contain SerializedFiles, but contrary to those of the Player Data, they include what is called a TypeTree¹.

Note: it is possible to generate TypeTrees for the Player data starting in Unity 2021.2. To do so, the ForceAlwaysWriteTypeTrees Diagnostic Switch must be enabled in the Editor Preferences (Diagnostic/Editor section).

The TypeTree is a data structure exposing how objects have been serialized, i.e. the name, type and size of their properties. It is used by Unity when loading an AssetBundle that was built by a previous Unity version (so you don't necessarily have to update all AssetBundles after upgrading a project to a newer version of Unity).

The content of a SerializedFile including a TypeTree can be converted to a human-readable format using the binary2text tool that can be found in the Tools folder of Unity. In the case of AssetBundles, the SerializedFiles must first be extracted using the WebExtract tool that is also in the Tools folder. For the Player Data, there is no TypeTree because it is included in a build and

The text file generated by binary2text can be very useful to diagnose issues with a build, but they are usually very large and difficult to navigate. Because of this, a tool called the AssetBundle Analyzer was created to make it easier to extract useful information from these files in the form of a SQLite database. The AssetBundle Analyzer has been quite successful but it has several issues. It is extremely slow as it runs WebExtract and binary2text on all the AssetBundles of a project and has to parse very large text files. It can also easily fail because the syntax used by binary2text is not standard and can even be impossible to parse in some occasions.

The UnityFileSystemApi library has been created to expose WebExtract and binary2text functionalities. This enables the creation of tools that can read Unity data files with TypeTrees. With it, it becomes very easy to create a binary2text-like tool that can output the data in any format or a new faster and simpler AssetBundle Analyzer.

Repository content

The repository contains the following items:

UnityFileSystem: source code of a .NET class library exposing the functionalities or the UnityFileSystemApi native library.
UnityFileSystem.Tests: test suite for the UnityFileSystem library.
UnityFileSystemTestData: the Unity project used to generate the test data.
TestCommon: a helper library used by the test projects.
UnityDataTool: a command-line tool providing several features that can be used to analyze the content of Unity data files.
Analyzer: a class library that can be used to extract key information from Unity data files and output it into a SQLite database (similar to the AssetBundle Analyzer).
TextDumper: a class library that can be used to dump SerializedFiles into a human-readable format (similar to binary2text).
ReferenceFinder: a class library that can be used to find reference chains from objects to other objects using a database created by the Analyzer

How to build

The projects in this solution require the .NET 6.0 SDK. You can use your favorite IDE to build them. They were tested in Visual Studio on Windows and Rider on Mac.

It is also possible to build the projects from the CLI using this command:

dotnet build -c Release

Disclaimer

This project is provided on an "as-is" basis and is not officially supported by Unity. It is an experimental tool provided as an example of what can be done using the UnityFileSystemApi. You can report bugs and submit pull requests, but there is no guarantee that they will be addressed.

Footnotes: 1: AssetBundles include the TypeTree by default but this can be disabled by using the DisableWriteTypeTree option.

unitydatatools's People

Contributors

Stargazers

Watchers

unitydatatools's Issues

Getting build target information from an asset bundle

I'd like to know if it's possible to use UnityDataTools for loading a built asset bundle and figuring out what is the BuildTarget it was built for.

I've looked at the example projects that are included in this repo but could not find any examples of such usage.

Not working on M1 mac

Unhandled exception. System.DllNotFoundException: Unable to load shared library 'UnityFileSystemApi' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libUnityFileSystemApi, 0x0001): tried: 'libUnityFileSystemApi' (no such file), '/usr/local/lib/libUnityFileSystemApi' (no such file), '/usr/lib/libUnityFileSystemApi' (no such file), '/Users/sjin/kbm-devspc/gitwork/UnityDataTools/UnityDataTool/bin/Release/net6.0/libUnityFileSystemApi' (no such file)
   at UnityDataTools.FileSystem.DllWrapper.Init()
   at UnityDataTools.FileSystem.UnityFileSystem.Init() in /Users/sjin/kbm-devspc/gitwork/UnityDataTools/UnityFileSystem/UnityFileSystem.cs:line 12
   at UnityDataTools.UnityDataTool.Program.Main(String[] args) in /Users/sjin/kbm-devspc/gitwork/UnityDataTools/UnityDataTool/Program.cs:line 16

It looks like it's missing the arm64-compatible UnityFileSystemApi.

The UnityFileSystemApi library is not backward compatible

The documentation says the following:

The UnityFileSystemApi library is distributed in the Tools folder of the Unity editor (starting in version 2022.1.0a14). For simplicity, it is also included in this repository. The library is backward compatible, which means that it can read data files generated by any previous version of Unity.

However, it does not seem to be the case. For example, when running UnityDataTool.exe dump on a file built by Unity 2021.3.33f1 using UnityFileSystemApi.dll from Unity 2022.3.22f, it fails to deserialize:

Invalid serialized file version. Expected version: 2022.3.22f1. Actual version: 2021.3.33f1.

Does this mean that UnityFileSystemApi must exactly match the version of Unity? If yes, the documentation should be updated.

.net 5 SDK reached end of life and no longer supported

Maybe moving to SDK 6.0 which is new LTS?

ShaderHandler.cs m_InsertSubProgramCommand need to put id in

"INSERT INTO shader_subprograms(shader, sub_shader, pass, pass_name, sub_program, hw_tier, shader_type, api) VALUES(@shader, @sub_shader, @pass, @pass_name, @sub_program, @hw_tier, @shader_type, @api)";

Should be:

"INSERT INTO shader_subprograms(id, shader, sub_shader, pass, pass_name, sub_program, hw_tier, shader_type, api) VALUES(@id, @shader, @sub_shader, @pass, @pass_name, @sub_program, @hw_tier, @shader_type, @api)";

Crashing when data files containt SerializeReferences

The tool will crash when there are SerializeReferences. There is a missing API in the UnityDataFileApi dynamic library and this will require some time to fix it. There is unfortunately no known workaround.

Update the `UnityFileSystem` library checked in the repo

The version of UnityFileSystem.{so, dylib, dll} checked into the repo should be updated to match newer versions of Unity. Since the library rejects archives built with newer versions of the Unity Editor.

For example, when running UnityDatatool locally, I get the following errors. (Which seems reasonable and more or less expected.)

Expected version: 2023.1.0a16. Actual version: 2022.3.4f1.
System.NotSupportedException: SerializedFile version not supported.

Full error message:

% ./UnityDataTool analyze "/unity-assets/"
objc[18223]: Class PlaceholderObject is implemented in both /src/github.com/Unity-Technologies/UnityDataTools/UnityDataTool/bin/Release/net6.0/osx-x64/UnityDataTool (0x100922ef0) and /src/github.com/Unity-Technologies/UnityDataTools/UnityDataTool/bin/Release/net6.0/osx-x64/libSystem.Native.dylib (0x109efd638). One of the two will be used. Which one is undefined.
Processing 3% (1/26) /unity-assets/Data/globalgamemanagersInvalid serialized file version. File: "/unity-assets/Data/globalgamemanagers". Expected version: 2023.1.0a16. Actual version: 2022.3.4f1.
Unknown error occurred while loading '/unity-assets/Data/globalgamemanagers'.

... snip ...

Error processing file /unity-assets/Data/Managed/Metadata/global-metadata.dat!
System.NotSupportedException: SerializedFile version not supported.
   at UnityDataTools.FileSystem.UnityFileSystem.HandleErrors(ReturnCode returnCode, String filename) in /src/github.com/Unity-Technologies/UnityDataTools/UnityFileSystem/UnityFileSystem.cs:line 75
   at UnityDataTools.FileSystem.UnityFileSystem.OpenSerializedFile(String path) in /src/github.com/Unity-Technologies/UnityDataTools/UnityFileSystem/UnityFileSystem.cs:line 49
   at UnityDataTools.Analyzer.SQLite.SQLiteWriter.WriteSerializedFile(String filename, String folder) in /src/github.com/Unity-Technologies/UnityDataTools/Analyzer/SQLite/SQLiteWriter.cs:line 168
   at UnityDataTools.Analyzer.AnalyzerTool.Analyze(String path, String databaseName, String searchPattern, Boolean skipReferences) in /src/github.com/Unity-Technologies/UnityDataTools/Analyzer/AnalyzerTool.cs:line 49
Processing 73% (19/26) /unity-assets/Data/Raw/AB/cap0007_f00_s0.abiFailed to load '/unity-assets/Data/Raw/AB/cap0007_f00_s0.abi'. File may be corrupted or was serialized with a newer version of Unity.

Note the warning about "Class PlaceholderObject" being implemented in two libraries. I assume that is benign because of the static linking when building with -p:PublishSingleFile=true -p:UseAppHost=true?

Errors When Updating

From the README.md, it sounded like fixing this should just be a matter of referencing newer versions of the UnityFileSystemApi library. However, this doesn't pan out.

I downloaded the latest version of the Unity Editor. And confirmed there was a UnityFileSystemApi.dylib file in the /Tools directory. (But no .dll or .so, which makes sense since I'm on a Mac.)

After copying the new UnityFileSystemApi.dylib into the same folder as UnityDataTool, or setting DYLD_LIBRARY_PATH="/Applications/Unity/Unity.app/Contents/Tools" it appears that the library can no longer be loaded.

There is the same error about Class PlaceholderObject, but a new one Unable to load shared library 'UnityFileSystemApi' or one of its dependencies..

./UnityDataTool analyze "/unity-assets/Data/"
objc[17153]: Class PlaceholderObject is implemented in both /src/github.com/Unity-Technologies/UnityDataTools/UnityDataTool/bin/Release/net6.0/osx-x64/UnityDataTool (0x1009dfef0) and /src/github.com/Unity-Technologies/UnityDataTools/UnityDataTool/bin/Release/net6.0/osx-x64/libSystem.Native.dylib (0x109ffe638). One of the two will be used. Which one is undefined.
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'UnityFileSystemApi' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libUnityFileSystemApi, 0x0001): tried: 'libUnityFileSystemApi' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibUnityFileSystemApi' (no such file), '/usr/lib/libUnityFileSystemApi' (no such file, not in dyld cache), 'libUnityFileSystemApi' (no such file), '/usr/local/lib/libUnityFileSystemApi' (no such file), '/usr/lib/libUnityFileSystemApi' (no such file, not in dyld cache)
   at UnityDataTools.FileSystem.DllWrapper.Init()
   at UnityDataTools.FileSystem.UnityFileSystem.Init() in /src/github.com/Unity-Technologies/UnityDataTools/UnityFileSystem/UnityFileSystem.cs:line 12
   at UnityDataTools.UnityDataTool.Program.Main(String[] args) in /src/github.com/Unity-Technologies/UnityDataTools/UnityDataTool/Program.cs:line 16
   at UnityDataTools.UnityDataTool.Program.<Main>(String[] args)
zsh: abort      ./UnityDataTool analyze

Since UnityFileSystemApi.dylib clearly exists, I'm guessing the error is actually because of the "or one of its dependencies" part? However, it's unclear what dependency it is looking for.

In case it was related, I did check that Terminal.app can "run software locally that does not meet the system's security policy". (In case the error is from the .dylib file not being signed or something.) But that didn't seem to change anything.

Summary

The UnityFileSystemApi libraries checked into this repo are now a little dated, meaning that out of the box you cannot build and run the tool to inspect newer asset files.

In theory, it's just a matter of checking in a newer build of those libraries into the repo. However, at least with my understanding of things, copying over newer versions of the files isn't quite so easy.

Possible to analyze sharedassets?

Hi,

Is it possible to use this tool to analyze/dump the data for a sharedassets file (e.g. sharedassets1.asset), or does it only work with asset bundles? I've been able to successfully analyze/dump assetbundles, but not the sharedassets, so I just want to know if that's intended to be supported, so I know whether it's worth me digging into it further or not...

Thanks

Usage from within Unity

I tried using these tools from inside Unity to analyze import artifacts and for build postprocessors of generated data, which to be best of my understanding fall under SerializedFile.

However, it seems that the app doesn't properly work with domain load and unload, after a first attempt I'm always getting

for subsequent tries.
Are there plans to get this to work?

I found a relevant comment in the UnityFileSystem constructor that leads me to believe this is a bug / not implemented:

static UnityFileSystem()
{
    // Initialize the native library.
    var r = DllWrapper.Init();
    HandleErrors(r);
            
    // TODO: should Cleanup be called by the AppDomain.Unload event or something else?
}

UnityDataTool raises an exception if non-English characters are included.

Correct Behavior

Works correctly if the path is only in English

PS C:\Users\kotaro.nakajima\Iroiro\testDir> ..\UnityDataTool\UnityDataTool.exe analyze . -p '*.pkg' -o test.db
Processing 100% (1/1) asset.pkg
Finalizing database...

Total time: 0.339 s

Error Behavior

An error will occur if Japanese or emoji are included.

PS C:\Users\kotaro.nakajima\Iroiro\テスト> ..\UnityDataTool\UnityDataTool.exe analyze . -p '*.pkg' -o test.db

Error processing file C:\Users\kotaro.nakajima\Iroiro\テスト\asset.pkg!
System.IO.FileNotFoundException: File not found.
   at UnityDataTools.FileSystem.UnityFileSystem.HandleErrors(ReturnCode returnCode, String filename) in D:\Soft\UnityDataTools\UnityFileSystem\UnityFileSystem.cs:line 66
   at UnityDataTools.FileSystem.UnityFileSystem.MountArchive(String path, String mountPoint) in D:\Soft\UnityDataTools\UnityFileSystem\UnityFileSystem.cs:line 33
   at UnityDataTools.Analyzer.AnalyzerTool.Analyze(String path, String databaseName, String searchPattern, Boolean skipReferences) in D:\Soft\UnityDataTools\Analyzer\AnalyzerTool.cs:line 39

Finalizing database...

Total time: 0.040 s

PS C:\Users\kotaro.nakajima\Iroiro\😋> ..\UnityDataTool\UnityDataTool.exe analyze . -p '*.pkg' -o test.db

Error processing file C:\Users\kotaro.nakajima\Iroiro\??\asset.pkg!
System.IO.FileNotFoundException: File not found.
   at UnityDataTools.FileSystem.UnityFileSystem.HandleErrors(ReturnCode returnCode, String filename) in D:\Soft\UnityDataTools\UnityFileSystem\UnityFileSystem.cs:line 66
   at UnityDataTools.FileSystem.UnityFileSystem.MountArchive(String path, String mountPoint) in D:\Soft\UnityDataTools\UnityFileSystem\UnityFileSystem.cs:line 33
   at UnityDataTools.Analyzer.AnalyzerTool.Analyze(String path, String databaseName, String searchPattern, Boolean skipReferences) in D:\Soft\UnityDataTools\Analyzer\AnalyzerTool.cs:line 39

Finalizing database...

Total time: 0.032 s

Exit codes from handler functions not propagated back to Main

Return codes from the handler functions defined in UnityDataTool/Program.cs don't get propagated back to the Main function. That means we get exit code 0 from the executable even when a handler has specifically defined a non-zero exit code.

Example:

$ UnityDataTool\bin\Release\net6.0\UnityDataTool.exe archive list README.md
Error opening archive!

$ echo %errorlevel%       
0