snowymouse / invader Goto Github PK

The Warthog taillights do not work when they should. They should be turning on when you press jump while driving a Warthog.

Doors may act strangely when fully open, resulting in them starting to close and then open. Some doors also do not open when they should. Also, door lights are not set to the correct color.

Currently, if either the tag path in the tag itself or the index list use double slashes while the other does not, tags will get stubbed and given a different index id despite being the same.

The released tool.exe font tag creation feature seems to not work as well as whatever version bungie used for the xbox font tags. An open replacement is really needed. It would also be nice to create halo fonts without wine or windows too.

If you build maps on Linux and use forward slashes to point to the scenario in the invader command argument, the forward slashes will be used in the compiled maps tag paths.
Building using escaped back slashes works as intended.

For reference I think this is stupid and maybe there should be a switch for it, but it's too important to ignore.

The following tags have values changed by tool.exe if the map is for multiplayer.
weapons\pistol\bullet.damage_effect
weapons\pistol\pistol.weapon
weapons\plasma rifle\plasma rifle.weapon

Without this, it would be easy to accidentally build maps with differing weapon stats when using the indexer.

One example is the beginning of b40. While scripts work enough to play through the level, the sleeping grunts in the opening cut-scene are not removed when they should be.

This cannot be properly tested as Halo Custom Edition (as well as the retail version of the game and probably Halo: The Masterchief Collection) does not support these types of tags, but tool.exe does support building these tags.

Add tags to the scenario tag references that are referenced by a script in the scenario tag.

This is required for recursive tag extraction/rebuilding of certain tags that are otherwise only referenced by the script.

Generate sounds.map, loc.map, and bitmaps.map from indices generated by invader-indexer, since invader-indexer can now create indices from resource maps.

Essentially the opposite to -n. Tags would be indexed under the only condition of matching paths with the resource maps.

The use-case is if you are editing some tags that contain different stock tags that what is currently in the resource map, and you would rather use the resource tags regardless.

I know you could just update the tags, but this could become very tedious especially if dealing with multiple tag directories and/or ripped maps you are changing slightly.

Something that could be made in the future is a tool that can generate .bitmap tags. ImageMagick (e.g. Magick++) could be used to handle the conversion.

This tool should be made to support all formats that tool.exe supports (i.e. 16-bit, 32-bit, DXT, P8 bump, and monochrome).

I thought I'd open an issue for this to post ideas.

A compatible map is likely to use different bitmap and shader tags as it is one of the main reasons to use this feature. Thus lots of stubs for network-unimportant tags will occur.

Currently tags are stubbed by generating string lists with the class and full path.
This is rather messy in my personal opinion and not so good if someone extracts the map and has all this cruft around.

A possible way to remove stubs is to fill the gaps with non-network object tags that exist after the last network object with the highest index id. In the event a map runs out if these tags due to containing less tags than than the original map, a string list stub could be created with the name stub/<indexid> as a fallback. That is very unlikely though as you would have to go through swaths of ui tags first to get there.

Another option is to compile in all the missing tags to keep the map complete should someone want to do this. I know I complained about orphan tags before but then I realised you could just reference them in the scenario tag references array. I'd generate a tag collection of the missing tags and use that since this is used for scripts too and we might not want to fill it up with too many tags.

You could possibly allow the user to choose between fat builds with all tags and optimised builds with a command line switch. Fat builds are rather wasteful of tag data and possibly raw data though but has a use if someone wants to make a change without removing anything.

Technically these options could be done by building the map internally twice when using -w, the first used to know what tags are is missing if any or getting a list of tags that can be used to optimize away stubs. Then the second build would then be the final cache file.

This will be a bit of an undertaking compared to other things.

Basically, if structs have the same data, pointers, and dependencies, and they don't have to be contiguous with another struct, then they only have to be included once.

This error occurs when attempting to build The Silent Cartographer using Gearbox tags extracted with Refinery.

$ ./invader-build levels/b30/b30
ASSERT_SIZE failed: 69286 < ref_size(3437299804)
error adding reflexive reflexive.surface_indices
error adding reflexive #0 for reflexive.subclusters
error adding reflexive reflexive.subclusters
error adding reflexive #49 for tag.clusters
error adding reflexive tag.clusters
Failed to compile levels\b30\b30a.scenario_structure_bsp
Failed to compile levels\b30\b30.scenario
Failed to compile the map.
data is out of bounds

The option could possibly be -s --safe-indexing

It would take the tag count from the original bitmaps/sounds/loc and then right before indexing check if the id goes above that value when that option is used, if so the tag gets built into the map.

Lets say you are using full custom resources but you want to build a map that works with the original stock resource maps. Well, you could use this new option and it would never index a tag if the index id would be higher than what is in the original resource maps ensuring will work with them.

Map scripts will crash the game if they were compiled in with sapien beforehand. They only work if the scenario was ripped directly from a map built with tool.exe.

Testing some random tags I found that these H2 ones have some issues. The shotgun will not fire all of its rounds. There might be something off with the carbine effect too.

Using these tags, build levels/test/bloodgulch/bloodgulch_test
http://vaporeon.io/hosted/halo/misc/h2_tag_test.7z

It would be really handy if invader-bitmap could extract bitmap tags back to raw data, preferably in some a way where it could be recompiled into the exact same tag without modification while using the best source available.

This would be useful for both editing existing tags without worrying about changing sequences or re-encoding tags with lossless source data in them into lossless bitmap tags.

Currently invader-archive does not save timestamps in the created archive, so on extraction they get reset to the current time.

Timestamps can be handy to keep, for instance sometimes you might not be sure what you edited last and how long ago. One trick is to use something like find tags -type f -printf "%T@ %Tc %p\n" | sort -n to figure out when things were last changed.

Lightning tags, such as the one that appears when the monitor is interacting with The Pillar of Autumn in The Maw, do not render.

Attempting to build the gearbox d40.map results in the following error.

Cannot resolve script reference explosion_small.
terminate called without an active exception
Aborted (core dumped)```

Invader currently uses header files for tag definitions. While this is very fast as all offsets and the ways these values are read are determined at compile-time, it also makes it difficult to do things like iterate through tag values.

Ideally these definitions should be done in a parseable format such as .json or .xml, and then they can be parsed. We can then generate header files from these definitions when Invader is built so performance won't be impacted.

This can make custom resource maps even less of a hassle.

It seems the target point is incorrect, being lower than it should.
http://vaporeon.io/hosted/halo/misc/anhalo.png

https://cdn.discordapp.com/attachments/424172560272588807/503101634843246592/20181020_150313.mp4

The Pillar of Autumn's initial cutscene crashes the game within a few frames. Some other places where crashes occur include some parts of the map such as when Cortana tells you to "do what you do best" after meeting Captain Keyes or when the Australian Marine tells you the "Captain on the bridge needs you ASAP" near the beginning of the level just before meeting Captain Keyes.

This is most likely not an issue with scripts, themselves, because the script syntax data and script string data are identical to the original a10.map blocks.

It's also likely that some issues here may affect other maps.

Invader should check tags for errors after building the map. Some tags may be completely valid as tags but may have an invalid index which could result in a crash, either due to user error, data corruption, or a tool breaking something. It is worth noting that tool.exe does some amount of verification upon using build-cache-file.

This feature should be optional, as it may not be necessary every single time a map is built. Also, this feature should be usable on existing cache files, thus it may make sense to have this be its own program rather than be strictly part of invader-build (invader-build could still call it as a function).

This feature was suggested by MosesofEgypt.

If #36 is finished, it will be possible to write a GUI that uses Invader's tag definitions for parsing and editing tag data.

There is already a cross-platform tag editor, Mozzarilla, and while it works on Linux and everything functions, it is obviously designed for Windows, as the interface has some issues on Linux that make some elements unreadable on dark themes. It also is designed around building the maps with tool.exe, not invader-build, so it doesn't support cross-tag directory dependencies.

By using Qt, the user's Qt styling can be respected, resulting in an interface that works well on Linux and other systems that use Qt. Also, because this is made specifically for Invader, it can support Invader features such as cross-tag directory dependencies. This will allow the user to have a core set of tags and use additional tag folders for their maps without worrying about modifying existing tags folders.

Currently, here are the main ways people can generate lightmaps:

tool.exe

This involves generating lightmaps using the tool.exe lightmaps command.

Pros

• Very simple to use - you just run a single command for each BSP and the lightmaps get generated
• Built into the Halo Editing Kit

Cons

• tool.exe is closed source
• Resolutions are limited
• CPU utilization is limited, so it's inefficient

Aether

This involves using a tool to extract the BSP and scenery to hand off to a 3D modeling tool to generate the lightmaps.

Pros

• Comes with a GUI that does much of the hard work for you
• Produces high resolution lightmaps
• Most Halo CE map makers use 3ds Max or Maya

Cons

• Aether executable is closed source
• Requires using closed source, paid 3D modeling software (3ds Max or Maya)
• Requires using closed source tool.exe at least once in order to generate the UVs

Blender

This involves using a tool to extract the BSP and scenery to hand off to Blender Cycles to generate the lightmaps.

Pros

• Blender is free and open source
• Produces high resolution lightmaps

Cons

• Requires using closed source tool.exe at least once in order to generate the UVs
• Difficult and time consuming to set up (could be made a lot easier)
• Most Halo CE map makers do not use Blender and would have to learn how to use it

As you can see, all three of these have at least two things in common:

• They require the usage of tool.exe. The primary goal of Invader is to reduce and minimize the reliance on the Halo Editing Kit.
• They require the usage of closed source software at least once. I personally feel that "closed source" and "modding" are contradictory in a way.

Writing a tool will likely be a big undertaking. In fact, this probably is unlikely to ever be added in the future. However, making such a tool could get rid of a lot of our problems:

• No need to pre-generate lightmaps with tool.exe for UVs
• No need to ever once touch closed source software for lightmap generation
• It brings us closer to never having to use the original, broken Halo Editing Kit

It seems in some cases the generated stub tags will have no name, leading to duplicates in the compiled map.

This happened after trying to clean a hacked map of orphan tags by indexing, de-protecting and rebuilding it twice. I assume that because all the stubs were now literal string list tags, there would be name collisions and invader did not handle this.

Rather than directly fix this it may be better to slightly change the way missing tags are handled anyway. Instead of stubbing all of them and saving full paths in the map, you could instead backfill the empty slots with new tags, and in the event the map being built has less tags than the index source, generate empty strings lists for any remaining empty slots with the naming convention stub/<id>, or hell just null them if you don't mind doing that. As long as the game and refinery can open it then that is all that should matter, really.

The string lists and paths of stubs don't really do anything for the map other than waste tag space. That information is better off as a build time log rather than something stored in the map.

The antenna on the warthog is missing.

Many of the AI encounters appear to stand in place and fire their weapons from there rather than wander around. Here are some examples where it is especially noticeable:

• The Pillar of Autumn - If you kill anyone on the bridge, Cortana will send invincible marines after you. These marines are supposed to search for you, but they stay in the room they spawn in.
• The Silent Cartographer - The marines are supposed to storm the beach. Instead, they stay by the pelicans and only fire upon enemies that come too close to them. Strangely, if you clear the beach and Foehammer drops off a warthog, two marines will still go to the warthog.
• Two Betrayals - When opening the door to leave the control room, you will encounter some Covenant and Sentinel enemies. Normally, they're supposed to be fighting each other, but instead they won't shoot anyone until they come within range.

This will allow the resource map to work with cache files that use external offsets instead of indices (i.e. aXX, bXX, cXX, dXX, and ui maps built with tool.exe) without corrupted sounds and bitmaps appearing.

This will also allow for people on non-English copies of the game to play English maps without corrupted sounds and bitmaps appearing while having maps that properly use indices use the correct language assets.

Obviously this should only apply to sounds or bitmaps, not loc which can only be referenced with indices.

Anything with weather (rain, snow, etc.) will not build without this tag.

AI encounters are sometimes not placed in the correct BSP, resulting in them T-posing when in the BSP they're supposed to be.

This was partially fixed in #3 where all AI were defaulted to the first BSP, but sometimes they aren't placed in the correct BSP. This issue should probably be investigated.

Currently invader-bitmap and invader-font will output whatever the source name is, however I think all tags should be converted to lowercase on creation to ease the pain on case-sensitive filesystems as cache files themselves are not case sensitive.

One thing bungie/gearbox got right is all stock tags are lowercase, and current tag extraction tools will change any custom tags to lowercase too if they are not already. Invader could do it's part by ensuring tags are always lowercase when possible.

Some lens flares seem to only appear when looked at almost directly. An example is the one in The Silent Cartographer that appears at the end of the level when the pelican is in the shaft.

Play the game on Heroic or Legendary to skip the tutorial.

If the crewman makes it to the door before you do, then the door will not open.

The decals on the outside of both Blood Gulch bases do not render when the map is built with invader.

shader_transparent_water tags do not currently compile properly causing them to not render correctly in some cases.

And example would be the refined "glass" on b40/c40.

The glow tag is not implemented. This will prevent maps with the energy sword from building, including several campaign maps.

Detail object collection tags do not seem to appear when placed. This can be immediately noticed in the second level of the game, Halo (a30.map). Either it's an issue with the tag, or it is an issue with the scenario or BSP tags.

AI will cease to work when you leave the first BSP, and any encounters you find will not be active and/or T-pose unless they have a scripted animation, such as Captain Keyes.

Currently the tools work the same was as Halo CE's tool.exe in that all paths need to be relative to the data or tags folder the content in contained in.

While fine for most usage, an annoyance with this are that bash completion can not be used on input files. Another annoyance is that if you wish to use some automated script to work with multiple arbitrary files, say with find for example you would first need to change returned file system paths to be relative as what the tools expect.

I nice way to handle this would be to have an argument that instead changes input to be treated as explicit paths.

As for the problem of knowing where the content should exist in the tag tree, you could simply compare the given input path to the path(s) used for tags and data. since the input path once fully expanded must contain the full tag or data path with in it, anything after that would then be the tag tree path. Also if the tag/data path does not exist in the given input path you should get told to go away.

Something that could be made in the future is a tool to generate .sound tags. The encoder that comes with tool.exe is flaming hot garbage. Better versions of the Ogg Vorbis encoder have been made since 2002.

This tool should support outputting both Ogg Vorbis sound tags and Xbox ADPCM which is what tool.exe does.

The file size in the header should remain at 0. Halo only uses this value to check if:

• The file size is over 384 MiB (if so, it's invalid)
• The file size is at least 128 MiB (if so, Halo does weird things like not properly read the file name in the header as well as leak file size descriptors)

Having it be set to 0 will still allow the map to be loaded properly while also avoiding the above issues, allowing for multi-GB map files to be opened with stock Halo.

While map CRC's can already be changed with the MEK, having a small self-contained dedicated tool that only changes the 4 bytes needed to set the CRC would be advantageous, allowing for much quicker testing and usage of indexed map builds.

Something that may be useful in the future is a tool that can compile HSC scripts into a scenario tag. Perhaps this tool could also extract scripts as well, outputting it to the data folder.

Although tag extraction already exists, I think it would be good for invader to be able to extract tags form maps as part of the toolset it provides.

This extraction tool should be that only, A tool that simply extracts maps as they are without second guessing them due to the abundance of corrupt maps. Corruption repair AKA deprotection should be left to another program.

A tool could be created that takes the same input as invader-build, but rather then building a cache file, it adds every tag that would have been used in that process to a 7z archive.

This tool could also support multiple tag paths like the other tools, so for example if you ran invader-archive levels/whatever/map -t tags1 -t tags2, you would get an archive containing the tags that would have been used from each directory.

This feature would be great for storing tags for archiving or giving to someone else to use.

While you could just extract the map again, those tags have extra data stripped like the original source scripts and uncompressed pixel data in bitmaps, so having a tool to archive those without manually picking through tag directories that could contain thousands of tags each would be very helpful.