nmlgc / ssg Goto Github PK

All MIDIs start with a GS Reset SysEx message (F0 41 10 42 12 40 00 7F 00 41 F7). I have a Yamaha keyboard that puts itself into a GS-"compatible" mode upon receiving this message, which effectively disables its high-quality remapping for GM instruments and thus makes the music sound worse than you would typically expect for a standard MIDI file.

In GM mode, we would

• replace this message with a GM Reset (F0 7E 7F 09 01 F7), and
• force the bank MSB (MIDI CC #0) to 0 on melody channels, and to 127 on the drum channel.

Bonus points for also adding an XG mode, which translates any of the GS SysEx messages to XG ones on the fly. This would first require us to fix the SysEx bugs in the original MIDI files, though.

Depends on #34.

Bundles the following issues, in order:

• #34
• #19
• #13
• #21
• #45
• #40
• #49
• #22
• #15
• #5
• #9
• #57
• Remove any remaining dependencies on Win32 APIs from core game code
• #39
• #4
• #41
• 64-bit builds
• Build the source code for Linux
• Add to the AUR
• Provide a Flatpak

This allows replay servers to verify the authenticity of replays by simply running the game and comparing its output to the expected metrics from the replay file.

Cellphone picture because it only appears in the original fullscreen mode where the regular Windows screenshot feature only gives you a black image, and the game's original screenshot feature (#15) isn't implemented yet. Will probably solve itself once we move away from DirectDraw (#4), but would otherwise be another bug we should probably fix.

As the code itself says:

The structure declarations above and the loading code itself point out exactly what these limitations are:

The RIFF container format used by .WAV files is organized in tagged chunks. The sound loading code ignores this fact though, and just assumes a standard .WAV chunk layout of fmt → data, and merely works around an optional fact chunk inbetween. Parsing the chunk structure correctly is not too hard, and there's no reason for not doing it.

This is only relevant for sound effect modding. Modders can easily insert .WAV files that have more header metadata, which wouldn't work with the original code, and the modder would have no idea. All .WAV files of the original game match these limitations.

Will become necessary before we add new fields to the CONFIG_DATA structure. The game needs to know how many bytes it should read, and we don't want loading to fail just because we've increased the size of the structure. Otherwise, configuration settings will be lost, and the Extra Stage will be re-locked.

The gameplay community is very interested in fixing the "fake deathbomb" quirk introduced in version 1.005: Visually, it might seem as if Shuusou Gyoku had deathbombs, but bombing shortly after getting hit won't in fact prevent losing a life, as such a bomb will be counted as a regular one on the next life. Version 1.0 did not have this issue, as it simply blocked bombing during invincibility periods.

However, since the issue fulfills the definition of a quirk, fixing it would create a replay-incompatible fork of the game, together with a new competition tier. But since we have source code, it's actually viable to introduce such a new tier by handling it as transparently as possible: A player should always know whether they're playing the original variant or our fork, and replays should come with a built-in indicator of the gameplay variant they were recorded on.
Since we're already distinguishing, we might as well provide both behaviors in the form of a switchable option. This will allow players to continue competing in the 1.005-compatible "fake deathbomb" tier while still getting all the technical improvements of our fork. And since we're already designing a new replay format, it also makes sense to directly build it in a way that can support mods. Heck, we might even use this to add a full-on "1.0" mode, in order to both preserve the difference between 1.0 and 1.005 and allow players to compete in this easier variant of the game.

• Implement the deathbomb fix behind a quirk toggle
• Add the deathbomb quirk toggle to the Config menu (requires #34)
• Display the state of the quirk toggle in one of the in-game side bars (got enough space there)
• Output non-1.005 replays with a different filename to signal incompatibility with the original release
• Add a hash of all ECL files to the replay header
• Add a further "gameplay core version" tag. This is also some kind of hash that identifies the entire C++ gameplay code. C++-level mods would change this to identify themselves, which allows another way of distinguishing modded replays even if the ECL scripts stay identical. Calculating this hash from the C++ source file text is probably overkill as it would turn mere refactors into entirely new games as far as the replay system is concerned. Then again, can we really be sure, even if we had a test suite with lots of replays? 🤔 Hashing C++ source code might still be the right thing to do, and we can always curate a list of known "refactor" hashes and treat those as identical to the original game. Or we could merely show a warning for replays that were recorded on a different gameplay core version.
  • If we do, it makes sense to isolate gameplay code as much as possible. Goes hand in hand with #37.
• Show ECL / gameplay core version in the replay menu… which we might want to enhance in general, see #36.

This feature would automatically launch a modal configuration window (in-engine, of course) in these three cases:

• The user first started the game and doesn't have a configuration file yet. This window would contain the most important settings on one screen:

  • Window / scaling / fullscreen (#7)
  • Volume (#1)
  • Language (#6 )
  • MIDI device and mode (#10)

• On later runs, some of the original configuration options might fail validation for multiple reasons: Display or MIDI devices may have changed, or the user might have tried to hack the configuration file in a hex editor and ended up with invalid values. In this case, the window could summarize all failed options with their intended value from the configuration file and the value they were reset to, and also provide the option to change them right from the same window.
  (Previously, such an error would have re-locked the Extra Stage (#30), but I removed this behavior in f00f90d.)

• The new config file versioning system implemented in #34 has to support old existing config files that don't yet contain any of the options that new builds might have added. The window would show all of these new options as a kind of "what's new" screen.

These three features are similar in upfront refactoring cost they require, so it makes sense to cover them within one big issue.

Just like with the map editor, I've been completely ignoring the contents of the SCLC/ directory so far. This one's mostly standard C code though, with just a few Windows types. We might as well make it fully cross-platform once we start looking at it.

The industry-standard base library for portable game engines. Since Ember2528 plans to fund a complete Linux port, it makes sense to move away from Win32 sooner rather than later. This way, we avoid writing any more Win32-exclusive code for things like SC-88Pro recordings, only for them to be rewritten in a more portable way later.

Further issues:

• SFX and future streamed BGM: #49
• Scaled and fullscreen window modes: #41
• Keyboard and joypad input: #22
• Graphics and rendering: #4

Non-issues:

• MIDI will continue to use pbg's original native Win32 code. SDL_mixer would be a strict downgrade, because it can only output to the default "MIDI mapper" device, which can no longer be easily changed as of Windows 7.

This obviously means that we stop maintaining all direct Win32 code. We're still going to leave the files in the repo, but just no longer compile them, allowing future pushes to still fund their continued maintenance. After all, the ReC98-adjacent community loves tinkering with old Windows systems…

The original game uses the native Windows GDI text rendering API. If we go cross-platform, we'd probably use FreeType?

Very open-ended, and basically the sum of a bunch of little features that I can deliver independently, depending on how much money comes in:

• Dump the original script in the translatable format
• Add a language setting to the Config menu (depends on #34)
• Dumb line-by-line replacement for in-game dialogue, endings, and UI text
• Dynamically growing the text box to fit more than 4 lines of dialogue (much better suited for this game than inserting new text boxes)
• In-game previewing (think thcrap skipgame/instant_ending)
• Automatic line breaks for in-game dialogue, endings, and the Music Room
• Image translations (we can pretty much copy the sprite-based overlay algorithms from thcrap_tsa there)
• The ultimate bonus feature: In-engine textbox editing

We can do better than storing a single replay per individual stage.

• Support full-game replays
• Add more metadata: Player name, timestamp, score per stage, shot type, …
• Implement a new replay menu that shows more metadata: Shot type, starting difficulty / lives / bombs, the above new metadata
  • Implement sortability by all types of metadata
  • Implement incremental search in all metadata categories

The graphics functions contain a bit of what looks like a 32-bit port of certain master.lib functions. Required for #4. Would take 1-2 ReC98 pushes, maybe?

8-bit-exclusive rendering calls

• SetLineWork()
• GrpHLineX()
• DrawTrapezoid()
• Grp_PClip()
• _2DGrpHline()

Math

• atan8()
• isqrt()
• rnd()

The 32-bit mode implemented in #3 made the game run well on modern Windows in general, but it seems to have introduced (or at least, not prevented) sporadic slowdowns at very specific places, for example during the defeat animation of the Extra Stage midboss. This happened to @WishMakers0 on stream:

https://clips.twitch.tv/CrowdedFlaccidAppleTBCheesePull-CWOmyHfFm1mcmZSf

We might need other DDrawCompat hacks after all.

Looks like this is possible with DirectDraw: http://archive.gamedev.net/archive/reference/articles/article960.html

Bonus points for selectable integer scaling factors.

Depends on #34.

While #3 implemented a proper 32-bit mode, DirectDraw's DWM8And16BitMitigation might still be active from running a previous version of a Shuusou Gyoku binary from the same path. This mitigation provides the emulated 8- and 16-bit modes even on modern Windows systems that don't support them, and are what causes the infamous slowdown in the first place. Ideally, we would remove this mitigation programmatically and not even offer players the option to select an actually unsupported, emulated video mode.

Doing that is not trivial hough:

• Removing the mitigation's registry keys manually requires admin rights, since they might be set system-wide under HKEY_LOCAL_MACHINE, in addition to HKEY_CURRENT_USER.
• Even if we did delete both keys programmatically, they wouldn't apply to the currently running process. DirectDraw checks for the state of the mitigation by calling DWM8And16Bit_IsShimApplied_CallOut() from apphelp.dll. This DLL is loaded into the game process at startup, and the flag read by the above function call is set on its DLL entry point, before any of our code gets to run.

That only leaves IAT patching as the most reliable method to override this flag, which is also what DDrawCompat arrived at:
https://github.com/narzoul/DDrawCompat/blob/7a59458d585fbf7eb1b243fc133c091bbdf561ce/DDrawCompat/Win32/DisplayMode.cpp#L276

Too complicated to handle within the same push as #3.

Requires input handling to be centralized first (#21), for the reasons mentioned in that issue. Changing to DirectInput 8 would remove the low-level keyboard hook used by DirectInput 7, which is also why just doing a raw version update wouldn't work.

Some scan results for comparison:

• DirectInput 7: Hybrid Analysis (with WH_KEYBOARD_LL), VirusTotal
• DirectInput 8: Hybrid Analysis (no WH_KEYBOARD_LL), VirusTotal

With DirectInput being long deprecated though, we might even want to upgrade to something newer.

This would also cover the alternative music modes implemented in #10 and #9.

Depends on #34.

If we ever do this, we also need to change the MIDI Port (保存はされません) help string in the main menu. 🙂

The current combination of /source-charset:utf-8 with /execution-charset:shift-jis allows us to have UTF-8 source files whose string literals are automatically translated to Shift-JIS at compile time:

This was necessary for the original codebase for two reasons:

• The *A() versions of GDI's text rendering APIs require Shift-JIS to render Japanese text with a SHIFTJIS_CHARSET font
• Some of the char buffers used to keep the interpolated strings in the main menu are sized to exactly fit Shift-JIS-encoded text, and would overflow if these literals used UTF-8:
  

  ssg/GIAN07/WindowCtrl.cpp

  Line 137 in e6267e9

  char InpHelp[34] = "パッド上のボタンを押すと変更";

The GCC and Clang option for this would be -fexec-charset, but Clang currently only supports UTF-8. Since any future text renderer is likely to require UTF-8 anyway, it's best to just migrate to /execution-charset:utf-8. Using UTF-16 internally would just complicate string interpolation, and having to do an explicit MultiByteToWideChar() in the GDI text renderer is definitely reasonable.

Will be the goal for ReC98 transaction T0165, so I'm going to do it myself. To ensure the clean-room property, I'm going to start from scratch on pbg's last commit (7dcab4f), replacing all code that wangqr contributed in 2019.

We can do better than just silently doing nothing whenever a replay can't be played back for any reason.

Allows even quicker restarts for everyone who doesn't play with the Wide Shot.

The cheaper option for compatibility with modern Windows. For a more thorough option that would also enable ports to non-Windows systems, see #4.

While the pre-built DDrawCompat DLLs are basically all you need to get Shuusou Gyoku running flawlessly on modern Windows, their nature of being proxy DLLs may introduce additional issues. In my testing today, they just flat out weren't loaded if there was a DWM8And16BitMitigation set for the binary's path in both

• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers and
• HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers

Wine users on Linux will also need to manually configure a DLL override.

This can be as simple and cheap as defining a DLL name that's different from ddraw.dll and statically linking to that one, or we could just look at its source code and manually fix the actual issues that require the DLL in the first place.

Might as well go fully cross-platform if we decide to change anything. We're probably looking at 5+ ReC98 pushes there, though.

Depends on #5.

Just in case someone wants to actually order it…

Thanks to KernelEx and Kstub822.zip, all my Shuusou Gyoku builds up to P0256 work on Windows 98 without any modifications. Here's a copy of Kstub822.zip, since it's getting increasingly hard to find.

However, KernelEx has never supported Windows 95, so we'd have to do some actual work:

• Link to unicows.lib to wrap all Unicode functions
• Set the MajorOperatingSystemVersion and MajorSubsystemVersion to 4, as part of the build process
• Compile with:
  • /arch:IA32 to not generate any SSE or newer instructions
  • /Zc:threadSafeInit- to avoid any references to the *SRWLockExclusive() functions, which are not wrapped by KernelEx or Kstub822
• Reimplement anything from the Visual C++ standard library that makes use of newer Win32 API functions
  If we're lucky and Clang's or MinGW's libc makes less use of these newer functions, it might make sense to wait for the Linux port where we change compilers.

-Tom-'s ShimAPI project might also come in handy here.

In the original game, they are rendered to a separate, larger rectangle that is then faded out as a whole. This cuts into the ending image on the left:

Notice how the staff member rectangle partly covers the gray shape in the left image with VIVIT.

With fully opaque staff member sprites, it looks correct.

The game crashes quite frequently in debug configurations due to what seems to be use-after-free bugs. Clearly defined pointer ownership can help here, and replacing all the LocalAlloc() and LocalFree() calls can also help with portability.

The 西方Ｐｒｏｊｅｃｔ lens ball animation (LENS.CPP) looks to be the most critical place that should be covered first.

The original DirectSound backend uses what seems to be a first-order low-pass filter when resampling the original sound effects to the native sampling rate. This adds a bunch of high-frequency content, which looks like this at 48000 Hz:

For the miniaudio backend implemented in #49, I went for a first-order low-pass filter as well. That gets quite close possible to the original DirectSound spectrum, and only adds a slight bit of additional crispiness to the high end:

However, some might argue that both are incorrect. If the original sound effects use sampling rates of 22050 Hz and 11025 Hz, shouldn't the spectrum look more like this?

This issue would deploy such a perfect resampler into the game, and add a toggle to the sound option menu for switching between accurate and crispy modes. Since the crispy mode is closer to what the game originally sounded like (and also sounds a lot better in my opinion), this feature is highly optional, and pretty much only meant for people who strictly define "accurate sound playback" as "not adding any frequency content that wasn't present in the original files".

Pausing and resuming a MIDI file is always going to be slightly awkward by just sending MIDI messages, but anything is better than just restarting the track.

As of P0217, these still require external locale emulation to work.

• Loading filenames with Japanese characters independent of locale
• Bonus feature: Checking for mojibake filenames and automatically fixing them
• Bonus feature: Checking for both mojibake and Japanese filenames, showing the difference between both files, and asking the user to pick one of them

Might be seen as a requirement if we want to port the game to systems with no native MIDI synth.
For the file/looping structure, we can just do what thcrap does; I got that part pretty much right in 2018.

The Romantique Tp recordings are the obvious source we could use to derive loopable versions for in-game use. However, they are riddled with timing glitches and feature other effects I can't explain; タイトルドメイド, for example, has quite a noticeable echo on the drums that can't be heard in BeatMARIO's stream (he used what seems to be a hardware SC-8850 there), nor in a Sound Canvas VA rip.

Therefore, there are two possible recordings we could use:

Romantique Tp recordings:
➕ Good reputation in the community and among real SC-88Pro owners
➕ Retain unspecified hardware quirks (including that weird reverb?)
➖ Timing glitches
➖ Loop cutting is annoying (and therefore expensive)

Sound Canvas VA rips:
➕ No timing glitches
➕ Loops will be perfect and inherently free of glitches – they can be constructed from separate renders of the intro and loop parts which are then spliced at exact sample positions, instead of being cut by ear in a trial-and-error process
❓ "It's just an emulation of the real thing"
➖ Romantique Tp will be angry (see https://www.shrinemaiden.org/forum/index.php?topic=18989.msg1276848#msg1276848)

• Loop cutting
  • Sound Canvas VA rips
    • Original soundtrack
    • Arranged soundtrack
  • Romantique Tp recordings
    • Original soundtrack
    • Arranged soundtrack
• Base decoding / playback functionality
  • Ogg Vorbis
  • FLAC
• PCM / MIDI soundtrack toggle Worked around by offering a separate MIDI-only BGM pack of the arranged soundtrack.
• Music pack selection
  • Open URL of BGM download page if no pack exists
  • Make sure to show the URL <Download> button even if the user already downloaded a pack
• Bonus points if we continue to silently decode the original MIDIs in the Music Room to retain that sweet note display. (We maybe should disable the tempo control feature then, though.) Works even with changed tempos!

Depends on #34.

Debug mode already has one, but we probably want one for release builds as well, and in all game states. We have more than enough screen space, after all.

The game calls Key_Read() in way too many places. It would be cleaner to do it once in WinMain(), and then pass the current inputs to the game state procedures as a parameter.

By itself, this would only be an architectural concern. What turns this into an actual bug is that the Game Over and High Score screens repeatedly call Key_Read() in a blocking while loop:

You really should not do this in a single-threaded Windows application. This completely prevents the Windows message loop from running, which in turn makes your program unresponsive to any sort of Windows event. Apart from pressing a key to end the loop, the only thing you can do in this state is to quit the program via the Task Manager or other equally forceful methods.

In fact, you would expect even key presses to be blocked in this state. It only works for the original game because DirectInput 7 installs its own low-level keyboard hook (WH_KEYBOARD_LL) that bypasses the Windows message system.

Fixing this would require a slight rearchitecture of the Game Over and High Score screens, replacing the blocking loops with proper wait states.

• "Validation" includes "checking the configured bit depth against the currently running GPU"
• "Reset" includes "re-locking the Extra stage"

Instead, invalid values should be reset to their defaults individually, while retaining valid ones.

• Basic functionality
• Bonus feature: UI feedback if any value has been reset

I would expect it to stay on Extra Stage, allowing me to immediately start a new round without moving the cursor down one option.

None of the functions in DirectXUTYs/MtoB.cpp have been implemented yet.

Separate from #40 because DxWnd exists and makes this technically optional. 😛 This issue is about the basic and rather simple SDL implementation, since SDL can easily handle all of these features; the hypothetical and unknowingly more difficult DirectDraw backport is tracked in #7.

• Windowed mode with arbitrary integer scaling up to the screen resolution
• Fake fullscreen
• True, mode-changing fullscreen (probably optional in this day and age)
• Option menu entries

Depends on #34 and #40.

Not as critical as the ones for replay files (#17), but there are quite a lot of potential file I/O and format validation error messages we could still write.

• #57

The game does support remapping the joypad, after all. The InputConfig structure also suggests that keyboard remapping was even planned at some point in development:

This would mostly be about adding the UI for it.

Both the original WinMM joypad support and the SDL one implemented in #22 are hardcoded to only read directional inputs from the first two axes of the joypad. This feature would add support for customizing these bindings in the Joy Pad menu, and also include support for POV hats and D-pads on controllers.

The drum notes in all MIDI files are very short (1/1920th notes), which causes them to not be played consistently on the standard Windows MIDI synth. This could be solved by lengthening them programmatically to 15/480 PPQN, or 128th notes, inside PBGMIDI.CPP.

The music starts playing again once you select a different track in the Music Room or start the game, but it would be nice to get immediate feedback in the main menu itself.

Win32 GDI can render Shift-JIS text even outside Japanese locale, but only if a Japanese font was selected into the DC. The garbled mess we're currently seeing in the ending (#19) is simply a result of the game not specifying any font for the ending text.

• Figure out the default GDI font on Japanese locale
• Explicitly set it

The game simply restarts a MIDI file from the beginning once it finished playing. That's also why the original MIDI files are rather large: They simply copy-pasted the loop section a couple of times to hide this nonexistent feature.

• Implement any or all of the common MIDI loop point conventions:
  • CC 2/4 (TH06-TH09)
  • CC 111 V=0/1 (RPG Maker)
  • CC 116/117 (XMI / EMIDI)
  • loopStart / loopEnd track markers (FF7 PC, HMI)
• Define loop points for the original MIDI files

So far, I've been completely ignoring the contents of the MapEdit2/ directory.

SDL's audio subsystem would be fully DIY and software-mixed, so we do have to use another library if we don't want to write our own panning and SFX/BGM looping code. SDL_mixer's architecture with a fixed amount of channels doesn't really fit a vintage DirectSound-based system with unlimited buffers, is quite bloated with codecs we'll probably never use, and would still require me to write BGM looping code myself. miniaudio, on the other hand, is pretty much perfect for want we want to do here:

• Directly portable to every platform that matters
• Can output via SDL_audio for platforms it doesn't support natively
• Sane defaults
• Decently documented
• Allows us to do everything we want to do in the foreseeable future in a single line of code
• Comes with exactly the codecs we need, and no more
• Made by the same developer who also wrote the best FLAC library back in 2018, which I used for thcrap's BGM modding

• Git tag
• Build timestamp
• Bonus feature: Unobtrusive update checking and notification

Very appropriate for issue #1.

Depends on #34.

We could either reuse the original one, or commission an artist to draw a new one if we want something more high-res and legally safer. Pretty cheap to implement if we do the former, while the latter will add whatever the artist charges.