Hi,

I tried a fresh install of the project into Laravel. It doesn't work at all.

All the aoe2record files I tried die with a "Can't read 4 bytes" exception while trying to set $initCameraY (PlayerInfoBlockAnalyzer.php, line 140), and all the mgz files I tried die with a "Got invalid map size" exception.

Are there problems with the current php version (7.2)?

As title, wondering if there are plan on making this work with DE. Or if anyone knows how much work it would be to support DE.

looks like I did add research icons in #41, but not civ icons.

In src/Model/GameSettings.php :

1. line 162-170

    /**
     * Returns game speed string.
     *
     * @return string
     */
    public function gameSpeedName()
    {
        return $this->rec->trans('game_speeds', $this->gameSpeed);
    }

$this->gameSpeed could be one of 100, 150, 200, but corresponding array in translation file is:

  'game_speeds' => 
  array (
    0 => 'Slow',
    1 => 'Normal',
    2 => 'Fast',
  ),

I resolved the problem by changing 0, 1, 2 to 100, 150, 200 .

2. line 140-150

/**
     * Returns map style string.
     *
     * @return string
     */
    public function mapStyleName()
    {
        $mapStyle = $this->rec->getResourcePack()
            ->getMapStyle($this->mapId);
        return $this->rec->trans('map_styles', $mapStyle);
    }

Funtion getMapStyle() in this snippet of code seems doesn't exist at all?

Thanks.

Usually the first four bytes of a recorded game store the size of the compressed header of the file, but save/exited multiplayer games store 0 instead. They are actually valid recorded game files, though. RecAnalyst should attempt to guess the compressed header size somehow.

Uhh. Sounds like I'm drawing the terrain wrong.

I'm getting Yucatan from mapName() from an HD Edition game played on Salt Marsh. Real World maps are fine though, and at least Arena and Arabia are too, so not sure where it goes wrong.

Thanks for your awesome work.

But the link to API doc in Readme.md seems broken?

Is there a new place for API documentation?

Thanks again.

I scanned through documentation and some code, but I don't see anything to read the actual game play information instead of just stats and initial points.

A couple of questions if you feel like answering, otherwise I just need to do more research.

• Is the record game file writen in useful chunks that one could parse gameplay out of it as the game proceeds? (assuming written at some regular interval)
• Alternatively do you know of documentation (or better yet code) for acting as an observer client? I assume voobly has some sort of code or perhaps purely a packet proxy for its observer mode.

Basically, I am looking to read the actual state changes of the game (preferably during gameplay, but postprocessing would be a good first step) to supplement the UI.

Since you seem to have experience digging into some of this stuff just looking for some thoughts if you have time. Thanks.

Likely not many changes in the format from 4.8 (#27). However will need to make sure that the Openage convert script can handle the new terrains etc., and add the resources here.

Version info:

Patch version: 5.0
Version string: VER 9.4
Subversion number: 12.50

::1:52854 [500]: /test.php - Uncaught exception 'Exception' with message 'Stream read error' in /home/boombatower/software/recanalyst/src/Stream.php:125
Stack trace:
#0 /home/boombatower/software/recanalyst/src/MemoryStream.php(167): RecAnalyst\Stream->readBuffer('\x00\x00\x00 \x01\x00\x00\x00\x02\x00\x00\x00\v\x00\x00...', 16777216)
#1 /home/boombatower/software/recanalyst/src/RecAnalyst.php(533): RecAnalyst\MemoryStream->readString('\x00\x00\x00 \x01\x00\x00\x00\x02\x00\x00\x00\v\x00\x00...')
#2 /home/boombatower/software/recanalyst/src/RecAnalyst.php(1354): RecAnalyst\RecAnalyst->analyzeHeaderStream()
#3 /home/boombatower/software/recanalyst/test.php(16): RecAnalyst\RecAnalyst->analyze()
#4 {main}
  thrown in /home/boombatower/software/recanalyst/src/Stream.php on line 125

Looks like it's getting the player name string length wrong and trying to read way too many bytes. Obviously, that could mean the parser got off track long before that.

SWGB uses mgx files, and as far as I can tell they're parsed perfectly fine as is. Supporting this would be pretty cheap. Needs a way to detect SWGB games vs. AoC games, and a different id → name mapping (eg. map IDs and unit IDs) to give decent results.

See https://www.aoezone.net/threads/userpatch-v1-5-beta-r7.138902/

Looks like the VersionAnalyzer needs to check for 9.F

Looking at the changes in the patch announcement post, it mentions there are dat file changes and changes in command structure.

• Command packets now include incremented IDs.
• Fixed frequent replay desync. Old replays from patch 4.7 or earlier will almost certainly desync due to dat file changes.

So we might need to skip the command packet ID in here somewhere:
https://github.com/goto-bus-stop/recanalyst/blob/master/src/Analyzers/BodyAnalyzer.php#L203-L208

And the data changes may require changes in the header analysis, but I'm not sure. We should add one or more HD 4.8 records in the test/recs/versions directory to check.

WololoKingdoms recorded games can be read, but some of its unit IDs are different, eg because of:

https://github.com/Tails8521/WololoKingdoms/blob/master/fixes/ai900unitidfix.cpp

We can deal with that in ResourcePacks\AgeOfEmpires::normalizeUnit, if we can detect when WololoKingdoms is in use.

https://symfony.com/doc/current/components/translation.html

Could auto-extract translations from the HD Edition language files.

#39

Continuing where that left off. I actually posted an answer there but it was probably suppressed as a notification for being in a closed thread (and couldnt find how to re-open it myself - cause i probably cant).

After your reply i tried that.
It seems to be throwing a 500 fatal error:
mod_fcgid: stderr: PHP Fatal error: Class 'RecAnalyst\RecordedGame' not found in /recanalyst/recanalyst.php on line 14, referer: /recanalyst/index.html

It's on a php 5.6+ host so that's out of the question.

The current code is as follows:

<?php

require 'vendor/autoload.php';

use RecAnalyst\RecordedGame;

$array = array();
$a = __DIR__ . "/games/" . $_FILES["file"]["name"][0];
move_uploaded_file($_FILES["file"]["tmp_name"][0], $a);
$array[] = $_FILES["file"]["name"][0];
foreach ($array as $val) {
    $filename = __DIR__ . '/games/' . $val;
// Read a recorded game from a file path.
    $rec = new RecordedGame($filename);
// Render a map image. Map images are instances of the \Intervention\Image
// library, so you can easily manipulate them.
    $rec->mapImage()
        ->resize(240, 120)
        ->save('minimap.png');
// Display players and their civilizations.
    echo 'Players: ' . "\n";
    foreach ($rec->players() as $player) {
        echo ' * ' . $player->name . ' (' . $player->civName() . ')' . "\n";
    }
    echo 'Minimap saved in minimap.png.' . "\n";
}

lichifeng made a lot of great improvements in their fork, it would be nice to backport them where applicable.

A lot of functions require referencing images in Resources folder which could not been found in this repo, I have looked into the biegleux latest version in Sourceforge and found some old resources. However, that project is only designed for AOC not AoFE, some of the images are missing. Could you also put the Resources folder you used for testing to the repo so that we could use it without regenerating the images for AoFE?

It seems game records generated on different Windows versions have different character encodings. Especially when dealing with game records from non-latin users, character encoding is a headache. Player names and chat messages cannot always display correctly.

I tried to resolve this with mb_detect_encoding() and mb_convert_encoding() but failed. It is hard for mb_detect_encoding() to make a good guess( Maybe a player name is too short? )

Since I mainly use recanalyst to analyze game records from chinese users, I simply decode strings extracted from records with GBK( common encoding for chinese characters) and then encode them with UTF-8. The result of this solution is acceptable for me, but apparently dirty and not elegant.

So here is my question: Is there a way to know encoding of strings in records explicitly?

Thanks.

Where do we start?

• Include PHP-SVG package
• Make changes to MapImage.php

From aiscripters:

WK is mod id 1, PCM is mod id 2, etc. With this, a rec analyzer can check the first player's resource 198 (float) and, if it exists, divide it by 1000 to get the mod id, modulo 1000 to get the version id, and split that with dots. For WK's 1572, the mod id (1572 / 1000) is 1 (WK) and version id (1572 % 1000) is 572. Combined with the name and split with dots: WK 5.7.2. If resource 198 doesn't exist (or it's 0 and the feitoria resource is also 0), the rec analyzer can just display that it's aoc. If it's 0 and the feitoria resource isn't 0, it's probably older WK.

At HeaderAnalyzer.php line 178:
mapsize is correctly parsed when position is padded by 60 instead of 58

At PlayerObjectsListAnalyzer.php line 346:
isMgx is false for the recorded game
but aokObjectEndSeparator is not found
instead, objectEndSeparator is found

At PlayerObjectListAnalyzer.php line 234:
the original playerInfoEndSeparator exist only if the player is GAIA
for other players, the constant is 0x00, 0x0B, 0x40, 0x00, 0x00, 0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x0B, 0x00, 0x02, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x0B
which is {0x00, 0x0B, 0x40, 0x00, 0x00, 0x00, 0x80, 0x00, 0x00, 0x00} + playerInfoEndSeparator

Fatal error: Uncaught Exception: Could not find GAIA object separator in C:\xampp\htdocs\ageoflan\recanalyst\vendor\recanalyst\recanalyst\src\Analyzers\PlayerObjectsListAnalyzer.php:242 Stack trace: #0 C:\xampp\htdocs\ageoflan\recanalyst\vendor\recanalyst\recanalyst\src\Analyzers\Analyzer.php(40): RecAnalyst\Analyzers\PlayerObjectsListAnalyzer->run() #1 C:\xampp\htdocs\ageoflan\recanalyst\vendor\recanalyst\recanalyst\src\RecordedGame.php(125): RecAnalyst\Analyzers\Analyzer->analyze(Object(RecAnalyst\RecordedGame)) #2 C:\xampp\htdocs\ageoflan\recanalyst\vendor\recanalyst\recanalyst\src\RecordedGame.php(143): RecAnalyst\RecordedGame->runAnalyzer(Object(RecAnalyst\Analyzers\PlayerObjectsListAnalyzer)) #3 C:\xampp\htdocs\ageoflan\recanalyst\vendor\recanalyst\recanalyst\src\Analyzers\Analyzer.php(65): RecAnalyst\RecordedGame->getAnalysis('RecAnalyst\Anal...', Array, 1573307) #4 C:\xampp\htdocs\ageoflan\recanalyst\vendor\recanalyst\recanalyst\src\Analyzers\PlayerInfoBlockAnalyzer.php(189): RecAnalyst\Analyzers\Analyzer->read(' in C:\xampp\htdocs\ageoflan\recanalyst\vendor\recanalyst\recanalyst\src\Analyzers\PlayerObjectsListAnalyzer.php on line 242

Testfiles can be found here:
https://www.aoezone.net/threads/lan-regicide-between-noobs.150582/

Hello,
I just want to ask you about locating recorded game info such as: Victorious Player, number of kills, and Civilization names.

I would appreciate it if you could tell me the Hex values or blocks that represent those.

Thank you,

While we can't figure out exactly what happens in a game (see #1), it should be possible to auto-detect some common strategies based on when buildings or units are created/queued and which techs are researched.

e.g., some potential heuristics, assuming a Dark Age start:

• Drush: 3ish militia built before minute 12
• Fast Castle: Clicking Castle without doing Loom, within 2 minutes after Feudal is complete
• Douche: New Town Center built in Dark Age
• Scrush: Multiple stables + lots of scouts early in Feudal Age
• more?

Could be:

foreach ($rec->players() as $player) {
    // $player->guessStrategy() === 'drush';
}

Right now Analyzers return class instances, or sometimes they don't. They also read data from the recorded game file on demand, which can be nice, but which makes it more difficult to store data that's been read. For example, if you want to upload a rec, and then show the localized civilization names later, that's not really possible because the civName() method is tied to the Analyzer class.

It'd be neat if RecAnalyst was essentially two steps: reading data, and a modeling layer on top of that data. The readers would just return JSON. It could still read on demand, but you could also read everything, store the JSON somewhere, and create a new RecAnalyst instance based on that JSON blob to get the nice API without having to re-analyze the rec.

Probably going to try to align the JSON format with recage.

I'm trying to use recanalyst to find the players that resigned in an HD game (and whether or not they were the owner).

I wrote a little script to do this for a recorded game given from the command line. The script just looks for the first person to resign and checks if player->owner. It doesn't check the team of the player who resigned.

(A usage for this script is:
php examples/get_resigned.php some_recorded_game.aoe2record; echo $?;

require __DIR__ . '/../vendor/autoload.php';
use RecAnalyst\RecordedGame;

$filename = $argv[1];
$rec = new RecordedGame($filename);

// EXIT 0 means you did not resign
// EXIT 1 means you did resign
// EXIT 2 means could not find anyone who resigned

// Go through each of the chat messages, search for if the chat message is:
// "<player> resigned" and player is owner.
foreach ($rec->body()->chatMessages as $chat) {
    if ($chat->player && $chat->player->owner && $chat->msg === 'resigned') {
        exit(1);
    }
}
foreach ($rec->body()->chatMessages as $chat) {
    if ($chat->player && $chat->msg === 'resigned'){
        exit(0);
    }
}

// Go through each of the players and search for resignTime + player is owner
foreach($rec->players() as $player){
  if($player->resignTime && $player->owner){
    exit(1);
  }
  elseif ($player->resignTime){
    exit(0);
  }
}
exit(2);

I'm getting map id = 0 for every game.

Currently we just read a few, would be cool to read everything into an actions() array or something.

Provide an official public API for all the data that can currently be read by RecAnalyst. Most of the stuff is already accessible through methods on the RecordedGame object, but I think there are some things that are only accessible as object properties that perhaps should not be public (the player starting location for example is player->initialState->location, I'm not sure that I want that initialState property there).

It already supports many more of the recent formats (various HD editions mostly), so supporting even more isn't necessarily a priority for a new release, and support for more versions can be added as patch releases anyway.

I think these are very similar to normal recorded game files, but with a bit of metadata prepended.

Start of the restore game data:

Start of header:

The "This match was played on Voobly" messages are cut off incorrectly. This happens because the chat parser assumes that the username in the chat message will be the same as the username of the player that "sent" the chat message (i.e., the player number that is stored in the chat message). However, Voobly's messages are sent from player number 1(?), but with the username "Voobly".

I think these include Expansions Used information near the start of the file (before AI). Not sure about other differences.

So after uploading the folder to my host, i proceeded to run
"php ../../../composer.phar require recanalyst/recanalyst"
from my SSH via PuTTY

I then tried to upload a file into this php file

<?php

require __DIR__ . '/src/vendor/autoload.php';

use RecAnalyst\RecordedGame;

$array = array();
$a = __DIR__ . "/games/" . $_FILES["file"]["name"][0];
move_uploaded_file($_FILES["file"]["tmp_name"][0], $a);
$array[] = $_FILES["file"]["name"][0];
foreach ($array as $val) {
    $filename = __DIR__ . '/games/' . $val;
// Read a recorded game from a file path.
    $rec = new RecordedGame($filename);
// Render a map image. Map images are instances of the \Intervention\Image
// library, so you can easily manipulate them.
    $rec->mapImage()
        ->resize(240, 120)
        ->save('minimap.png');
// Display players and their civilizations.
    echo 'Players: ' . "\n";
    foreach ($rec->players() as $player) {
        echo ' * ' . $player->name . ' (' . $player->civName() . ')' . "\n";
    }
    echo 'Minimap saved in minimap.png.' . "\n";
}

Which is pretty much the basic example with a simple twist that it takes the uploaded file and puts it into the folder and then passes it to RecAnalyst.

After trying to upload a file, i came accross this error in the log files.

mod_fcgid: stderr: PHP Fatal error: Class 'RecAnalyst\RecordedGame' not found in ********/httpdocs/recanalyst/recanalyst.php on line 16

As i'm a complete newbie to Composer and stuff, i'm guessing there's something that went wrong when trying to load via autloader.php

The directory structure is something like this

Can i get some pointers in the right direction if you got the time?

Regards,
Ray.

First of all thanks for this awesome API!

I was wondering if it is possible to show fishes when generating minimap screenshots? That would make the mini-map more complete.

Hello,
I hope you are doing well!

I wanted to ask whether there is a .dll file containing all resources of #https://github.com/goto-bus-stop/recanalyst/blob/f1f9bf8d92cb9e998de3e9b5e73cde32bab03b1c/resources/lang/en/ageofempires.php ?

Thank you!

Hello i tried to analyze many games of v5.7 and v5.8 with extension aoe2record but i receive error 500. On the log file i see a warning can't read 4 bytes and it comes from header analyze section. Can someone help me? thanks!

There is a problem with encoding UTF-8 Chars.
https://www.bilder-upload.eu/bild-04c7ab-1549662488.png.html
For example in my pic there is a player named cremeh?tchen(normaly cremehütchen). For default it is in ISO-8859-1 but other ones are in UTF8.
$enc = mb_detect_encoding($player->name, "UTF-8,ISO-8859-1"); printf($enc); printf(iconv($enc, "UTF-8", $player->name));

with iconnv i can change it as you can already see in the pic.

Why is that happening?

$player->achievements()->tech->feudalTime

And

$player->feudalTime

both comes in a different way so Utils::formatGameTime cannot convert the first correct.

This is in an HD Edition recorded game of a campaign game, so it might also be the scenario/campaign stuff causing it instead of just the version.

The 15555 value is hexadecimal a3 5f 02 00 (little-endian), which is a separator value in the scenario header block.

I kind of guessed that the thing I tried in f42fdb6 was correct, but perhaps the double-skip there is not accurate for campaign games and/or HD Edition games.

aoe2record files have a very early header with a bunch of game information that's displayed when you select them in the Saved/Recorded Games list. Starts at byte 0x0c in the decompressed game header.

Current intel:

struct aoe2recordString {
  short length;
  short separator; // 0x60 0xA0
  char content[length];
}

typedef int longbool;

struct aoe2recordHeader {
  float oneThousand; // = 1.0e3 = 1000. This is 1004 in Patch 4.8 so it might be a version number
  int oneThousand; // = 1000
  longbool unknown;
  int datasets_count; // maybeeee?
  int unknows[datasets_count + 2];
  int mapId;
  int unknowns[6]; // IIRC this is 6 but it might be something else
  int separator; // 0xA3 0x5F 0x02 0x00
  aoe2recordString mapName;
  aoe2recordString unknown;
  int separator;
  float gameSpeed; // 1.00 / 1.50 / 2.00 instead of the 100 / 150 / 200 elsewhere in the file
  int popLimit;
  int numPlayers;
  int unknown; // always zero?
  int victoryAmount; // maybe! not sure what "Victory Amount" even means.
                     // could also be isMultiplayer
  int separator;
  bool unknown[0x0c]; // 1s and 0s, probably flags
  int separator;
  // Data for every player:
  int unknown[8];
  struct {
    int unknown;
    int civId; // seems to be civId + 1? or?
  } dummyStructName[8];
  byte color[8];
  byte team[8];
  byte unknown[8];
  int separator;
  int gameVersion[8];
  int dataVersion[8]; // these numbers are really huge and I don't know what they mean. It might be a hash?
  aoe2recordString playerNames[8];
  // and then a bunch of unknown stuff ...
}

There should be expansion pack info in here somewhere but I haven't found it yet.

Somewhere in here are:

• Scenario (might be the unknown name?)
• Ranked match (yes / no)
• Allow spectators (yes / no)
• Limited Civ Pool (what values?)
• Starting Age
• Ending Age
• Game type (Random map / campaign / etc)
• Map size
• Visibility
• Victory Condition
• "Random Map" (Unsure, probably/maybe for custom random map files)

biegleux's RecAnalyst is a straight port from his Pascal package by the same name. The API at this point is not particularly nice for PHP applications.

I'm planning on overhauling the API into something like:

$rec = new RecAnalyst\RecordedGame('rec.mgz');
$rec->analyze();
$players = $rec->getPlayers(); // will call ->analyze if needed etc
$arch = new RecAnalyst\Archive('recs.zip');
foreach ($arch as $rec) {
  $rec->getVictoryCondition(); // or $rec->victoryCondition… not sure
}

Still need to flesh out some large details. Will probably also involve a new name to avoid confusion with biegleux's package. (It's already incompatible, actually 😧 )

There's some more data in front of the chat messages block that should be skipped in HD Edition games.

The population limit is here:

In AoC files, the chat starts 3 bytes after. But in HD Edition, there's something like 12 bytes more.

v3.x and below did this so v4 should have it too.

Scenarios can have bitmaps embedded and they're also added in recorded games.