Slow start about libloot HOT 7 CLOSED

Significant times:

[06:53:13.113143] [info]: LOOT Version: 0.13.1+fc664279
[06:53:13.263640] [info]: Attempting to open link: http://loot/ui/index.html
[06:53:15.434012] [info]: Getting LOOT's supported languages.
[06:53:16.147175] [trace]: Sending progress update: Parsing, merging and evaluating metadata...
[06:53:16.859902] [info]: Loading 360 plugins using 6 threads, with up to 60 plugins per thread.
[06:53:16.866274] [trace]: Starting plugin loading.
[06:53:50.456895] [debug]: Parsing metadata list(s).
[06:53:50.949079] [trace]: Getting masterlist metadata for: Skyrim.esm
[06:53:56.937978] [trace]: Getting masterlist metadata for: ccbgssse010-petdwarvenarmoredmudcrab.esl
[06:53:59.825111] [info]: Getting LOOT's version.

So that's:

• 3 seconds just to load the initial UI
  • Hopefully the polymer 3 UI update will improve that.
• 34 seconds to parse plugin headers and process them
  • Most time seems to be spent checking if the plugin loads an archive, as each plugin will iterate over the Data directory contents looking for BSAs (~ 0.5s per plugin). A hashset of BSAs could be populated before loading plugins and reused instead.
• 6 seconds getting masterlist metadata that was mostly spent calculating DLC plugin CRCs
  • It looks like CRC caching isn't working here.
• 3 seconds getting masterlist metadata for the other ~355 plugins
  • It looks like CRC caching isn't working here.
  • The whole load order is fetched for each plugin, which also adds up (~ 1.5 ms per plugin) and is probably easy to avoid.

from libloot.

Just a further bit of information, I'm running LOOT from inside MO2.

from libloot.

@eFrysTon Try replacing your loot_api.dll with this one, and upload the debug log that results in.

from libloot.

I'll add a few Debug logs for comparison.

Only DLC outside MO2.

LOOTDebugLog.txt

Start up with no metadata, only the settings.toml in appdata\local\loot folder.
Then updated metadata, sorted & restarted.

Only DLC running from MO2.

LOOTDebugLog.txt

Start up with no metadata, only the settings.toml in appdata\local\loot folder.
Then updated metadata, sorted & restarted.

from libloot.

LOOTDebugLog - slowish start.zip

There are some slight differences because my load order has changed slightly. But I don't think these changes should have a huge impact on the time.

Before:
[06:53:13.113143]
[06:53:59.825111]
~46 sec

After:
[18:08:15.338328]
[18:08:25.455277]
~10 sec

from libloot.

Some more detail from the new log:

[18:08:15.338328] [info]: LOOT Version: 0.13.1+fc664279
[18:08:15.558051] [info]: Attempting to open link: http://loot/ui/index.html
[18:08:17.913766] [info]: Getting LOOT's supported languages.
[18:08:17.914230] [info]: Getting LOOT's settings.
[18:08:18.678810] [trace]: Sending progress update: Parsing, merging and evaluating metadata...
[18:08:20.950628] [info]: Loading 374 plugins using 6 threads, with up to 63 plugins per thread.
[18:08:21.112473] [trace]: Starting plugin loading.
[18:08:21.207521] [debug]: Parsing metadata list(s).
[18:08:21.666102] [trace]: Getting masterlist metadata for: Skyrim.esm
[18:08:22.294237] [trace]: Getting masterlist metadata for: ccbgssse010-petdwarvenarmoredmudcrab.esl
[18:08:25.455277] [info]: Getting LOOT's version.

Observations:

• Loading settings is weirdly slow, something to look into in the LOOT application.
• Finding valid plugins to load is a bit slow (~ 2s), multithreading that like how plugin loading is might help.
• Plugin loading went from 34 seconds (for 360 plugins) to < 0.09 seconds, so that's good. 👍
• Getting masterlist metadata for the official plugins went from 6 seconds to 0.6 seconds. 👍
• Getting masterlist metadata for other plugins still takes 3 seconds, mostly due to CRC calculations. There's nothing particularly inefficient going on, getting metadata in parallel might help but that's something for the LOOT application to do.

So I might be able to shave off another second or so in the LOOT API without too much effort, otherwise the remaining areas of improvement are in the LOOT application.

from libloot.

Finding valid plugins is actually done in the LOOT application. As all the low hanging fruit are now outside the LOOT API, I'll close this issue and open another in the loot/loot tracker.

from libloot.