[Bug] Still losing information about stable-diffusion-webui-civitai-helper HOT 4 CLOSED

It's really slow when you have a 100s of gigabytes of data.

Probably an oversight on my part. I'll see about working that out.

I thought it might just update existing structure

This is not really in the works. Currently changing the structure of the metadata is done by just re-downloading the metadata and overwriting it. Part of the reason I haven't implemented appending new data to existing model info is because it would require comparing every field and making some assumptions (like whether the user edited it or did the model author change the metadata?), which is still easier than picking an entire branch, moving it to a different part of the model data, and hoping I haven't fucked with anything.

and add the last time it checked and version and maybe another property stating Civitai update failed. Something like so.

This is a work in progress. I've gotten some unused shim code in for this, I just need to finalize it and write in the function calls to use it.

I also now get this error, which I didn't before.

This was an oversight on my part and should be fixed now. I don't know why I just assumed all models would be safetensors.

from stable-diffusion-webui-civitai-helper.

I'm closing this issue since the main bug has been addressed. I will continue further work on updating the existing model metadata, but I do not want to give potential users reading the issue list the impression that they may lose data when that issue has been resolved. If you wish to track development on updating the metadata in-place, please open a new issue as a feature request and I will keep you updated. :)

from stable-diffusion-webui-civitai-helper.

I tried to do some refactoring. Let me know if this is still happening.

from stable-diffusion-webui-civitai-helper.

Thank you for the update. I just managed to try it now. And it seems to hash all the files even when the info file already contains the checksum. It's really slow when you have a 100s of gigabytes of data.

I had to remove all the files and just put the wanted test files, just to check if it works. It does not edit or add any detail. Just skips the metadata.

Civitai Helper: Civitai does not have this model
Civitai Helper: 'New metadata id () does not match old metadata id (129610)', aborting

I thought it might just update existing structure and add the last time it checked and version and maybe another property stating Civitai update failed. Something like so.

    "extensions": {
        "sd_civitai_helper": {
            "version": "1.7.5",
            "last_update": 1697223432,
            "update": failed
        }
    }

If the metadata exist and its still on Civitai I get this:

Civitai Helper: Model metadata not needed for Barbie_❴4B0FE67F4F❵.safetensors
Civitai Helper: Downloading model image.

Which perfect but it does not update the extensions property. It's still at 1.7.2, not updated to 1.7.5 version.

    "extensions": {
        "sd_civitai_helper": {
            "version": "1.7.2"
        }
    }

But at least it does not delete the metadata. So that works. But hashing all the files again when you have the hash and file size stored in the info seems reductive.

I also now get this error, which I didn't before.

Civitai Helper: Civitai does not have this model
Traceback (most recent call last):
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "T:\SD\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\scripts\ch_lib\model_action_civitai.py", line 125, in scan_model
    success = scan_single_model(filename, root, model_type, refetch_old, delay)
  File "T:\SD\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\scripts\ch_lib\model_action_civitai.py", line 78, in scan_single_model
    model_info = dummy_model_info(item, sha256_hash, model_type)
  File "T:\SD\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\scripts\ch_lib\model_action_civitai.py", line 177, in dummy_model_info
    file_metadata = sd_models.read_metadata_from_safetensors(path)
  File "T:\SD\stable-diffusion-webui\modules\sd_models.py", line 269, in read_metadata_from_safetensors
    assert metadata_len > 2 and json_start in (b'{"', b"{'"), f"{filename} is not a safetensors file"
AssertionError: T:\SD\stable-diffusion-webui\models\Stable-diffusion\NovelAI.ckpt is not a safetensors file

It's a ckpt file. I have no clue why this occurs.

from stable-diffusion-webui-civitai-helper.