fizzledorf / ait Goto Github PK

View Code? Open in Web Editor NEW

49.0 49.0 12.0 284 KB

This was orginally written by: https://github.com/hlky

License: Apache License 2.0

Python 100.00%

ait's People

Contributors

Stargazers

Watchers

Forkers

maxlanc asagi4 gameltb tashaskyup kuron88 mafei198 m1kep roop03988 dushwe deinferno

ait's Issues

Error Running Workflows

Of date none of the workflows runs through?
Prompt executed in 8.11 seconds
got prompt
INFO:comfyui-prompt-control:Resolving wildcards...
Prompt executor has been patched by Job Iterator!
3
Found 5 modules for windows v1 sm80 1 512 unet
Using 8763aa0cc5f5008bd0734664ae47e874c2641d9e8293e8bd84c8c8ca7f6084d0
!!! Exception during processing !!!
Traceback (most recent call last):
File "A:\AI_Files\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "A:\AI_Files\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "A:\AI_Files\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "A:\AI_Files\ComfyUI\nodes.py", line 1236, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "A:\AI_Files\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 176, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "A:\AI_Files\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\hacky.py", line 9, in informative_sample
return original_sample(*args, **kwargs)
File "A:\AI_Files\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 300, in sample
sampler.model_denoise = comfy.samplers.CFGNoisePredictor(model_wrapper)
File "A:\AI_Files\ComfyUI\custom_nodes\ComfyUI_smZNodes\smZNodes.py", line 814, in init
self.inner_model2.num_timesteps = model.num_timesteps
File "A:\AI_Files\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1614, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'AITemplateModelWrapper' object has no attribute 'num_timesteps'

Issues with Inpainting pipeline

Hi! Thanks for the great node. I could use it in the example and it works very well: I had to change:

self.control_model_wrapped = comfy.sd.ModelPatcher(self.control_model, load_device=comfy.model_management.get_torch_device(), offload_device=comfy.model_management.unet_offload_device())

for

self.control_model_wrapped = comfy.model_patcher.ModelPatcher(self.control_model, load_device=comfy.model_management.get_torch_device(), offload_device=comfy.model_management.unet_offload_device())

However, I am trying to use it in a workflow that I have that has Inpainting and ControlNet. I have issues with the inpainting model and with controlnet.

When I use the node with a non-inpainting model and with Canny ControlNet got this error:

Error occurred when executing KSampler:

can't multiply sequence by non-int of type 'float'

File "/home/ubuntu/ComfyUI/execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/home/ubuntu/ComfyUI/execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/home/ubuntu/ComfyUI/execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/home/ubuntu/ComfyUI/nodes.py", line 1265, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 176, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 310, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 742, in sample
samples = getattr(k_diffusion_sampling, "sample_{}".format(self.sampler))(self.model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar)
File "/home/ubuntu/ComfyUI/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/k_diffusion/sampling.py", line 707, in sample_dpmpp_sde_gpu
return sample_dpmpp_sde(model, x, sigmas, extra_args=extra_args, callback=callback, disable=disable, eta=eta, s_noise=s_noise, noise_sampler=noise_sampler, r=r)
File "/home/ubuntu/ComfyUI/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/k_diffusion/sampling.py", line 539, in sample_dpmpp_sde
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/home/ubuntu/ComfyUI/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 323, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
File "/home/ubuntu/ComfyUI/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/k_diffusion/external.py", line 125, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "/home/ubuntu/ComfyUI/comfy/k_diffusion/external.py", line 151, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 311, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 289, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 242, in calc_cond_uncond_batch
c['control'] = control.get_control(input_x, timestep_, c, len(cond_or_uncond))
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 435, in get_control
x *= self.strength

When I changed the workflow and use an inpainting model and the same Canny ControlNet I get:

Error occurred when executing KSampler:

Error in function: AITemplateModelContainerSetManyConstants

File "/home/ubuntu/ComfyUI/execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/home/ubuntu/ComfyUI/execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/home/ubuntu/ComfyUI/execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/home/ubuntu/ComfyUI/nodes.py", line 1265, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 176, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 274, in sample
AITemplate.unet[module['sha256']] = AITemplate.loader.apply_unet(
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/ait/load.py", line 222, in apply_unet
return self.apply(aitemplate_module, ait_params)
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/ait/load.py", line 201, in apply
aitemplate_module.set_many_constants_with_tensors(ait_params)
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/ait/module/model.py", line 840, in set_many_constants_with_tensors
self.set_many_constants(ait_tensors)
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/ait/module/model.py", line 791, in set_many_constants
self.DLL.AITemplateModelContainerSetManyConstants(
File "/home/ubuntu/ComfyUI/custom_nodes/AIT/AITemplate/ait/module/model.py", line 212, in _wrapped_func
raise RuntimeError(f"Error in function: {method.__name__}")

It works fine if I run it without ControlNet and with a non-inpainting model.

Thanks!

Latest ComfyUI updates break plugin on Windows.

I'm not sure what is causing it but the ksampler is throwing a 'tuple' object has no attribute 'model' error. Maybe caused by changes to the sampler here?:
comfyanonymous/ComfyUI@4eab00e

[Bug] Conditioning Combine node error

Got this error when using the core Conditioning Combine node:
Error: [SetValue] Dimension got value out of bounds; expected value to be in [1, 2], but got 3.

Unable to compile modules after update

After updating build commands that previously worked are failing now.

python unet.py --hf-hub-or-path "runwayml/stable-diffusion-v1-5" --width 512 512 --height 512 512 --batch-size 2 2 --clip-chunks 1 --model-name "v1_unet_512_512_2_2_c1_test" --work-dir "/home/primary/Documents/code/ML/AIT/modules"

2023-09-09 14:08:36,292 INFO <aitemplate.compiler.compiler> compiled the final .so file elapsed time: 0:03:22.475189
[14:08:36] model_container.cu:69: Device Runtime Version: 11080; Driver Version: 12010
[14:08:36] model_container.cu:83: Hardware accelerator device properties:
  Device:
     ASCII string identifying device: NVIDIA GeForce RTX 4090
     Major compute capability: 8
     Minor compute capability: 9
     UUID: GPU-3caeb811-53d4-7669-01dc-ca502ea01048
     Unique identifier for a group of devices on the same multi-GPU board: 0
     PCI bus ID of the device: 12
     PCI device ID of the device: 0
     PCI domain ID of the device: 0
  Memory limits:
     Constant memory available on device in bytes: 65536
     Global memory available on device in bytes: 25385107456
     Size of L2 cache in bytes: 75497472
     Shared memory available per block in bytes: 49152
     Shared memory available per multiprocessor in bytes: 102400
[14:08:36] model_container.cu:87: Init AITemplate Runtime with 1 concurrency
Traceback (most recent call last):
  File "/home/primary/Documents/code/ML/AIT/AITemplate/unet.py", line 141, in <module>
    compile_diffusers()
  File "/home/primary/mambaforge/envs/AITemplate/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/primary/mambaforge/envs/AITemplate/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/primary/mambaforge/envs/AITemplate/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/primary/mambaforge/envs/AITemplate/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/primary/Documents/code/ML/AIT/AITemplate/unet.py", line 104, in compile_diffusers
    compile_unet(
  File "/home/primary/Documents/code/ML/AIT/AITemplate/ait/compile/unet.py", line 305, in compile_unet
    vram = round(total_usage / 1024 / 1024)
                 ~~~~~~~~~~~~^~~~~~
TypeError: unsupported operand type(s) for /: 'Model' and 'int'

ControlNets unload every step (very slow performance)

The extension seems to do something that breaks controlnet usage even when no AITemplate loaders are in use.

The attached image contains a workflow (modified straight from a ComfyUI controlnet example, so it should be pretty minimal) that works when AIT is not in the custom_nodes folder, and breaks when it is.

The error it fails with is:

Error occurred when executing KSampler:

Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

  File "/home/sd/git/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/home/sd/git/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/home/sd/git/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/home/sd/git/ComfyUI/nodes.py", line 1207, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
  File "/home/sd/git/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 175, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
  File "/home/sd/git/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 308, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/home/sd/git/ComfyUI/comfy/samplers.py", line 727, in sample
    samples = getattr(k_diffusion_sampling, "sample_{}".format(self.sampler))(self.model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar)
  File "/home/sd/git/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/sd/git/ComfyUI/comfy/k_diffusion/sampling.py", line 539, in sample_dpmpp_sde
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/home/sd/git/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sd/git/ComfyUI/comfy/samplers.py", line 317, in forward
    out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
  File "/home/sd/git/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sd/git/ComfyUI/comfy/k_diffusion/external.py", line 125, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "/home/sd/git/ComfyUI/comfy/k_diffusion/external.py", line 151, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "/home/sd/git/ComfyUI/comfy/samplers.py", line 305, in apply_model
    out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
  File "/home/sd/git/ComfyUI/comfy/samplers.py", line 283, in sampling_function
    cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
  File "/home/sd/git/ComfyUI/comfy/samplers.py", line 235, in calc_cond_uncond_batch
    c['control'] = control.get_control(input_x, timestep_, c, len(cond_or_uncond))
  File "/home/sd/git/ComfyUI/comfy/controlnet.py", line 161, in get_control
    control = self.control_model(x=x_noisy.to(self.control_model.dtype), hint=self.cond_hint, timesteps=t, context=context.to(self.control_model.dtype), y=y)
  File "/home/sd/git/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sd/git/ComfyUI/comfy/cldm/cldm.py", line 283, in forward
    emb = self.time_embed(t_emb)
  File "/home/sd/git/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sd/git/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward
    input = module(input)
  File "/home/sd/git/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sd/git/ComfyUI/comfy/ops.py", line 18, in forward
    return torch.nn.functional.linear(input, self.weight, self.bias)
Queue size: 0
Extra options

AITemplate must move something to the wrong device at some point, but I haven't been able to figure out where it happens.

MultiAreaConditioning Compatibility

I have been using AIT for a while now and while it is REALLY FAST there are some kinks I've found out.
I'm using the MultiAreaConditioning node from this repo: https://github.com/Davemane42/ComfyUI_Dave_CustomNode

And when using the LoadAITemplate node connected to the normal KSampler it pukes out the following error.
It does with any combination in the inputs of MultiAreaCond, besides the default of two conditions only

And this is the related workspace to the node, all other nodes to the right of the pic are the usual hi res fix and face detail combo

Is there any chance of adding compatibility or a patch for it?

support for fooocus

since fooocus is using comfy as inference engine, it is possible to integrate ait to fooocus. are there any plans for supporting ait inference in fooocus?

Compilation fails after fb84060

After changes in fb84060 compilation fails, works fine before.

python unet.py --hf-hub-or-path "runwayml/stable-diffusion-v1-5" --width 512 512 --height 512 512 --batch-size 1 1 --clip-chunks 10 --model-name "v15_unet_512_512" --work-dir "/app/tmp/"

Traceback (most recent call last):
  File "AITemplate/unet.py", line 141, in <module>
    compile_diffusers()
  File "venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "AITemplate/unet.py", line 104, in compile_diffusers
    compile_unet(
  File "AITemplate/ait/compile/unet.py", line 271, in compile_unet
    Y = ait_mod(
        ^^^^^^^^
  File "venv/lib/python3.11/site-packages/aitemplate/frontend/nn/module.py", line 389, in _call_impl
    return forward_call(*input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "AITemplate/ait/modeling/unet_2d_condition.py", line 394, in forward
    sample = upsample_block(
             ^^^^^^^^^^^^^^^
  File "venv/lib/python3.11/site-packages/aitemplate/frontend/nn/module.py", line 389, in _call_impl
    return forward_call(*input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "AITemplate/ait/modeling/unet_blocks.py", line 671, in forward
    hidden_states = upsampler(hidden_states, upsample_size)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.11/site-packages/aitemplate/frontend/nn/module.py", line 389, in _call_impl
    return forward_call(*input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "AITemplate/ait/modeling/resnet.py", line 71, in forward
    x = nn.Upsampling2d(scale_factor=2.0, mode="nearest")(x, out)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.11/site-packages/aitemplate/frontend/nn/module.py", line 389, in _call_impl
    return forward_call(*input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.11/site-packages/aitemplate/frontend/nn/upsample.py", line 47, in forward
    assert len(args) == 1
           ^^^^^^^^^^^^^^
AssertionError

Compiling modules on Windows?

Is there any information on how to reproduce the windows-native modules you have provided at https://huggingface.co/Fizzledorf/AITemplate_V1_V2/tree/main/modules/windows ? From what I've seen, the AITemplate codebase seems very linux-specific, and I can't find any information anywhere on cross-compiling from Linux to Windows either (not even with WSL). How were those modules made?

Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Hello

When I use AIT and ComfyUi (both on the latest branch), it gives me this error.
I start ComfyUi with this command: python3 main.py --listen --port 21128 --gpu-only and I'm on Ubuntu 22.04.

Please help! :)

Error occurred when executing KSampler:

Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

File "/home/cybertimon/Repositories/ComfyUI/execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/home/cybertimon/Repositories/ComfyUI/execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/home/cybertimon/Repositories/ComfyUI/execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/home/cybertimon/Repositories/ComfyUI/nodes.py", line 1236, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/home/cybertimon/Repositories/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 176, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/home/cybertimon/Repositories/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 310, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/home/cybertimon/Repositories/ComfyUI/comfy/samplers.py", line 785, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler(), sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/home/cybertimon/Repositories/ComfyUI/comfy/samplers.py", line 690, in sample
samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/home/cybertimon/Repositories/ComfyUI/comfy/samplers.py", line 630, in sample
samples = getattr(k_diffusion_sampling, "sample_{}".format(sampler_name))(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **extra_options)
File "/home/cybertimon/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/cybertimon/Repositories/ComfyUI/comfy/k_diffusion/sampling.py", line 137, in sample_euler
denoised = model(x, sigma_hat * s_in, **extra_args)
File "/home/cybertimon/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/cybertimon/Repositories/ComfyUI/comfy/samplers.py", line 323, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
File "/home/cybertimon/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/cybertimon/Repositories/ComfyUI/comfy/k_diffusion/external.py", line 125, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "/home/cybertimon/Repositories/ComfyUI/comfy/k_diffusion/external.py", line 151, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "/home/cybertimon/Repositories/ComfyUI/comfy/samplers.py", line 311, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
File "/home/cybertimon/Repositories/ComfyUI/comfy/samplers.py", line 289, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
File "/home/cybertimon/Repositories/ComfyUI/comfy/samplers.py", line 265, in calc_cond_uncond_batch
output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
File "/home/cybertimon/Repositories/ComfyUI/comfy/model_base.py", line 63, in apply_model
return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options).float()
File "/home/cybertimon/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/cybertimon/Repositories/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 622, in forward
emb = emb + self.label_emb(y)
File "/home/cybertimon/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/cybertimon/.local/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/home/cybertimon/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/cybertimon/.local/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/home/cybertimon/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/cybertimon/Repositories/ComfyUI/comfy/ops.py", line 18, in forward
return torch.nn.functional.linear(input, self.weight, self.bias)

""log_vml_cpu" not implemented for 'Half'" when using AITemplate on certain models

error :

!!! Exception during processing !!!
Traceback (most recent call last):
  File "/home/gnuwu/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/home/gnuwu/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/home/gnuwu/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/home/gnuwu/ComfyUI/nodes.py", line 1211, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
  File "/home/gnuwu/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 176, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
  File "/home/gnuwu/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 294, in sample
    sampler = comfy.samplers.KSampler(real_model, steps=steps, device=device, sampler=sampler_name, scheduler=scheduler, denoise=denoise, model_options=model.model_options)
  File "/home/gnuwu/ComfyUI/comfy/samplers.py", line 548, in __init__
    self.model_wrap = CompVisVDenoiser(self.model_denoise, quantize=True)
  File "/home/gnuwu/ComfyUI/comfy/samplers.py", line 299, in __init__
    super().__init__(model, model.alphas_cumprod, quantize=quantize)
  File "/home/gnuwu/ComfyUI/comfy/k_diffusion/external.py", line 158, in __init__
    super().__init__(((1 - alphas_cumprod) / alphas_cumprod) ** 0.5, quantize)
  File "/home/gnuwu/ComfyUI/comfy/k_diffusion/external.py", line 48, in __init__
    self.register_buffer('log_sigmas', sigmas.log())
RuntimeError: "log_vml_cpu" not implemented for 'Half'

Vanilla "Apply ControlNet (Advanced)" breaks after loading extension

"Apply ControlNet (Advanced)" node is broken when this extension is loaded - without using any custom nodes. "Apply ControlNet" node seems to work fine

Error below:

!!! Exception during processing !!!
Traceback (most recent call last):
  File "C:\Users\josef\projects\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "C:\Users\josef\projects\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "C:\Users\josef\projects\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "C:\Users\josef\projects\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\nodes.py", line 657, in apply_controlnet
    c_net = control_net.copy().set_cond_hint(control_hint, strength, (1.0 - start_percent, 1.0 - end_percent))
TypeError: ControlNet.set_cond_hint() takes from 2 to 3 positional arguments but 4 were given

Minimal workflow example for reproducing
advanced_minimal_example.zip

Works fine without AIT in custom_nodes

[Bug] Batch size > 1

Large positive prompts trigger 'Error in function: AITemplateModelContainerRun'

This is great, gives about a 40% speed boost for me when it works, but oddly going above above some upper limit of characters in the positive prompt throws a 'Error in function: AITemplateModelContainerRun' when executing. Syntax nor number of tokens doesn't seem to affect it. Nor does the same happen for negative prompt. And the limit is different depending on the nodes that are loaded, model used, etc but I don't notice a pattern there necessarily. With the default workflow with AITemplateLoader and AITemplateVAEDecode subbed/chained in, basev1.5pruned model, the character limit appears to be ~590 for instance.

Traceback attached. Any clue what's going on?
stack.txt

Higher Batchsize VAE Decode/Encode?

Sorry this isn't technically related to this project but to your hugginface AITemplate models.

I was just wondering, if it is possible to have AIT models for >1 batchsize for VAE Decode/Encode?

Thanks for your hard work!

Trying to use SDXL AiTemplate on linux but ending up with this error

Hello! I try to use this AiTemplate plugin on linux, but I receive this error when using it on linux ubuntu 22.04:

Found 3 modules for linux xl sm80 1 1024 unet
Using b5caabe98aeb69bada9d1566c897aed66a84d4fb21f31482160d7ef9987f04fd
!!! Exception during processing !!!
Traceback (most recent call last):
  File "/home/cybertimon/Repositories/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/home/cybertimon/Repositories/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/home/cybertimon/Repositories/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/home/cybertimon/Repositories/ComfyUI/nodes.py", line 1206, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
  File "/home/cybertimon/Repositories/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 171, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
  File "/home/cybertimon/Repositories/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 259, in sample
    AITemplate.unet[module['sha256']] = AITemplate.loader.load_module(module['sha256'], module['url'])
  File "/home/cybertimon/Repositories/ComfyUI/custom_nodes/AIT/AITemplate/ait/load.py", line 69, in load_module
    self.download_module(sha256, url)
  File "/home/cybertimon/Repositories/ComfyUI/custom_nodes/AIT/AITemplate/ait/load.py", line 56, in download_module
    g.write(f.read())
  File "/usr/lib/python3.10/lzma.py", line 200, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.10/_compression.py", line 118, in readall
    while data := self.read(sys.maxsize):
  File "/usr/lib/python3.10/_compression.py", line 103, in read
    data = self._decompressor.decompress(rawblock, size)
_lzma.LZMAError: Input format not supported by decoder

I'm on the sdxl branch.
Thanks for help!

modules.json management

import argparse
import json
import os
import hashlib
import lzma
import tempfile
from tqdm import tqdm

"""
#{
"cuda":"sm80",
"batch_size":1,
"resolution":64,
"sd":"v1",
"model":"vae",
"os":"windows",
"url":"",
"compressed_sha256":"",
"compressed_file_size":0,
"sha256":"",
"file_size":0
}
Directory structure
{os}/
    {cuda}/
        bs{batch_size}/
            {resolution}/
                `{sd}_{module_type}_{min}_{resolution}.so.xz`
"""

def sha256sum(filename):
    h = hashlib.sha256()
    b = bytearray(128*1024)
    mv = memoryview(b)
    with open(filename, 'rb', buffering=0) as f:
        for n in iter(lambda : f.readinto(mv), 0):
            h.update(mv[:n])
    return h.hexdigest()

# extract .xz to temp file, sha256sum, and return, also return file size
def sha256sum_xz(filename):
    with tempfile.TemporaryDirectory() as tmpdirname:
        with lzma.open(filename) as f:
            with open(os.path.join(tmpdirname, "module.so"), "wb") as f2:
                f2.write(f.read())
        return sha256sum(os.path.join(tmpdirname, "module.so")), os.path.getsize(os.path.join(tmpdirname, "module.so"))

def modules_json_to_sha256sums(modules_json):
    with open(modules_json, "r") as f:
        modules = json.load(f)
    sha256sums = []
    pbar = tqdm(total=0)
    for module in modules:
        sha256sums.append(module["compressed_sha256"])
        pbar.update(1)
    pbar.close()
    return sha256sums

def folder_to_json(modules_folder, modules_json_out, url_prefix, modules_json_in=None):
    existing_modules = []
    if modules_json_in is not None:
        existing_modules = modules_json_to_sha256sums(modules_json_in)
    pbar = tqdm(total=0)
    modules = []
    for os_name in os.listdir(modules_folder):
        os_path = os.path.join(modules_folder, os_name)
        if not os.path.isdir(os_path):
            continue
        for cuda in os.listdir(os_path):
            cuda_path = os.path.join(os_path, cuda)
            if not os.path.isdir(cuda_path):
                continue
            for batch_size in os.listdir(cuda_path):
                batch_size_path = os.path.join(cuda_path, batch_size)
                if not os.path.isdir(batch_size_path):
                    continue
                for resolution in os.listdir(batch_size_path):
                    resolution_path = os.path.join(batch_size_path, resolution)
                    if not os.path.isdir(resolution_path):
                        continue
                    for module in os.listdir(resolution_path):
                        if "clip" in module:
                            continue
                        module_path = os.path.join(resolution_path, module)
                        if not os.path.isfile(module_path):
                            continue
                        compressed_sha256 = sha256sum(module_path)
                        if compressed_sha256 in existing_modules:
                            print(f"Module: {module_path} already exists in {modules_json_in}, skipping")
                            pbar.update(1)
                            continue
                        module_info = module.split("_")
                        print(module_info)
                        sd = module_info[0]
                        module_type = module_info[1]
                        if module_type == "vae" and sd != "v1":
                            print(f"Module: {module_path} can be removed, v1 VAE modules support all sd values")
                        if module_type == "vae" and module_info[2] == "encode":
                            module_type = "vae_encode"
                            min_resolution = module_info[3]
                            resolution = module_info[4].split(".")[0]
                        else:
                            min_resolution = module_info[2]
                            resolution = module_info[3].split(".")[0]
                        decompressed_sha256, decompressed_file_size = sha256sum_xz(module_path)
                        module_json = {
                            "cuda": cuda,
                            "batch_size": int(batch_size[2:]),
                            "resolution": int(resolution),
                            "sd": sd,
                            "model": module_type,
                            "os": os_name,
                            "url": f"{url_prefix}{os_name}/{cuda}/{batch_size}/{resolution}/{module}",
                            "compressed_sha256": compressed_sha256,
                            "compressed_file_size": os.path.getsize(module_path),
                            "sha256": decompressed_sha256,
                            "file_size": decompressed_file_size
                        }
                        modules.append(module_json)
                        pbar.update(1)
    pbar.close()
    with open(modules_json_out, "w") as f:
        json.dump(modules, f)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--modules-folder",
    )
    parser.add_argument(
        "--modules-json-out"
    )
    parser.add_argument(
        "--modules-json-in",
        default=None
    )
    parser.add_argument(
        "--url-prefix",
        help="https://huggingface.co/{hf-path}/resolve/main/"
    )
    args = parser.parse_args()

    folder_to_json(args.modules_folder, args.modules_json_out, args.url_prefix, args.modules_json_in)

This script can assist with managing modules.json until a more permanent solution is developed.

Usage:

It works with the same folder structure as huggingface repo
With only --modules-json-out specified it will process every file in the folder, hashing both the decompressed and compressed versions, and writing out the json with details of all modules.
If --modules-json-in is specified, only the new modules are written to --modules-json-out, check is based on compressed_sha256 value
--url-prefix is the base of the huggingface download url, e.g. https://huggingface.co/Fizzledorf/AITemplate_V1_V2/resolve/main/modules/

Therefore the procedure for adding new modules would be to compress them, I use 7-zip on Windows and xz on Linux,
move to the appropriate folder, then run the script and provide the existing modules.json, and finally update existing modules.json with the new data

cc @FizzleDorf

After the last update, an error occurred.

File ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 414
else:
^^^^
IndentationError: expected an indented block after 'if' statement on line 413

Module input name update - Affects recent SDXL modules

In my local version and the SDXL support PR to facebookincubator/AITemplate I renamed inputs from inputN.

All modules will be recompiled soon, this will include the new naming scheme, and enable 8px increment. Suggestions on resolutions, batch sizes etc that should be supported are welcome, and could be shared here or in a separate issue.

Name changes are described below and need to be applied in AITemplate/ait/inference.py.

To use the recent SDXL modules, I would recommend applying only the UNet changes and using the existing VAE modules, as the changes are less involved. Note that the changes will then be incompatible with existing UNet modules for v1 and v2.

Depending on how long compiling a new set takes, and how long adoption of new modules would take, it may be advantageous to implement some sort of compatibility check, the module has capability of returning the input and output name to index maps, I'll leave this open to comment, SM70/SM75 may take longer to replace although I'm not sure of their popularity.

UNet

input0 -> latent_model_input
input1 -> timesteps
input2 -> encoder_hidden_states
input3 -> class_labels (unused by this plugin)

VAE

vae_input -> pixels when encode, latent when decode
vae_sample -> random_sample, only used by encode

Note: module selection for VAE uses v1, meaning the recent VAE modules labelled with sd: xl will not be used, the main difference between the recent and existing is logging, they will still work.

CLIP

Unused by this plugin, but it's input0 -> input_ids and input1 to position_ids

New modules

Attached are fresh modules with new naming and 8px increment, batch 1, Windows, sm80
Note plugin does not support selection of refiner modules at this time.
VAE Decode and VAE Encode aren't versioned because it is compatible between v1, v2 and xl
The filename format for the is also different, {type}{sd}{max_resolution}, and the extension is .dll, minimum resolution is always 8px
There should be some group input on the filename format, and which resolutions and batch sizes will be supported, whether to get rid of the folders and put cuda version/os etc in the filename

clip.zip
unet_control_v1.zip
unet_v1.zip
unet_v2.zip
unet_xl.zip
unet_xlr.zip
vae_decode.zip
vae_encode.zip

These modules are also being hosted for my upcoming UI that uses AIT, over at modules.diffuse.rs, that file windows_sm80.json contains info for Windows SM80 modules, there will be others following the same format when available (linux_sm80.json, windows_sm75.json etc), the modules themselves are hosted on the same domain i.e. sm80_unet_xlr_2048.dll.xz, linux modules will have .so extension.
This plugin can be changed to use this source, or modules can be rehosted from that source if desired.

`Error: Got cutlass error: Error Internal`

If you get Error: Got cutlass error: Error Internal the module needs to be recompiled, report it here.

The following modules return this error on my machine:

If these modules are working on your machine, report it here.

BS?

when will there be modules for custom batch sizes? I compiled my own, but I can't get the node to see them.

FreeU support?

Can AITemplate support the recent FreeU node added to ComfyUI, or at least its functionality? It uses a transformer patch which gets ignored by AIT inference. This is unfortunate since the FreeU algorithm can have very interesting effects on generation quality.

I tried poking around the code but I couldn't find a place to plug it in. It might be that it needs a specifically compiled FreeU module, but that goes over my head currently.

Is there a way for AIT to support FreeU? I'm not opposed to writing a patch myself, but I need a pointer in the right direction.

[Bug] ddim, uni_pc, uni_pc_bh2

enable or disable, this error occurs when using ddim, uni_pc, or uni_pc_bh2:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

add google colab support

[Bug] LoRAs handled incorrectly

Reproduction:

Alternate between bypassing and enabling the AITLoader node while using a LoRA.
Gens eventually become completely different

AMD support

I played around with AMD's AITemplate fork on my 7900XTX a while ago, and it seems to be the fastest Stable Diffusion implementation on AMD hardware right now. That said, RDNA3 support hasn't been upstreamed yet, but it is available here: navi3_rel_ver_1.0

[BUG]: Sampler failing on latest commit

I'm not sure if this is caused by the smart memory management changes Comfy introduced or a strange interaction with previews (turning previews off doesn't fix it) but can't run any workflows.

Error occurred when executing KSamplerAdvanced:

module 'comfy.model_management' has no attribute 'load_if_low_vram'

File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1240, in sample
return common_ksampler(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 175, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\hacky.py", line 9, in informative_sample
return original_sample(*args, **kwargs)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 308, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 733, in sample
samples = getattr(k_diffusion_sampling, "sample_{}".format(self.sampler))(self.model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 701, in sample_dpmpp_2m_sde_gpu
return sample_dpmpp_2m_sde(model, x, sigmas, extra_args=extra_args, callback=callback, disable=disable, eta=eta, s_noise=s_noise, noise_sampler=noise_sampler, solver_type=solver_type)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 613, in sample_dpmpp_2m_sde
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 323, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 125, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 151, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 311, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 289, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 241, in calc_cond_uncond_batch
c['control'] = control.get_control(input_x, timestep_, c, len(cond_or_uncond))
File "C:\Users\hoffl\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 409, in get_control
self.control_model = comfy.model_management.load_if_low_vram(self.control_model)

ADD FEATURE [REQUEST]

hey, would it be possible to add another AIT node for upscaling using upscale model? so far that's the only part of the AIT workflow I made that's holding it back from going under 10 seconds per image.

cobntrolnet is borked with it

got most recent version of this repo, it errors out

File "G:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 125, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 151, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 311, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 289, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 241, in calc_cond_uncond_batch
c['control'] = control.get_control(input_x, timestep, c, len(cond_or_uncond))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 416, in get_control
control = self.aitemplate_controlnet(x_noisy, t, cond, self.cond_hint)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 378, in aitemplate_controlnet
return controlnet_inference(
^^^^^^^^^^^^^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\inference.py", line 146, in controlnet_inference
exe_module.run_with_tensors(inputs, ys, graph_mode=False)
File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 566, in run_with_tensors
outputs_ait = self.run(
^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 469, in run
return self._run_impl(
^^^^^^^^^^^^^^^
File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 408, in _run_impl
self.DLL.AITemplateModelContainerRun(
File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 212, in _wrapped_func
raise RuntimeError(f"Error in function: {method.name}")
RuntimeError: Error in function: AITemplateModelContainerRun

issue: TAESD preview fail.

AIT/AITemplate/AITemplate.py

Line 171 in 155bd9d

preview_bytes = previewer.decode_latent_to_preview_image(preview_format, x0)

Need to move to device other than cpu.

            x0 = x0.to(comfy.model_management.get_torch_device())

Compiling modules on Linux

This issue can serve as a discussion point for any issues related to compiling modules on Linux

My setup is as follows:

I use WSL, with Ubuntu 20.04

wsl --install Ubuntu-20.04
ubuntu2004.exe

CUDA installation is specific to WSL, for other distros see cuda-downloads.
Earliest known working CUDA toolkit version is 11.6, I'd recommend using the latest

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.2.1/local_installers/cuda-repo-wsl-ubuntu-12-2-local_12.2.1-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-12-2-local_12.2.1-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-12-2-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda

Ubuntu 20.04 includes Python 3.8.10, minimum Python version is 3.8, and I use PyTorch nightly CUDA 12.1, PyTorch version working at time of AITemplate release was 1.12.1, so anything newer than that should work
The following instructions use the build of AITemplate mentioned here, it includes the pending PRs, it is set up for CMake so that is disabled here, CUDA path is set so nvcc is found, a venv is used

sudo apt install python3-pip python3-venv unzip
wget https://github.com/FizzleDorf/AIT/files/12283714/aitemplate.zip
unzip aitemplate.zip
python3 -m venv venv
mv aitemplate/ venv/lib/python3.8/site-packages/
venv/bin/python -m pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121
venv/bin/python -m pip install click transformers diffusers accelerate
export CUDA_HOME=/usr/local/cuda
export PATH=${CUDA_HOME}/bin:${PATH}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
export AIT_USE_CMAKE_COMPILATION=0

To note, using that build is not necessary, and it will eventually become outdated, Installation from source is like

git clone --recursive https://github.com/facebookincubator/AITemplate
cd python
python setup.py bdist_wheel
pip install dist/*.whl

Until the pending PRs mentioned are merged they will need to be merged to a local branch before installation, or the changes applied to the installed package.

Here's some SDXL UNet modules I compiled earlier, covering 1024, 1536, 2048 max resolution, batch size 1, sm80
unet_xl_linux.zip

This is actually a 7z archive rather than .zip due to GitHub's file extension constraints.

cc @FizzleDorf These UNet modules are ready uploaded to Huggingface and added to modules.json, existing Linux VAE modules will work for SDXL with the fp16 fixed version

Compatibility issues caused by recent ComfyUI modifications.

Traceback (most recent call last):
  File "/home/me/git/ComfyUI/nodes.py", line 1694, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/me/git/ComfyUI/custom_nodes/AIT/__init__.py", line 1, in <module>
    from .AITemplate.AITemplate import AITemplateLoader, AITemplateControlNetLoader, AITemplateVAEDecode, AITemplateVAEEncode, VAEEncodeForInpaint, AITemplateEmptyLatentImage, AITemplateLatentUpscale
  File "/home/me/git/ComfyUI/custom_nodes/AIT/AITemplate/AITemplate.py", line 335, in <module>
    from comfy.sd import ControlBase
ImportError: cannot import name 'ControlBase' from 'comfy.sd' (/home/me/git/ComfyUI/comfy/sd.py)

ControlBase has been moved from comfy.sd to comfy.controlnet due to recent changes.

About supported resolution.

I'm contemplating how to support AIT in the Detailer node, and it seems quite challenging to apply in a different way.

So, when the model passed to the Detailer is an AIT model, I intend to approach it by restricting the upscale factor to the resolution supported by AIT.

To achieve this, I need to determine the supported resolutions from the model passed to the Detailer. Is it possible for the model to provide such information?

Bring back the ability to load any model

trying to use "25fdbed983314f8b23c361180e861da73f8c24f4a194ecd6bac01fc8bb2d483a.so" receives this error

`Found 3 modules for windows xl sm80 1 1024 unet
Using f64294f9bae73c4f742a4f61e8567460d1abefb461c792f69e8922c74331cd27
100%|██████████████████████████████████████████████████████████████████████████████████| 67/67 [00:08<00:00, 9.82it/s]
Found 3 modules for windows v1 sm80 1 1024 unet
Using 25fdbed983314f8b23c361180e861da73f8c24f4a194ecd6bac01fc8bb2d483a
[08:15:48]: NVIDIA GeForce RTX 4070 Ti
[08:15:48]: AITemplate modules compiled by hlky
[08:15:48]: Error: SetConstant did not receive correct number of bytes for unbound constant time_embedding_linear_1_weight: expected 819200 but got 1179648. Check that the provided tensor's shape is correct.
!!! Exception during processing !!!
Traceback (most recent call last):
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\execution.py", line 144, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\execution.py", line 74, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\execution.py", line 67, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\nodes.py", line 1165, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 170, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 269, in sample
AITemplate.unet[module['sha256']] = AITemplate.loader.apply_unet(
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\custom_nodes\AIT\AITemplate\ait\load.py", line 215, in apply_unet
return self.apply(aitemplate_module, ait_params)
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\custom_nodes\AIT\AITemplate\ait\load.py", line 194, in apply
aitemplate_module.set_many_constants_with_tensors(ait_params)
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 839, in set_many_constants_with_tensors
self.set_many_constants(ait_tensors)
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 790, in set_many_constants
self.DLL.AITemplateModelContainerSetManyConstants(
File "C:\Users\joker\OneDrive\Desktop\A.I\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 211, in _wrapped_func
raise RuntimeError(f"Error in function: {method.name}")
RuntimeError: Error in function: AITemplateModelContainerSetManyConstants

Prompt executed in 25.88 seconds`

Module support

Intro

Modules have to be compiled for a range of resolutions, batch sizes, windows and linux, and multiple cuda architectures. This process takes a long time, so the goal is to minimize the number of modules that have to be recompiled in case of an update like has just happened with 8px increment support, while maximizing the overall support.

Why are there multiple modules?

The profiling that occurs when compiling AITemplate modules selects the underlying kernels based on resolution, batch size and cuda architecture.
Most importantly is that a module profiled on sm80 is not likely to work on sm75 because sm75 supports different kernels.
The other factor is performance, one kernel may perform better at higher resolution and another at smaller resolution.
The amount of memory allocated by the module is pre-calculated based on the profiling.

What should be supported?

At the moment, there are modules for batch size 1, windows and linux, sm80. Check your cuda architecture here.
sm80 covers everything 80+. sm70 and sm75 however are unique. sm6X and older are unsupported. there needs to be some testing done on whether sm89 benefits from modules profiled for it

unet

Note: v1 control is for use with v1 controlnet modules, controlnet requires an accompanying unet module that accepts the additional inputs

v1
v2
xl
xlr
v1 control

unet that could be supported

v1 inpaint
v2 inpaint
v2 control

vae

vae modules are compatible across architectures, however there could be support for fp32 modules, these would work with the vae models that don't work in fp16

controlnet

v1 is supported, v2 could be supported

Resolutions

vae

256, 512, 1024, 2048, 4096

unet

768 (not for xl), 1024, 1536, 2048, 4096

controlnet

768, 1024, 1536, 2048

Vote on what you will use

React to the comments below to vote on what you will use

Anything else?

VRAM usage as estimated by AITemplate is in modules.json, in mb, excluding the size of the weights, this might help narrow the choices of resolution and batch size.
It would also be great to verify these figures against what people are seeing.

Loading LoRA and changing weights generates noisy images

Thank you for your hard work on the AITemplate port for comfy!

I have found a bug, whenever you change a lora weight, change a lora or connect a new one, the generated image becomes a noisy mess, this is when using the keep loaded option enabled, and if disabled it can sometimes trigger the same type of generation when using a loRA even when not changing its weights nor adding/changing the lora itself.

The bug corrects when changing the whole model or by restarting comfy, seems related to the autoload AIT module feature. And is more likely to generate without lora change when using 3 or more loras.

The following image has only the AIT nodes and a custom CLIP encode which I tested using the normal CLIP encode and the bug persists

Images not getting diffused correctly at 840x1176 (windows xl sm80 1 1176 unet)

On a resolution of ~~840x1776~~ 840x1176 (windows xl sm80 1 1176 unet) images are not getting diffused correctly. It goes away if you lower the resolution to ex. 848x1184 (windows xl sm80 1 1184 unet works) and AIT switches to a different module.

EDIT: resolution typo corrected to 840x1176

It's not seed, prompt or sampler dependent.

@Shaistrong kindly helped confirming the issue.

Tested on Windows 11, 4090 (536.67), latest version of ComfyUI and the AIT custom node

first stages during diffusion (ksampler image)

final output image

module 'comfy.sd' has no attribute 'ModelPatcher'

Was reported here comfyanonymous/ComfyUI#1426

!!! Exception during processing !!!
Traceback (most recent call last):
File "K:\AI ART UNIVERSE\COMFY-UI\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "K:\AI ART UNIVERSE\COMFY-UI\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "K:\AI ART UNIVERSE\COMFY-UI\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "K:\AI ART UNIVERSE\COMFY-UI\ComfyUI_windows_portable\ComfyUI\nodes.py", line 577, in load_controlnet
controlnet = comfy.controlnet.load_controlnet(controlnet_path)
File "K:\AI ART UNIVERSE\COMFY-UI\ComfyUI_windows_portable\ComfyUI\comfy\controlnet.py", line 394, in load_controlnet
control = ControlNet(control_model, global_average_pooling=global_average_pooling)
File "K:\AI ART UNIVERSE\COMFY-UI\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 347, in init
self.control_model_wrapped = comfy.sd.ModelPatcher(self.control_model, load_device=comfy.model_management.get_torch_device(), offload_device=comfy.model_management.unet_offload_device())
AttributeError: module 'comfy.sd' has no attribute 'ModelPatcher'

[Bug] LoRA issues

~~If you hook up a LoraLoader node before the AITemplateLoader node, every change to the LoRA will further corrupt gens. Putting the LoraLoader node after the AITemplateLoader node seems to be okay.~~

EDIT: Applying LoRAs in general and alternating their values corrupts gens.

Reproduction:

Generate an image with a fixed seed and a LoRA applied.
Decrement LoRA values to 0.99. Queue to generate an image but quickly cancel.
Increment LoRA values back to 1.00. Generate an image.
See wildly different image

There seems to be no issue if I let the images generate without canceling. I think the issue has something to do with cancelling gens.

issue: compatibility patch required

AIT/AITemplate/AITemplate.py

Line 348 in d544b7b

 self.control_model_wrapped = comfy.sd.ModelPatcher(self.control_model, load_device=comfy.model_management.get_torch_device(), offload_device=comfy.model_management.unet_offload_device()) 

ModelPatcher is moved to separated file(comfy/model_patcher.py) from comfy/sd.py

#48

VAE Encode (AI Template) Shifts Image Down and to the Right.

Example:
https://i.imgur.com/8QTd3cH.png
Simple Setup to reproduce:
Example.txt

Question about this and A1111

It will be viable to A1111 SD? I dont really like to use Comfy but tested AIT and render time is very good.

AITemplate sometimes fails with [SetValue] Dimension got value out of bounds: expected value to be in [1,2], got 3

I've found a case where AITemplate fails a generation that succeeds without it.

It's got something to do with how ComfyUI batches conds; in some cases, it'll do chunks with size > 2, which breaks AITemplate.

I can work around the problem by removing the batching logic in comfy/samplers.py after the line to_batch = to_batch_temp[:1] and limiting it to two at most. See this patch

I've attached a workflow to reproduce the issue (on my system, at least).

To run the workflow you'll need my prompt control nodes from https://github.com/asagi4/comfyui-prompt-control since I haven't hit this bug without timestep ranges yet.

It doesn't seem to happen with all seeds, either. I used RevAnimated for the reproduction, but any model should do.

bug.zip

got prompt
2
3
3
Found 4 modules for windows v1 sm80 1 512 unet_control
Using 1b7bca45b73c3a9dd17a11070217f44d3ae6102d3b0d33400b3ed34e9a516712
  0%|                                                                                              | 0/30 [00:00<?, ?it/s]Found 4 modules for windows v1 sm80 1.0 512 controlnet
Using ef60d80894710e8e5a6fffd2d55a5da2c6a6c5adff75946b99a1e18d75f02449
Got cutlass error: Error InternalError: Got cutlass error: Error Internal
  0%|                                                                                              | 0/30 [00:00<?, ?it/s]
!!! Exception during processing !!!
Traceback (most recent call last):
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1236, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 176, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\hacky.py", line 22, in informative_sample
    raise e
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\hacky.py", line 9, in informative_sample
    return original_sample(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 310, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 741, in sample
    samples = getattr(k_diffusion_sampling, "sample_{}".format(self.sampler))(self.model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 137, in sample_euler
    denoised = model(x, sigma_hat * s_in, **extra_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1531, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 322, in forward
    out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1531, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 125, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 151, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 310, in apply_model
    out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 288, in sampling_function
    cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 241, in calc_cond_uncond_batch
    c['control'] = control.get_control(input_x, timestep_, c, len(cond_or_uncond))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 419, in get_control
    control = self.aitemplate_controlnet(x_noisy, t, cond, self.cond_hint)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 381, in aitemplate_controlnet
    return controlnet_inference(
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\inference.py", line 146, in controlnet_inference
    exe_module.run_with_tensors(inputs, ys, graph_mode=False)
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 566, in run_with_tensors
    outputs_ait = self.run(
                  ^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 469, in run
    return self._run_impl(
           ^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 408, in _run_impl
    self.DLL.AITemplateModelContainerRun(
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\ait\module\model.py", line 212, in _wrapped_func
    raise RuntimeError(f"Error in function: {method.__name__}")
RuntimeError: Error in function: AITemplateModelContainerRun

With the AIT Loader and AIT ControlNet nodes muted, I get another error NameError: name 'c_crossattn' is not defined:

!!! Exception during processing !!!
Traceback (most recent call last):
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1236, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 176, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\hacky.py", line 9, in informative_sample
    return original_sample(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 310, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 741, in sample
    samples = getattr(k_diffusion_sampling, "sample_{}".format(self.sampler))(self.model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 137, in sample_euler
    denoised = model(x, sigma_hat * s_in, **extra_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1531, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 322, in forward
    out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1531, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 125, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 151, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 310, in apply_model
    out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 288, in sampling_function
    cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 241, in calc_cond_uncond_batch
    c['control'] = control.get_control(input_x, timestep_, c, len(cond_or_uncond))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Anon\Downloads\Programs\ComfyUI_windows_portable\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 413, in get_control
    context = c_crossattn
              ^^^^^^^^^^^
NameError: name 'c_crossattn' is not defined

fizzledorf / ait Goto Github PK

ait's People

Contributors

Stargazers

Watchers

Forkers

ait's Issues

UNet

VAE

CLIP

New modules

Intro

Why are there multiple modules?

What should be supported?

unet

unet that could be supported

vae

controlnet

Resolutions

vae

unet

controlnet

Vote on what you will use

Anything else?

Recommend Projects

Recommend Topics

Recommend Org