aszc-dev / comfyui-coremlsuite Goto Github PK

When I have no LoRa everything works perfectly fine. However when I added LoRa to the conversion the output is always garbage (shown in picture). Is there something I did wrong in my workflow with regard to LoRa?

Is LoRa not supported in turbo SDXL models conversions? This LoRa (Trained on non-turbo though) works perfectly fine with this model in DrawThings so I dont think its a LoRa/Model compatibility issue.

The CoreML detail hook provider only allows certain possibilities. Is it possible to make it flexible to non-standard dimensions? For example, I converted an SDXL model to the resolution 832x1216, but there is no corresponding value. Is it possible to modify the fields or add the values?

When converting a checkpoint, the model will be converted to the .mlpackage file. However, after conversion, the node will check whether there exists a .mlmodelc format file in the 'unet' directory. This results in an error and prevents further progress. I am using the 'convert checkpoint' example workflow

I'm really impressed by the quality of the documentation. I'm glad that my initial suggestion led to such clarity.

A few additional questions to address in the readme that might help clarify things:

1. People are used to refer to Apple hardware as "MPS". What's the difference between MPS, GPU (which you mention), and ANE? Is MPS=ANE? Is MPS=GPU?
2. Which one should people choose between ANE, GPU, and MPS? What difference does it make, if any?
3. Most Apple users adopting ComfyUI are used to adding export PYTORCH_ENABLE_MPS_FALLBACK=1 to their launch script or as a variable in their environment. Is it still necessary? If not, how do I remove it? If yes, how does that influence the behavior of the nodes with ANE/GPU/MPS?
4. Obviously, the least disruptive scenario is the use of the experimental Core ML Adapter node. Aside from the possible errors you already mentioned, is there a performance penalty compared to manually converting the models to mlmodelc or mlpackage formats?
5. Does the Core ML Adapter node experimentally support SDXL? Will it ever, or is there a technical limitation?
6. People have learned that safetensors is a safe format compared to pickles. What about mlmodelc or mlpackage formats? Can people download either from a place like Civitai and be safe? Or should they always download safetensors files and convert them on their machines?
7. Do Core ML converted models behave in the same way as their safetensors counterparts if all the parameters of a generation remain identical? Or should people expect a different image output despite the seed and everything is identical?
8. If people use a conversion node, does the conversion take place every time you queue a generation? If so, isn't it better to convert only once and then use a load node with the newly converted checkpoint?
9. SDXL on ANE is marked as not supported. Will it ever be supported or are there technical reasons that prevent it?
10. The Limitation section of the Readme mentions SD1.5 and SD2.1, but makes no mentions of SDXL. Are there other limitations people should be aware of?

You have done a massive job in converting the core ComyfUI nodes so that the Apple community can finally enjoy some decent performance. I'm very grateful for the contribution. Thank you.

Updated to latest comfy, your package and..
and wether I ask for CPU & GPU or with ANE I get the SAME results in terms of iterations (around 1.8 on a max 3)

if I use asitop, I see that the ANE is never activated
and if I use CPU and ANE, it uses the CPU and is slow ..

Here is my compo: drop it into comfy

cheers

A CoreML user reports that after 1777b54d021 patch of ComfyUI, only noise image is generated.

I noticed the convertor is keeping the mlpackage and models around after conversion, shouldn't the convertor drop the prelude file to save space (maybe an option in the node to keep the uncompiled version, as with each size needing a model, having 2x the files adds up quick.

Technically we end up with 3x models, the modelc the mlpackage and the safe tensor, would be nice to make keeping that middle one optional, and by default to just remove it after compilation.

Hi, want to ask if any issues are known about working with Apples M3max. My Manager always shows „Import failed“ after installing about ComfyUI Manager. Thanks!

cannot import name 'ModelSamplingDiscreteLCM' from 'comfy_extras.nodes_model_advanced'.

There should be an easy way of running sanity checks/regression tests to make sure that development doesn't introduce bugs. At least, workflows from README should always work.

Hey guys, im trying to run IPAdapter with CoreMLSuite but is not working, is there a why to make it work?

It seems this node had been intergrated with load checkpoint node, how could we load clip now with core ml workflow?

When run by hires.fix flow, i got this error, what should i do?

!!! Exception during processing !!! Traceback (most recent call last): File "/Users/lihuibiao/ComfyUI/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/Users/lihuibiao/ComfyUI/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/Users/lihuibiao/ComfyUI/execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "/Users/lihuibiao/ComfyUI/nodes.py", line 1344, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) File "/Users/lihuibiao/ComfyUI/nodes.py", line 1314, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "/Users/lihuibiao/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 173, in animatediff_sample return orig_comfy_sample(model, noise, *args, **kwargs) File "/Users/lihuibiao/ComfyUI/comfy/sample.py", line 37, in sample samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 755, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 657, in sample return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 644, in sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 623, in inner_sample samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 534, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options) File "/opt/homebrew/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/Users/lihuibiao/ComfyUI/comfy/k_diffusion/sampling.py", line 137, in sample_euler denoised = model(x, sigma_hat * s_in, **extra_args) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 272, in __call__ out = self.inner_model(x, sigma, model_options=model_options, seed=seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 610, in __call__ return self.predict_noise(*args, **kwargs) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 613, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 258, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks) File "/Users/lihuibiao/ComfyUI/comfy/model_base.py", line 97, in apply_model model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float() File "/Users/lihuibiao/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 23, in __call__ chunked_out = [ File "/Users/lihuibiao/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 25, in <listcomp> self.coreml_model(**input_kwargs.coreml_kwargs(self.expected_inputs)), File "/opt/homebrew/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 97, in __call__ self._verify_inputs(**kwargs) File "/opt/homebrew/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 90, in _verify_inputs raise TypeError( TypeError: Expected shape (1, 4, 64, 64), got (1, 4, 96, 96) for input: sample

How to reproduce:

Device - Base model M1 Air (7 Core GPU, 8GB RAM)

1. Install ComfyUI
2. Install CoreMLSuite

Start ComfyUI with FP16

python main.py --force-fp16

3. Load the workflow

Error log:

➜  ComfyUI git:(master) python3 main.py --force-fp16
Total VRAM 8192 MB, total RAM 8192 MB
Forcing FP16.
/Users/bvndls/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
Set vram state to: SHARED
Device: mps
VAE dtype: torch.float32
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
scikit-learn version 1.3.2 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Torch version 2.2.0.dev20231027 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.

Import times for custom nodes:
   0.4 seconds: /Users/bvndls/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite

Starting server

To see the GUI go to: http://127.0.0.1:8188
got prompt
INFO:coreml_suite.logger:Loading Unet.mlmodelc to CPU_AND_NE
INFO:python_coreml_stable_diffusion.coreml_model:Loading /Users/bvndls/ComfyUI/models/unet/Unet.mlmodelc
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 3.5 seconds.
Requested to load SD1ClipModel
Loading 1 new model
model_type EPS
adm 0
WARNING:coreml_suite.logger:No latent image provided, using empty tensor.
WARNING:coreml_suite.logger:Batch size is different from expected input size. Chunking and/or padding will be applied.
Requested to load CoreMLModelWrapper
Loading 1 new model
  0%|                                                                                                                                                                                                           | 0/20 [00:00<?, ?it/s]
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "/Users/bvndls/ComfyUI/execution.py", line 153, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/Users/bvndls/ComfyUI/execution.py", line 83, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/Users/bvndls/ComfyUI/execution.py", line 76, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/Users/bvndls/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 58, in sample
    return super().sample(
  File "/Users/bvndls/ComfyUI/nodes.py", line 1237, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
  File "/Users/bvndls/ComfyUI/nodes.py", line 1207, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
  File "/Users/bvndls/ComfyUI/comfy/sample.py", line 100, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 728, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler(), sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 633, in sample
    samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
  File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 589, in sample
    samples = getattr(k_diffusion_sampling, "sample_{}".format(sampler_name))(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **extra_options)
  File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/comfy/k_diffusion/sampling.py", line 580, in sample_dpmpp_2m
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 287, in forward
    out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, model_options=model_options, seed=seed)
  File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/comfy/k_diffusion/external.py", line 129, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "/Users/bvndls/ComfyUI/comfy/k_diffusion/external.py", line 155, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 275, in apply_model
    out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed)
  File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 253, in sampling_function
    cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, model_options)
  File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 229, in calc_cond_uncond_batch
    output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
  File "/Users/bvndls/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 45, in apply_model
    chunked_x = chunk_batch(x, sample_shape)
  File "/Users/bvndls/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/latents.py", line 14, in chunk_batch
    return [torch.cat((latent_image, padding), dim=0)]
RuntimeError: torch.cat(): all input tensors must be on the same device. Received mps:0 and cpu

Prompt executed in 5.47 seconds

Selecting different compute_unit or running ComfyUI without --force-fp16 or with either --gpu-only or --disable-smart-memory didn't help

Will SDXL Turbo support be possible i saw you got SDXL support working, i'm still reading up on Turbo, but the implementation details seem to point towards it being a different scheduler and some form of layer on top of SDXL as i read that you can pull turbo of of the base model and apply it to finetunes.

So does that mean we can technically just use it as a normal SDXL Model in the convertor and just need to use a different sampler to handle turbo noise correctly?

Edit: Seems that Turbo works fine using the standard comfy workflow, just using the Convertor + Adapter into the SamplerCustom, not sure if a SamplerDiscrete needs to be inlined or not, it doesn't seem to me a difference direct/eps/lcm set... but maybe i'm missing something.

I really think that the model strengths should be tagged in the files, its gonna definitly make for some confusion especially with more complicated workflows that have multiple loras and different lora values, if your worried about name length or something for the models maybe need to move away from storing the gen-params in the name and maybe go the way of having a .yaml named after the model next to it that stores the conversion details?

Some benchmark data showing comparison between safetensors MPS implementation and Core ML, presented in a table in readme could be helpful. It should contain machine info (Processor + RAM), conversion data (attention implementation), image size, compute unit, model used (SD1.5, SDXL, LCM, any one of them with LCM LoRA).

The example workflow works for me using e.g. using a converted dreamshaper_8.

However, if I convert an inpainting model (tried several, but e.g. sd-v1-5-inpainting.ckpt) to Core ML, I try to run the example workflow with it, the Core ML Sampler errors (see error 1 below). I've tried CPU_AND_NE, CPU_AND_GPU.

I think the example workflow automatically creates a latent the right size. However if I explicitly plug in an "empty latent image" with the right size (512x512, as the model was converted and double checked with the model name", the sampler errors as well (see error 2 below)

I believe the two errors may be related to the inpainting model, as using "empty latent image" with the right size with dreamshaper_8 works well.

Are inpainting models not supported? Any way to work it around? Is it an issue with this model in particular?

error 1:

Error occurred when executing CoreMLSampler:

The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1

Traceback (most recent call last):
File "/Users/user/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/Users/user/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/Users/user/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 65, in sample
return super().sample(
File "/Users/user/ComfyUI/nodes.py", line 1344, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/Users/user/ComfyUI/nodes.py", line 1314, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 111, in uncond_multiplier_check_cn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 22, in informative_sample
raise e
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample
return original_sample(*args, **kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
File "/Users/user/ComfyUI/comfy/sample.py", line 37, in sample
samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 761, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 663, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 650, in sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 629, in inner_sample
samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/Users/user/ComfyUI/comfy/samplers.py", line 534, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/Users/user/ComfyUI/comfy/k_diffusion/sampling.py", line 583, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/Users/user/ComfyUI/comfy/samplers.py", line 272, in call
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 616, in call
return self.predict_noise(*args, **kwargs)
File "/Users/user/ComfyUI/comfy/samplers.py", line 619, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 258, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
File "/Users/user/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch
output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 63, in apply_model_uncond_cleanup_wrapper
return orig_apply_model(self, *args, **kwargs)
File "/Users/user/ComfyUI/comfy/model_base.py", line 98, in apply_model
return self.model_sampling.calculate_denoised(sigma, model_output, x)
File "/Users/user/ComfyUI/comfy/model_sampling.py", line 12, in calculate_denoised
return model_input - model_output * sigma
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1

Prompt executed in 1.67 seconds

error 2:

Error occurred when executing CoreMLSampler:

Expected shape (1, 9, 64, 64), got (1, 4, 64, 64) for input: sample
Traceback (most recent call last):
File "/Users/user/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/Users/user/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/Users/user/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 65, in sample
return super().sample(
File "/Users/user/ComfyUI/nodes.py", line 1344, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/Users/user/ComfyUI/nodes.py", line 1314, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 111, in uncond_multiplier_check_cn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample
return original_sample(*args, **kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
File "/Users/user/ComfyUI/comfy/sample.py", line 37, in sample
samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 761, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 663, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 650, in sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 629, in inner_sample
samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/Users/user/ComfyUI/comfy/samplers.py", line 534, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/Users/user/ComfyUI/comfy/k_diffusion/sampling.py", line 583, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/Users/user/ComfyUI/comfy/samplers.py", line 272, in call
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 616, in call
return self.predict_noise(*args, **kwargs)
File "/Users/user/ComfyUI/comfy/samplers.py", line 619, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 258, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
File "/Users/user/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch
output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 63, in apply_model_uncond_cleanup_wrapper
return orig_apply_model(self, *args, **kwargs)
File "/Users/user/ComfyUI/comfy/model_base.py", line 97, in apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 23, in call
chunked_out = [
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 25, in
self.coreml_model(**input_kwargs.coreml_kwargs(self.expected_inputs)),
File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 97, in call
self._verify_inputs(**kwargs)
File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 90, in _verify_inputs
raise TypeError(
TypeError: Expected shape (1, 9, 64, 64), got (1, 4, 64, 64) for input: sample

Prompt executed in 0.06 seconds

So i've been trying to implement Enumerated Shapes and Ranges to try to get away from having to have a crap ton of the same model for various sizes.. and wow its SO SLOW.

Not sure if you've played around with it yet but either i'm doing something wrong or it's just really bad.

Not just the first run but even repeat generations...

Tested with C+N ORIG @ 512x512 (mainly because i forgot to swap to C+G before i rolled back my code)...
Git Main: 1s/it
Stock Pytorch: 1.27s/it
Enumerated: 5.05s/it. (the ranged i forgot to write down but it was similar).

GIT Main Model:

Enumerated Model:

That would be very handy to keep up with the updates.

https://github.com/ltdrdata/ComfyUI-Manager#how-to-register-your-custom-node-into-comfyui-manager

@aszc-dev
Anything in here that might lend itself to CoreMLSuite efforts?
MLX: An array framework for Apple silicon

https://github.com/ml-explore/mlx

MLX is an array framework for machine learning on Apple silicon, brought to you by Apple machine learning research.

Some key features of MLX include:

• Familiar APIs: MLX has a Python API that closely follows NumPy. MLX also has a fully featured C++ API, which closely mirrors the Python API. MLX has higher-level packages like mlx.nn and mlx.optimizers with APIs that closely follow PyTorch to simplify building more complex models.
• Composable function transformations: MLX has composable function transformations for automatic differentiation, automatic vectorization, and computation graph optimization.
• Lazy computation: Computations in MLX are lazy. Arrays are only materialized when needed.
• Dynamic graph construction: Computation graphs in MLX are built dynamically. Changing the shapes of function arguments does not trigger slow compilations, and debugging is simple and intuitive.
• Multi-device: Operations can run on any of the supported devices (currently, the CPU and GPU).
• Unified memory: A notable difference from MLX and other frameworks is the unified memory model. Arrays in MLX live in shared memory. Operations on MLX arrays can be performed on any of the supported device types without moving data.

MLX is designed by machine learning researchers for machine learning researchers. The framework is intended to be user-friendly, but still efficient to train and deploy models. The design of the framework itself is also conceptually simple. We intend to make it easy for researchers to extend and improve MLX with the goal of quickly exploring new ideas.

The design of MLX is inspired by frameworks like NumPy, PyTorch, Jax, and ArrayFire.

I'm running the ksampler and it generates an image fine after loras, etc, works great, but when i try to then also use the converted model in a experimental node to feed into facedetailer's model node i get back an error

TypeError: Expected shape (1, 4, 64, 64), got (1, 4, 135, 97) for input: sample

not sure if i'm using the experimental adapter incorrectly or if i need to do something to get it to work.

Question : what should i put in controlnet model .fp16.safetensors or .safetensors ?

The reason is I'm using fp16 lineart safetensors and it seems not affecting the result

Thanks

I have installed it through comfy Manager and went through the process of installing the requirements twice and still not sure how to sort this issue. Can you identify which package is causing this and how to resolve. Thanks for the wonderful work.

scikit-learn version 1.3.0 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Fail to import BlobReader from libmilstoragepython. No module named 'coremltools.libmilstoragepython'
Fail to import BlobWriter from libmilstoragepython. No module named 'coremltools.libmilstoragepython'
Traceback (most recent call last):
  File "D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1784, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CoreMLSuite\__init__.py", line 6, in <module>
    from coreml_suite.nodes import (
  File "D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CoreMLSuite\coreml_suite\nodes.py", line 4, in <module>
    from python_coreml_stable_diffusion.coreml_model import CoreMLModel
ModuleNotFoundError: No module named 'python_coreml_stable_diffusion'

Cannot import D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CoreMLSuite module for custom nodes: No module named 'python_coreml_stable_diffusion
```'

I had the following issue for a few days, but I didn't have the time to report it earlier. It used to work fine before.

scikit-learn version 1.3.0 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Torch version 2.2.0.dev20231004 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.
Traceback (most recent call last):
File "xyz/ComfyUI/nodes.py", line 1735, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/init.py", line 6, in
from coreml_suite.nodes import CoreMLLoaderUNet, CoreMLSampler, CoreMLModelAdapter
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 11, in
from coreml_suite.lcm.utils import add_lcm_model_options, lcm_patch, is_lcm
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/init.py", line 1, in
from .nodes import COREML_CONVERT_LCMT
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/nodes.py", line 6, in
from coreml_suite.lcm import converter as lcm_converter
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/converter.py", line 9, in
from diffusers import UNet2DConditionModel, LCMScheduler
ImportError: cannot import name 'LCMScheduler' from 'diffusers' (xyz/ComfyUI/venv/lib/python3.11/site-packages/diffusers/init.py)

Cannot import xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite module for custom nodes: cannot import name 'LCMScheduler' from 'diffusers' (xyz/ComfyUI/venv/lib/python3.11/site-packages/diffusers/init.py)

The result is that the ComfyUI fails the import.

Any chance you can restore support for nightly PyTorch builds? Thank you.

How to reproduce:

Device - Base model M1 Air (7 Core GPU, 8GB RAM)

1. Install ComfyUI
2. Install CoreMLSuite
3. Install ComfyUI Manager

Start ComfyUI with FP16

python main.py --force-fp16

4. Load the workflow

5. Load the snake image

Error log:

(comfy) ➜  ComfyUI git:(master) ✗ python3 main.py --force-fp16
** ComfyUI start up time: 2023-10-31 16:27:25.261506

Prestartup times for custom nodes:
   0.0 seconds: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Manager

Total VRAM 8192 MB, total RAM 8192 MB
Forcing FP16.
/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
Set vram state to: SHARED
Device: mps
VAE dtype: torch.float32
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
### Loading: ComfyUI-Manager (V0.36)
### ComfyUI Revision: 1631 [c837a173] | Released on '2023-10-30'
Registered sys.path: ['/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_pycocotools', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_oneformer', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/__init__.py', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_mmpkg', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_midas_repo', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_detectron2', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/controlnet_aux', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src', '/Users/bvndls/ComfyUI/ComfyUI/comfy', '/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/git/ext/gitdb', '/Users/bvndls/ComfyUI/ComfyUI', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload', '/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages', '/var/folders/dq/k_8x61qj6zgg60mnx259fyqh0000gn/T/tmp1wzcun6i', '../..']
/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/node_wrappers/dwpose.py:26: UserWarning: DWPose: Onnxruntime not found or doesn't come with acceleration providers, switch to OpenCV with CPU device. DWPose might run very slowly
  warnings.warn("DWPose: Onnxruntime not found or doesn't come with acceleration providers, switch to OpenCV with CPU device. DWPose might run very slowly")
scikit-learn version 1.3.2 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Torch version 2.2.0.dev20231030 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.

Import times for custom nodes:
   0.1 seconds: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Manager
   0.3 seconds: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux
   0.4 seconds: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite

Starting server

To see the GUI go to: http://127.0.0.1:8188
FETCH DATA from: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Manager/extension-node-map.json
got prompt
INFO:coreml_suite.logger:Loading ControlledUnet.mlmodelc to CPU_AND_NE
INFO:python_coreml_stable_diffusion.coreml_model:Loading /Users/bvndls/ComfyUI/ComfyUI/models/unet/ControlledUnet.mlmodelc
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 3.8 seconds.
clip unexpected: ['text_model.embeddings.position_ids']
Requested to load SD1ClipModel
Loading 1 new model
[] []
model_type EPS
adm 0
WARNING:coreml_suite.logger:No latent image provided, using empty tensor.
Requested to load CoreMLModelWrapper
Requested to load ControlNet
Loading 2 new models
  0%|                                                                                                                                                                                                           | 0/20 [00:00<?, ?it/s]
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "/Users/bvndls/ComfyUI/ComfyUI/execution.py", line 153, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/Users/bvndls/ComfyUI/ComfyUI/execution.py", line 83, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/Users/bvndls/ComfyUI/ComfyUI/execution.py", line 76, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 52, in sample
    return super().sample(
  File "/Users/bvndls/ComfyUI/ComfyUI/nodes.py", line 1237, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
  File "/Users/bvndls/ComfyUI/ComfyUI/nodes.py", line 1207, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/sample.py", line 100, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 728, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler(), sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 633, in sample
    samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 589, in sample
    samples = getattr(k_diffusion_sampling, "sample_{}".format(sampler_name))(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **extra_options)
  File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/k_diffusion/sampling.py", line 580, in sample_dpmpp_2m
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 287, in forward
    out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, model_options=model_options, seed=seed)
  File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/k_diffusion/external.py", line 129, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/k_diffusion/external.py", line 155, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 275, in apply_model
    out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 253, in sampling_function
    cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, model_options)
  File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 229, in calc_cond_uncond_batch
    output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
  File "/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 43, in apply_model
    chunked_out = [
  File "/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 44, in <listcomp>
    self._apply_model(
  File "/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 83, in _apply_model
    np_out = self.diffusion_model(**model_input_kwargs)["noise_pred"]
  File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 97, in __call__
    self._verify_inputs(**kwargs)
  File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 90, in _verify_inputs
    raise TypeError(
TypeError: Expected shape (2, 320, 64, 64), got (1, 320, 64, 64) for input: additional_residual_0

Prompt executed in 11.72 seconds

Resorces Used:

1. ControlNet from here
2. ControlledUnet.mlmodelc from here
3. CLIP from runwayml/stable-diffusion-v1-5 model.fp16.safetensors
4. VAE from runwayml/stable-diffusion-v1-5 diffusion_pytorch_model.fp16.safetensors

Tried with various combinations of UNets, ControlNets, CLIPs, VAEs and images, either throws this error, or generates an image without following the LoadImage connected to ApplyControlNet, exmaple below

Just wondering have you ran any comparisons between running against the MPS (metal) vs using the ANE for inference?

load failed, here is Traceback:

Traceback :
File "/Users/user/ComfyUI/nodes.py", line 1735, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/init.py", line 7, in
from coreml_suite.lcm import (
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/init.py", line 1, in
from .lcm_sampler import CoreMLSamplerLCM
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/lcm_sampler.py", line 5, in
from diffusers.utils.torch_utils import randn_tensor
ModuleNotFoundError: No module named 'diffusers.utils.torch_utils'

Cannot import /Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite module for custom nodes: No module named 'diffusers.utils.torch_utils'

Ya you knew i was gonna ask :) I know you've got your hands full but hope you can have the time maybe to look into dealing with the Stable Video Diffusion pipeline into coreml, not sure how possible it is given it's so new.

Whenever I try to load a lora when it gets to the sampler I get an error....

INFO:coreml_suite.logger:Batch size: 1
INFO:coreml_suite.logger:Width: 512, Height: 512
INFO:coreml_suite.logger:ControlNet support: False
INFO:coreml_suite.logger:Attention implementation: SPLIT_EINSUM
INFO:coreml_suite.logger:LoRAs used:
INFO:coreml_suite.logger:  epiCRealismHelper.safetensors - strength: 1.0
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "/Users/chris/AI/ComfyUI/execution.py", line 153, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chris/AI/ComfyUI/execution.py", line 83, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chris/AI/ComfyUI/execution.py", line 76, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chris/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 273, in convert
    converter.convert(
  File "/Users/chris/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/converter.py", line 300, in convert
    ref_pipe.set_adapters([adapter_name], adapter_weights=[strength])
  File "/Users/chris/AI/ComfyUI/venv/lib/python3.11/site-packages/diffusers/loaders.py", line 2476, in set_adapters
    self.unet.set_adapters(adapter_names, adapter_weights)
  File "/Users/chris/AI/ComfyUI/venv/lib/python3.11/site-packages/diffusers/loaders.py", line 723, in set_adapters
    raise ValueError("PEFT backend is required for `set_adapters()`.")
ValueError: PEFT backend is required for `set_adapters()`.```

Dear Developer,

I am using the 'Load LoRA to use with Core ML' node and the 'Convert Checkpoint to CoreML' node in my workflow. I successfully generated the combined .mlmodelc file (SD model: Cetus-Mix sd_v1-5.safetensor, LoRA model: Anime Lineart / Manga-like.safetensor, Combined model name: cetusMix_Whalefall2_lineArt_V4-16_1x512x768_se_unet.mlmodelc), but the generated images are blurry color blocks in the subsequent steps. I have also noticed that others might have had similar issues. Could you guide me on resolving this?

Thank you for your dedication; this project significantly contributes to Mac users.

Sincerely,

workflow screen shot

generated combined core ML model (sd model with lora model)

Updated the name because i realized whats causing this, i'm trying to recreate a workflow and the negative prompt is very long, and it seems that this totally destroys things, it seems like the ksampler is limited to 77 tokens

Workflow: easywf.json
I ran updates for comfy and CMLS

But i'm getting the following error

Loading 1 new model
  0%|                                                                                                                                                                                          | 0/20 [00:00<?, ?it/s]
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "/Users/user/AI/ComfyUI/execution.py", line 153, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/execution.py", line 83, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/execution.py", line 76, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 65, in sample
    return super().sample(
           ^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/nodes.py", line 1286, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/nodes.py", line 1256, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample
    return original_sample(*args, **kwargs)  # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/sample.py", line 100, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 711, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 617, in sample
    samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 556, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/k_diffusion/sampling.py", line 655, in sample_dpmpp_3m_sde
    denoised = model(x, sigmas[i] * s_in, **extra_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 277, in forward
    out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 267, in forward
    return self.apply_model(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 264, in apply_model
    out = sampling_function(self.inner_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 252, in sampling_function
    cond, uncond = calc_cond_uncond_batch(model, cond, uncond, x, timestep, model_options)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 230, in calc_cond_uncond_batch
    output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/comfy/model_base.py", line 73, in apply_model
    model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 23, in __call__
    chunked_out = [
                  ^
  File "/Users/user/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 25, in <listcomp>
    self.coreml_model(**input_kwargs.coreml_kwargs(self.expected_inputs)),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 97, in __call__
    self._verify_inputs(**kwargs)
  File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 90, in _verify_inputs
    raise TypeError(
TypeError: Expected shape (1, 768, 1, 77), got (1, 768, 1, 385) for input: encoder_hidden_states

Prompt executed in 267.22 secondsuser

Hi! Really like your repo, and am trying to utilize the Core ML Adapter (Experimental) node, and I understand:

This is an experimental node and may not work with all models and nodes. Please use with caution and pay attention to the expected inputs of the model.

When I try to link your CoreML suite into the following repo [workflows], https://github.com/cubiq/ComfyUI_InstantID/blob/main/examples/InstantID_basic.json, I get errors such as:
assert y.shape[0] == x.shape[0] AttributeError: 'NoneType' object has no attribute 'shape'

So my question is, how do I pay attention to the expected inputs of the model. like you suggest?

Any tips!? Thanks in advance!!!

I was getting an error that ComfyUI couldn't import ComfyUI-CoreMLSuite. This may not have been the issue but it fixed it for me:
In:
custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/unet.py

overrides module not installed. I added that to requirements.txt.
https://pypi.org/project/overrides/

Import times for custom nodes:
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/gcLatentTunnel.py
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/SDXLAspectRatio.py
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/FreeU_Advanced
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI_TiledKSampler
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/wlsh_nodes
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI_Dave_CustomNode
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/mikey_nodes
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ymc-node-suite-comfyui
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI-QualityOfLifeSuit_Omar92
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI_UltimateSDUpscale
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI_Comfyroll_CustomNodes
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI-LCM
0.1 seconds: /Users/rovo/ComfyUI/custom_nodes/SeargeSDXL
0.1 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI-Manager
0.2 seconds (IMPORT FAILED): /Users/rovo/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite

It seems there's no ModelSamplingDiscreteLCM in comfy_extras.nodes_model_advanced file

Did I do something wrong ?

Thanks

Fresh install, I'm not sure if the torch version warning is actually relevant. On MacOS 14.4.

scikit-learn version 1.4.1.post1 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API. Torch version 2.2.1 has not been tested with coremltools. You may run into unexpected errors. Torch 2.1.0 is the most recent version that has been tested. /Users/francis/Library/Python/3.9/lib/python/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( Traceback (most recent call last): File "/Users/francis/ComfyUI/nodes.py", line 1887, in load_custom_node module_spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/Users/francis/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite-main/__init__.py", line 6, in <module> from coreml_suite.nodes import ( File "/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite-main/coreml_suite/nodes.py", line 9, in <module> from coreml_suite import converter File "/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite-main/coreml_suite/converter.py", line 318, in <module> lora_weights: list[tuple[str | os.PathLike, float]] = None, TypeError: unsupported operand type(s) for |: 'type' and 'ABCMeta'