aszc-dev / comfyui-coremlsuite Goto Github PK
View Code? Open in Web Editor NEWA set of custom nodes for ComfyUI that allow you to use Core ML models in your ComfyUI workflows.
License: GNU General Public License v3.0
A set of custom nodes for ComfyUI that allow you to use Core ML models in your ComfyUI workflows.
License: GNU General Public License v3.0
When I have no LoRa everything works perfectly fine. However when I added LoRa to the conversion the output is always garbage (shown in picture). Is there something I did wrong in my workflow with regard to LoRa?
Is LoRa not supported in turbo SDXL models conversions? This LoRa (Trained on non-turbo though) works perfectly fine with this model in DrawThings so I dont think its a LoRa/Model compatibility issue.
The CoreML detail hook provider only allows certain possibilities. Is it possible to make it flexible to non-standard dimensions? For example, I converted an SDXL model to the resolution 832x1216, but there is no corresponding value. Is it possible to modify the fields or add the values?
When converting a checkpoint, the model will be converted to the .mlpackage file. However, after conversion, the node will check whether there exists a .mlmodelc format file in the 'unet' directory. This results in an error and prevents further progress. I am using the 'convert checkpoint' example workflow
I'm really impressed by the quality of the documentation. I'm glad that my initial suggestion led to such clarity.
A few additional questions to address in the readme that might help clarify things:
export PYTORCH_ENABLE_MPS_FALLBACK=1
to their launch script or as a variable in their environment. Is it still necessary? If not, how do I remove it? If yes, how does that influence the behavior of the nodes with ANE/GPU/MPS?Core ML Adapter
node. Aside from the possible errors you already mentioned, is there a performance penalty compared to manually converting the models to mlmodelc
or mlpackage
formats?Core ML Adapter
node experimentally support SDXL? Will it ever, or is there a technical limitation?safetensors
is a safe format compared to pickles. What about mlmodelc
or mlpackage
formats? Can people download either from a place like Civitai and be safe? Or should they always download safetensors
files and convert them on their machines?You have done a massive job in converting the core ComyfUI nodes so that the Apple community can finally enjoy some decent performance. I'm very grateful for the contribution. Thank you.
Updated to latest comfy, your package and..
and wether I ask for CPU & GPU or with ANE I get the SAME results in terms of iterations (around 1.8 on a max 3)
if I use asitop, I see that the ANE is never activated
and if I use CPU and ANE, it uses the CPU and is slow ..
Here is my compo: drop it into comfy
cheers
A CoreML user reports that after 1777b54d021 patch of ComfyUI, only noise image is generated.
I noticed the convertor is keeping the mlpackage and models around after conversion, shouldn't the convertor drop the prelude file to save space (maybe an option in the node to keep the uncompiled version, as with each size needing a model, having 2x the files adds up quick.
Technically we end up with 3x models, the modelc the mlpackage and the safe tensor, would be nice to make keeping that middle one optional, and by default to just remove it after compilation.
Hi, want to ask if any issues are known about working with Apples M3max. My Manager always shows „Import failed“ after installing about ComfyUI Manager. Thanks!
cannot import name 'ModelSamplingDiscreteLCM' from 'comfy_extras.nodes_model_advanced'.
There should be an easy way of running sanity checks/regression tests to make sure that development doesn't introduce bugs. At least, workflows from README should always work.
Hey guys, im trying to run IPAdapter with CoreMLSuite but is not working, is there a why to make it work?
It seems this node had been intergrated with load checkpoint node, how could we load clip now with core ml workflow?
When run by hires.fix flow, i got this error, what should i do?
!!! Exception during processing !!! Traceback (most recent call last): File "/Users/lihuibiao/ComfyUI/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/Users/lihuibiao/ComfyUI/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/Users/lihuibiao/ComfyUI/execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "/Users/lihuibiao/ComfyUI/nodes.py", line 1344, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) File "/Users/lihuibiao/ComfyUI/nodes.py", line 1314, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "/Users/lihuibiao/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 173, in animatediff_sample return orig_comfy_sample(model, noise, *args, **kwargs) File "/Users/lihuibiao/ComfyUI/comfy/sample.py", line 37, in sample samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 755, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 657, in sample return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 644, in sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 623, in inner_sample samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 534, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options) File "/opt/homebrew/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/Users/lihuibiao/ComfyUI/comfy/k_diffusion/sampling.py", line 137, in sample_euler denoised = model(x, sigma_hat * s_in, **extra_args) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 272, in __call__ out = self.inner_model(x, sigma, model_options=model_options, seed=seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 610, in __call__ return self.predict_noise(*args, **kwargs) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 613, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 258, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) File "/Users/lihuibiao/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks) File "/Users/lihuibiao/ComfyUI/comfy/model_base.py", line 97, in apply_model model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float() File "/Users/lihuibiao/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 23, in __call__ chunked_out = [ File "/Users/lihuibiao/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 25, in <listcomp> self.coreml_model(**input_kwargs.coreml_kwargs(self.expected_inputs)), File "/opt/homebrew/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 97, in __call__ self._verify_inputs(**kwargs) File "/opt/homebrew/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 90, in _verify_inputs raise TypeError( TypeError: Expected shape (1, 4, 64, 64), got (1, 4, 96, 96) for input: sample
Device - Base model M1 Air (7 Core GPU, 8GB RAM)
Start ComfyUI with FP16
python main.py --force-fp16
➜ ComfyUI git:(master) python3 main.py --force-fp16
Total VRAM 8192 MB, total RAM 8192 MB
Forcing FP16.
/Users/bvndls/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
warnings.warn(
Set vram state to: SHARED
Device: mps
VAE dtype: torch.float32
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
scikit-learn version 1.3.2 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Torch version 2.2.0.dev20231027 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.
Import times for custom nodes:
0.4 seconds: /Users/bvndls/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite
Starting server
To see the GUI go to: http://127.0.0.1:8188
got prompt
INFO:coreml_suite.logger:Loading Unet.mlmodelc to CPU_AND_NE
INFO:python_coreml_stable_diffusion.coreml_model:Loading /Users/bvndls/ComfyUI/models/unet/Unet.mlmodelc
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 3.5 seconds.
Requested to load SD1ClipModel
Loading 1 new model
model_type EPS
adm 0
WARNING:coreml_suite.logger:No latent image provided, using empty tensor.
WARNING:coreml_suite.logger:Batch size is different from expected input size. Chunking and/or padding will be applied.
Requested to load CoreMLModelWrapper
Loading 1 new model
0%| | 0/20 [00:00<?, ?it/s]
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "/Users/bvndls/ComfyUI/execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/Users/bvndls/ComfyUI/execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/Users/bvndls/ComfyUI/execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/Users/bvndls/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 58, in sample
return super().sample(
File "/Users/bvndls/ComfyUI/nodes.py", line 1237, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/Users/bvndls/ComfyUI/nodes.py", line 1207, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/Users/bvndls/ComfyUI/comfy/sample.py", line 100, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 728, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler(), sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 633, in sample
samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 589, in sample
samples = getattr(k_diffusion_sampling, "sample_{}".format(sampler_name))(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **extra_options)
File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/Users/bvndls/ComfyUI/comfy/k_diffusion/sampling.py", line 580, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 287, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, model_options=model_options, seed=seed)
File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/bvndls/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/bvndls/ComfyUI/comfy/k_diffusion/external.py", line 129, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "/Users/bvndls/ComfyUI/comfy/k_diffusion/external.py", line 155, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 275, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed)
File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 253, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, model_options)
File "/Users/bvndls/ComfyUI/comfy/samplers.py", line 229, in calc_cond_uncond_batch
output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
File "/Users/bvndls/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 45, in apply_model
chunked_x = chunk_batch(x, sample_shape)
File "/Users/bvndls/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/latents.py", line 14, in chunk_batch
return [torch.cat((latent_image, padding), dim=0)]
RuntimeError: torch.cat(): all input tensors must be on the same device. Received mps:0 and cpu
Prompt executed in 5.47 seconds
Selecting different compute_unit or running ComfyUI without --force-fp16 or with either --gpu-only or --disable-smart-memory didn't help
Will SDXL Turbo support be possible i saw you got SDXL support working, i'm still reading up on Turbo, but the implementation details seem to point towards it being a different scheduler and some form of layer on top of SDXL as i read that you can pull turbo of of the base model and apply it to finetunes.
So does that mean we can technically just use it as a normal SDXL Model in the convertor and just need to use a different sampler to handle turbo noise correctly?
Edit: Seems that Turbo works fine using the standard comfy workflow, just using the Convertor + Adapter into the SamplerCustom, not sure if a SamplerDiscrete needs to be inlined or not, it doesn't seem to me a difference direct/eps/lcm set... but maybe i'm missing something.
I really think that the model strengths should be tagged in the files, its gonna definitly make for some confusion especially with more complicated workflows that have multiple loras and different lora values, if your worried about name length or something for the models maybe need to move away from storing the gen-params in the name and maybe go the way of having a .yaml named after the model next to it that stores the conversion details?
Some benchmark data showing comparison between safetensors MPS implementation and Core ML, presented in a table in readme could be helpful. It should contain machine info (Processor + RAM), conversion data (attention implementation), image size, compute unit, model used (SD1.5, SDXL, LCM, any one of them with LCM LoRA).
The example workflow works for me using e.g. using a converted dreamshaper_8.
However, if I convert an inpainting model (tried several, but e.g. sd-v1-5-inpainting.ckpt) to Core ML, I try to run the example workflow with it, the Core ML Sampler errors (see error 1 below). I've tried CPU_AND_NE, CPU_AND_GPU.
I think the example workflow automatically creates a latent the right size. However if I explicitly plug in an "empty latent image" with the right size (512x512, as the model was converted and double checked with the model name", the sampler errors as well (see error 2 below)
I believe the two errors may be related to the inpainting model, as using "empty latent image" with the right size with dreamshaper_8 works well.
Are inpainting models not supported? Any way to work it around? Is it an issue with this model in particular?
error 1:
Error occurred when executing CoreMLSampler:
The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
Traceback (most recent call last):
File "/Users/user/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/Users/user/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/Users/user/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 65, in sample
return super().sample(
File "/Users/user/ComfyUI/nodes.py", line 1344, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/Users/user/ComfyUI/nodes.py", line 1314, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 111, in uncond_multiplier_check_cn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 22, in informative_sample
raise e
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample
return original_sample(*args, **kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
File "/Users/user/ComfyUI/comfy/sample.py", line 37, in sample
samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 761, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 663, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 650, in sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 629, in inner_sample
samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/Users/user/ComfyUI/comfy/samplers.py", line 534, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/Users/user/ComfyUI/comfy/k_diffusion/sampling.py", line 583, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/Users/user/ComfyUI/comfy/samplers.py", line 272, in call
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 616, in call
return self.predict_noise(*args, **kwargs)
File "/Users/user/ComfyUI/comfy/samplers.py", line 619, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 258, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
File "/Users/user/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch
output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 63, in apply_model_uncond_cleanup_wrapper
return orig_apply_model(self, *args, **kwargs)
File "/Users/user/ComfyUI/comfy/model_base.py", line 98, in apply_model
return self.model_sampling.calculate_denoised(sigma, model_output, x)
File "/Users/user/ComfyUI/comfy/model_sampling.py", line 12, in calculate_denoised
return model_input - model_output * sigma
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
Prompt executed in 1.67 seconds
error 2:
Error occurred when executing CoreMLSampler:
Expected shape (1, 9, 64, 64), got (1, 4, 64, 64) for input: sample
Traceback (most recent call last):
File "/Users/user/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/Users/user/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/Users/user/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 65, in sample
return super().sample(
File "/Users/user/ComfyUI/nodes.py", line 1344, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/Users/user/ComfyUI/nodes.py", line 1314, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 111, in uncond_multiplier_check_cn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample
return original_sample(*args, **kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
File "/Users/user/ComfyUI/comfy/sample.py", line 37, in sample
samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 761, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 663, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 650, in sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 629, in inner_sample
samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/Users/user/ComfyUI/comfy/samplers.py", line 534, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/Users/user/ComfyUI/comfy/k_diffusion/sampling.py", line 583, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/Users/user/ComfyUI/comfy/samplers.py", line 272, in call
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 616, in call
return self.predict_noise(*args, **kwargs)
File "/Users/user/ComfyUI/comfy/samplers.py", line 619, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "/Users/user/ComfyUI/comfy/samplers.py", line 258, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
File "/Users/user/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch
output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks)
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 63, in apply_model_uncond_cleanup_wrapper
return orig_apply_model(self, *args, **kwargs)
File "/Users/user/ComfyUI/comfy/model_base.py", line 97, in apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 23, in call
chunked_out = [
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 25, in
self.coreml_model(**input_kwargs.coreml_kwargs(self.expected_inputs)),
File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 97, in call
self._verify_inputs(**kwargs)
File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 90, in _verify_inputs
raise TypeError(
TypeError: Expected shape (1, 9, 64, 64), got (1, 4, 64, 64) for input: sample
Prompt executed in 0.06 seconds
So i've been trying to implement Enumerated Shapes and Ranges to try to get away from having to have a crap ton of the same model for various sizes.. and wow its SO SLOW.
Not sure if you've played around with it yet but either i'm doing something wrong or it's just really bad.
Not just the first run but even repeat generations...
Tested with C+N ORIG @ 512x512 (mainly because i forgot to swap to C+G before i rolled back my code)...
Git Main: 1s/it
Stock Pytorch: 1.27s/it
Enumerated: 5.05s/it. (the ranged i forgot to write down but it was similar).
That would be very handy to keep up with the updates.
https://github.com/ltdrdata/ComfyUI-Manager#how-to-register-your-custom-node-into-comfyui-manager
@aszc-dev
Anything in here that might lend itself to CoreMLSuite efforts?
MLX: An array framework for Apple silicon
https://github.com/ml-explore/mlx
MLX is an array framework for machine learning on Apple silicon, brought to you by Apple machine learning research.
Some key features of MLX include:
MLX is designed by machine learning researchers for machine learning researchers. The framework is intended to be user-friendly, but still efficient to train and deploy models. The design of the framework itself is also conceptually simple. We intend to make it easy for researchers to extend and improve MLX with the goal of quickly exploring new ideas.
The design of MLX is inspired by frameworks like NumPy, PyTorch, Jax, and ArrayFire.
I'm running the ksampler and it generates an image fine after loras, etc, works great, but when i try to then also use the converted model in a experimental node to feed into facedetailer's model node i get back an error
TypeError: Expected shape (1, 4, 64, 64), got (1, 4, 135, 97) for input: sample
not sure if i'm using the experimental adapter incorrectly or if i need to do something to get it to work.
Question : what should i put in controlnet model .fp16.safetensors or .safetensors ?
The reason is I'm using fp16 lineart safetensors and it seems not affecting the result
Thanks
I have installed it through comfy Manager and went through the process of installing the requirements twice and still not sure how to sort this issue. Can you identify which package is causing this and how to resolve. Thanks for the wonderful work.
scikit-learn version 1.3.0 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Fail to import BlobReader from libmilstoragepython. No module named 'coremltools.libmilstoragepython'
Fail to import BlobWriter from libmilstoragepython. No module named 'coremltools.libmilstoragepython'
Traceback (most recent call last):
File "D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1784, in load_custom_node
module_spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CoreMLSuite\__init__.py", line 6, in <module>
from coreml_suite.nodes import (
File "D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CoreMLSuite\coreml_suite\nodes.py", line 4, in <module>
from python_coreml_stable_diffusion.coreml_model import CoreMLModel
ModuleNotFoundError: No module named 'python_coreml_stable_diffusion'
Cannot import D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CoreMLSuite module for custom nodes: No module named 'python_coreml_stable_diffusion
```'
I had the following issue for a few days, but I didn't have the time to report it earlier. It used to work fine before.
scikit-learn version 1.3.0 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Torch version 2.2.0.dev20231004 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.
Traceback (most recent call last):
File "xyz/ComfyUI/nodes.py", line 1735, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/init.py", line 6, in
from coreml_suite.nodes import CoreMLLoaderUNet, CoreMLSampler, CoreMLModelAdapter
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 11, in
from coreml_suite.lcm.utils import add_lcm_model_options, lcm_patch, is_lcm
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/init.py", line 1, in
from .nodes import COREML_CONVERT_LCMT
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/nodes.py", line 6, in
from coreml_suite.lcm import converter as lcm_converter
File "xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/converter.py", line 9, in
from diffusers import UNet2DConditionModel, LCMScheduler
ImportError: cannot import name 'LCMScheduler' from 'diffusers' (xyz/ComfyUI/venv/lib/python3.11/site-packages/diffusers/init.py)Cannot import xyz/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite module for custom nodes: cannot import name 'LCMScheduler' from 'diffusers' (xyz/ComfyUI/venv/lib/python3.11/site-packages/diffusers/init.py)
The result is that the ComfyUI fails the import.
Any chance you can restore support for nightly PyTorch builds? Thank you.
Device - Base model M1 Air (7 Core GPU, 8GB RAM)
Start ComfyUI with FP16
python main.py --force-fp16
(comfy) ➜ ComfyUI git:(master) ✗ python3 main.py --force-fp16
** ComfyUI start up time: 2023-10-31 16:27:25.261506
Prestartup times for custom nodes:
0.0 seconds: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Manager
Total VRAM 8192 MB, total RAM 8192 MB
Forcing FP16.
/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
warnings.warn(
Set vram state to: SHARED
Device: mps
VAE dtype: torch.float32
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
### Loading: ComfyUI-Manager (V0.36)
### ComfyUI Revision: 1631 [c837a173] | Released on '2023-10-30'
Registered sys.path: ['/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_pycocotools', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_oneformer', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/__init__.py', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_mmpkg', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_midas_repo', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/custom_detectron2', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/controlnet_aux', '/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/src', '/Users/bvndls/ComfyUI/ComfyUI/comfy', '/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/git/ext/gitdb', '/Users/bvndls/ComfyUI/ComfyUI', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload', '/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages', '/var/folders/dq/k_8x61qj6zgg60mnx259fyqh0000gn/T/tmp1wzcun6i', '../..']
/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux/node_wrappers/dwpose.py:26: UserWarning: DWPose: Onnxruntime not found or doesn't come with acceleration providers, switch to OpenCV with CPU device. DWPose might run very slowly
warnings.warn("DWPose: Onnxruntime not found or doesn't come with acceleration providers, switch to OpenCV with CPU device. DWPose might run very slowly")
scikit-learn version 1.3.2 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Torch version 2.2.0.dev20231030 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.
Import times for custom nodes:
0.1 seconds: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Manager
0.3 seconds: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/comfyui_controlnet_aux
0.4 seconds: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite
Starting server
To see the GUI go to: http://127.0.0.1:8188
FETCH DATA from: /Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Manager/extension-node-map.json
got prompt
INFO:coreml_suite.logger:Loading ControlledUnet.mlmodelc to CPU_AND_NE
INFO:python_coreml_stable_diffusion.coreml_model:Loading /Users/bvndls/ComfyUI/ComfyUI/models/unet/ControlledUnet.mlmodelc
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 3.8 seconds.
clip unexpected: ['text_model.embeddings.position_ids']
Requested to load SD1ClipModel
Loading 1 new model
[] []
model_type EPS
adm 0
WARNING:coreml_suite.logger:No latent image provided, using empty tensor.
Requested to load CoreMLModelWrapper
Requested to load ControlNet
Loading 2 new models
0%| | 0/20 [00:00<?, ?it/s]
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "/Users/bvndls/ComfyUI/ComfyUI/execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/Users/bvndls/ComfyUI/ComfyUI/execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/Users/bvndls/ComfyUI/ComfyUI/execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 52, in sample
return super().sample(
File "/Users/bvndls/ComfyUI/ComfyUI/nodes.py", line 1237, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/Users/bvndls/ComfyUI/ComfyUI/nodes.py", line 1207, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/sample.py", line 100, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 728, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler(), sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 633, in sample
samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 589, in sample
samples = getattr(k_diffusion_sampling, "sample_{}".format(sampler_name))(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **extra_options)
File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/k_diffusion/sampling.py", line 580, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 287, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, model_options=model_options, seed=seed)
File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/k_diffusion/external.py", line 129, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/k_diffusion/external.py", line 155, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 275, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 253, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, model_options)
File "/Users/bvndls/ComfyUI/ComfyUI/comfy/samplers.py", line 229, in calc_cond_uncond_batch
output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
File "/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 43, in apply_model
chunked_out = [
File "/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 44, in <listcomp>
self._apply_model(
File "/Users/bvndls/ComfyUI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 83, in _apply_model
np_out = self.diffusion_model(**model_input_kwargs)["noise_pred"]
File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 97, in __call__
self._verify_inputs(**kwargs)
File "/Users/bvndls/ComfyUI/comfy/lib/python3.9/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 90, in _verify_inputs
raise TypeError(
TypeError: Expected shape (2, 320, 64, 64), got (1, 320, 64, 64) for input: additional_residual_0
Prompt executed in 11.72 seconds
Tried with various combinations of UNets, ControlNets, CLIPs, VAEs and images, either throws this error, or generates an image without following the LoadImage connected to ApplyControlNet, exmaple below
Just wondering have you ran any comparisons between running against the MPS (metal) vs using the ANE for inference?
load failed, here is Traceback:
Traceback :
File "/Users/user/ComfyUI/nodes.py", line 1735, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/init.py", line 7, in
from coreml_suite.lcm import (
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/init.py", line 1, in
from .lcm_sampler import CoreMLSamplerLCM
File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/lcm_sampler.py", line 5, in
from diffusers.utils.torch_utils import randn_tensor
ModuleNotFoundError: No module named 'diffusers.utils.torch_utils'
Cannot import /Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite module for custom nodes: No module named 'diffusers.utils.torch_utils'
Ya you knew i was gonna ask :) I know you've got your hands full but hope you can have the time maybe to look into dealing with the Stable Video Diffusion pipeline into coreml, not sure how possible it is given it's so new.
Whenever I try to load a lora when it gets to the sampler I get an error....
INFO:coreml_suite.logger:Batch size: 1
INFO:coreml_suite.logger:Width: 512, Height: 512
INFO:coreml_suite.logger:ControlNet support: False
INFO:coreml_suite.logger:Attention implementation: SPLIT_EINSUM
INFO:coreml_suite.logger:LoRAs used:
INFO:coreml_suite.logger: epiCRealismHelper.safetensors - strength: 1.0
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "/Users/chris/AI/ComfyUI/execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/AI/ComfyUI/execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/AI/ComfyUI/execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 273, in convert
converter.convert(
File "/Users/chris/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/converter.py", line 300, in convert
ref_pipe.set_adapters([adapter_name], adapter_weights=[strength])
File "/Users/chris/AI/ComfyUI/venv/lib/python3.11/site-packages/diffusers/loaders.py", line 2476, in set_adapters
self.unet.set_adapters(adapter_names, adapter_weights)
File "/Users/chris/AI/ComfyUI/venv/lib/python3.11/site-packages/diffusers/loaders.py", line 723, in set_adapters
raise ValueError("PEFT backend is required for `set_adapters()`.")
ValueError: PEFT backend is required for `set_adapters()`.```
Dear Developer,
I am using the 'Load LoRA to use with Core ML' node and the 'Convert Checkpoint to CoreML' node in my workflow. I successfully generated the combined .mlmodelc file (SD model: Cetus-Mix sd_v1-5.safetensor, LoRA model: Anime Lineart / Manga-like.safetensor, Combined model name: cetusMix_Whalefall2_lineArt_V4-16_1x512x768_se_unet.mlmodelc), but the generated images are blurry color blocks in the subsequent steps. I have also noticed that others might have had similar issues. Could you guide me on resolving this?
Thank you for your dedication; this project significantly contributes to Mac users.
Sincerely,
Updated the name because i realized whats causing this, i'm trying to recreate a workflow and the negative prompt is very long, and it seems that this totally destroys things, it seems like the ksampler is limited to 77 tokens
Workflow: easywf.json
I ran updates for comfy and CMLS
But i'm getting the following error
Loading 1 new model
0%| | 0/20 [00:00<?, ?it/s]
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "/Users/user/AI/ComfyUI/execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 65, in sample
return super().sample(
^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/nodes.py", line 1286, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/nodes.py", line 1256, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample
return original_sample(*args, **kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/sample.py", line 100, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 711, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 617, in sample
samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 556, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/k_diffusion/sampling.py", line 655, in sample_dpmpp_3m_sde
denoised = model(x, sigmas[i] * s_in, **extra_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 277, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 267, in forward
return self.apply_model(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 264, in apply_model
out = sampling_function(self.inner_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 252, in sampling_function
cond, uncond = calc_cond_uncond_batch(model, cond, uncond, x, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/samplers.py", line 230, in calc_cond_uncond_batch
output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/comfy/model_base.py", line 73, in apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 23, in __call__
chunked_out = [
^
File "/Users/user/AI/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 25, in <listcomp>
self.coreml_model(**input_kwargs.coreml_kwargs(self.expected_inputs)),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 97, in __call__
self._verify_inputs(**kwargs)
File "/Users/user/AI/ComfyUI/venv/lib/python3.11/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 90, in _verify_inputs
raise TypeError(
TypeError: Expected shape (1, 768, 1, 77), got (1, 768, 1, 385) for input: encoder_hidden_states
Prompt executed in 267.22 secondsuser
Hi! Really like your repo, and am trying to utilize the Core ML Adapter (Experimental) node, and I understand:
This is an experimental node and may not work with all models and nodes. Please use with caution and pay attention to the expected inputs of the model.
When I try to link your CoreML suite into the following repo [workflows], https://github.com/cubiq/ComfyUI_InstantID/blob/main/examples/InstantID_basic.json, I get errors such as:
assert y.shape[0] == x.shape[0] AttributeError: 'NoneType' object has no attribute 'shape'
So my question is, how do I pay attention to the expected inputs of the model. like you suggest?
Any tips!? Thanks in advance!!!
I was getting an error that ComfyUI couldn't import ComfyUI-CoreMLSuite. This may not have been the issue but it fixed it for me:
In:
custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/lcm/unet.py
overrides module not installed. I added that to requirements.txt.
https://pypi.org/project/overrides/
Import times for custom nodes:
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/gcLatentTunnel.py
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/SDXLAspectRatio.py
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/FreeU_Advanced
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI_TiledKSampler
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/wlsh_nodes
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI_Dave_CustomNode
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/mikey_nodes
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ymc-node-suite-comfyui
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI-QualityOfLifeSuit_Omar92
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI_UltimateSDUpscale
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI_Comfyroll_CustomNodes
0.0 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI-LCM
0.1 seconds: /Users/rovo/ComfyUI/custom_nodes/SeargeSDXL
0.1 seconds: /Users/rovo/ComfyUI/custom_nodes/ComfyUI-Manager
0.2 seconds (IMPORT FAILED): /Users/rovo/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite
It seems there's no ModelSamplingDiscreteLCM in comfy_extras.nodes_model_advanced file
Did I do something wrong ?
Thanks
Fresh install, I'm not sure if the torch version warning is actually relevant. On MacOS 14.4.
scikit-learn version 1.4.1.post1 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API. Torch version 2.2.1 has not been tested with coremltools. You may run into unexpected errors. Torch 2.1.0 is the most recent version that has been tested. /Users/francis/Library/Python/3.9/lib/python/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( Traceback (most recent call last): File "/Users/francis/ComfyUI/nodes.py", line 1887, in load_custom_node module_spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/Users/francis/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite-main/__init__.py", line 6, in <module> from coreml_suite.nodes import ( File "/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite-main/coreml_suite/nodes.py", line 9, in <module> from coreml_suite import converter File "/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite-main/coreml_suite/converter.py", line 318, in <module> lora_weights: list[tuple[str | os.PathLike, float]] = None, TypeError: unsupported operand type(s) for |: 'type' and 'ABCMeta'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.