ComfyUI serverless worker broken after adding new nodes to network volume about comfyui HOT 6 CLOSED

It should, you're right.

Are you running with variable WORKSPACE_SYNC=true for the serverless worker?

I suspect the problem is that the worker is trying to use the on-image python environment rather than the synced workspace version. That shouldn't happen so I'll look into it asap.

I'm not a fan of the workspace sync though because RunPod disks are slow and it adds more runtime completely Vs building the image with required nodes and models.

from comfyui.

Hi. Thanks for your reply.
I didn't try to turn WORKSPACE_SYNC=true on the serverless worker yet. Did that and it works!

I set it all up mostly following this tutorial and was assuming that WORKSPACE_SYNC has to be always off on the serverless template: https://medium.com/@iknowkungfu/setup-complex-ai-image-workflows-at-scale-using-a-fleet-of-cloud-gpus-and-comfyui-1ac9a6c4b0b3

It worked with setting up on a regular pod with sync=on for the initial setup and turning off for serverless but broke on updating the backend.

I agree that ideally I would build images including everything but since I run quite complicated workflows with many custom nodes this largely exceeds my Docker capabilities.

Thanks a lot for your great work!

from comfyui.

However, I am gettin a lot of 'Server not ready after timeout (30s)' errors now.
Probably will have to look into building a custom Docker image after all.

from comfyui.

The timeout is coming from the current mechanism for switching the environment from /opt to /workspace - It's a bad design decision that I plan to work around.

Currently the /opt/micromamba directory is removed and replaced with a symlink (to /workspace/environments/...), but this can take some time on slow disks. The better solution is to have the environment always set by symlinks as there will not be a delete step.

My fault but I'll fix it as soon as I can - it's still better to build the image though because networked disks aren't great for this usecase.

Really, syncing part of image so that it can be modified throws away much of the benefits that docker brings. I understand why users want to do it, I just think it introduces more points of failure which I'd never want in a production application. I will add warnings to the documentation

from comfyui.

Thanks.
I started building a custom Docker image based on ai-dock.
I made a little script, which analyzes Comfy workflows and lists the custom node Githubs.

Added them to .../layer1/init.sh

NODES=(
    "https://github.com/ltdrdata/ComfyUI-Manager"
    "https://github.com/Extraltodeus/ComfyUI-AutomaticCFG",
    "https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved",
    "https://github.com/Seedsa/Fooocus_Nodes",
    "https://github.com/chrisgoringe/cg-use-everywhere",
)

However, the custom nodes seem to not be available to the worker.
Do I need to remove WORKSPACE=/runpod-volume from the environment variables? Or still copy the nodes somewhere into the docker image?

from comfyui.

No longer supporting old builds due to project direction changes. New API wrapper will be documented soon for all serverless use cases

from comfyui.