[Feature request]: Programmatic use and API about comfyui HOT 22 OPEN

/I could not find an api documentation, so here's what I gleamed from the source code and everyone's comments :

GET

/history
Last queued renders, including all nodes used in the prompt, and all output images.

/history/{UID}
A specific history entry by uid. When calling prompt, the result will contain this uid.

/view?filename={FILENAME}
Access the images from history via this.

/extensions
paths to the javascript files for each extension.

/embeddings
Presumably a list of all embeddings. These aren't available as inputs in any Node, unlike models (see OTHERS).

/object_info
List of all types of nodes that are available. (Not the actual node instances, just the node types.)
This exposes a lot of important enumerable values in the form of input types.

/queue
What's happening or queued at the moment. Surprisingly comprehensive.

/system_stats
os, python version, torch devices (like gpus) and vram used.

POST

/prompt
https://github.com/comfyanonymous/ComfyUI/blob/master/script_examples/basic_api_example.py

The json data payload must be stored under the name "prompt". In JS, it would look like so :

const res = await axios.post('http://127.0.0.1:8188/prompt',{
	prompt:json_data_object	
});

You can get an example of the json_data_object by enabling Dev Mode in the ComfyUI settings, and then clicking the newly added export button.

/interrupt
Interrupts the execution of the running prompt and starts the next one in the queue.

/queue
/history
In POST, these routes should accept the payloads clear:true and delete:number[] .

/upload/image
/upload/mask
This uploads an image and returns a short path under which the image can be referred to in comfyui.
Takes payload 'image' in body.

SYNTAX

The following typescript definitions are incomplete and may be wrong entirely.
Please leave a comment if you spot errors or if you have improvements.

// This is what apis like "prompt" expect.
export type JSON_ComfyGraph = {
    [uid: string|number]: JSON_ComfyNode
}

// A node instance.
export type JSON_ComfyNode = {
    // The type of the node as found in 'GET /object_info' by key
    class_type: string;
    // All incoming connections
    inputs: {
        [input_name: string]: JSON_ValueRef;
    }
}

// Any value that can is passed into a ComfyNode
// This can be either an atomic value, like a string or a number, or a reference to another node's output.
export type JSON_ValueRef = [target_node_uid: string, output_socket_index: number] | string | number | boolean


// All possible input/output types. This type is incomplete.
// string[] represents an enum dropdown.
export type JSON_ComfyPrimitive = "MODEL" | "CLIP" | "STYLE_MODEL" | "CLIP_VISION_OUTPUT" | "CONTROL_NET" | "UPSCALE_MODEL" | "CLIP_VISION" | "VAE" | "LATENT" | "INT" | "FLOAT" | "IMAGE" | "TEXT" | "CONDITIONING" | "EXTRA_PNGINFO" | "MASK" | string[];

// Some types accept options like restrictions or special ui operations. So they are wrapped in an array of type + options
export type JSON_ComfyPrimitiveDeclaration = 
// An empty array means the name of the output/input is the same as the type. Case insensitive. Eg an output named 'latent' defaults to type "LATENT".
[] 
| [JSON_ComfyPrimitive]
| [
    "INT" | "FLOAT",
    {
        default?: number,
        min?: number,
        max?: number,
        step?: number,
        round?: number
    }
]
| [
    "TEXT"|string[],
    {
        multiline?:boolean,
        default?:string
    }
]
| [
    "IMAGE"|string[],
    {
        image_upload?:boolean,
        default?:string
    }
]

// This is what apis like "object_info" return.
export type JSON_ComfyNodeTypes = {
    [key: string]: {
        // This is what creates the submenus in the context menu.
        category: string,
        // Verbal description of the node for use in the ui.
        description: string,
        // The name which should be displayed in the editor.
        display_name: string,
        input: {
            // List of all input types
            required: {
                [name: string]: JSON_ComfyPrimitiveDeclaration
            },
            hidden: {
                [name: string]: JSON_ComfyPrimitiveDeclaration
            }
        },
        // Unique name
        name: string,
        // Unique names of all the outputs
        output: string[],
        // Is the output a list?
        output_is_list: boolean[],
        // Labels of all the outputs
        output_name: string[],
        // Is this node an output? Useful for evaluating if a graph is valid.
        output_node: boolean;
    }
}

// This is what apis like "history" return.
export type JSON_History = {
    [uid: string]: JSON_HistoryEntry;
}

export type JSON_HistoryEntry = {
    prompt: [id_maybe: number, uid:string, list_of_used_nodes:JSON_ComfyGraph, no_idea:any, no_idea2:[string]];
    outputs: {
        images: JSON_ImageReference[];
    }[];
}

export type JSON_ImageReference = {
    filename: string;
    subfolder: string;
    type: "output";
}

OTHER

You can retrieve the list of all available models from the CheckpointLoader object.
GET /object_info
Then CheckpointLoader->input->required->ckpt_name->0 .

from comfyui.

if you update to the latest you can "enable dev mode options" in the settings (gear beside the "Queue Size: ") this will enable a button on the UI to save workflows in api format.

You can then use these api jsons in these script examples: https://github.com/comfyanonymous/ComfyUI/tree/master/script_examples

from comfyui.

Is there a guide on how the API works

from comfyui.

I took a look at how the websocket service works, and created a simple class to use the already existing API. It facilitated my workflow a lot and maybe some of you can find it useful.

https://github.com/fidecastro/ComfyUI-serverless/

from comfyui.

See: https://github.com/comfyanonymous/ComfyUI/blob/master/script_examples/basic_api_example.py

from comfyui.

Is there any way to clear the output folder with API?

from comfyui.

@comfyanonymous is there a way to turn off nodes other than setting strength it doesn't seem like the api version of a workflow allows mode modification

You can create xml workflows on fly and pass them to /promp with POST. Also I've found a 3rd article with API description based on code analysis.

from comfyui.

/I could not find an api documentation, so here's what I gleamed from the source code and everyone's comments :

GET

/history Last queued renders, including all nodes used in the prompt, and all output images.

/history/{UID} A specific history entry by uid. When calling prompt, the result will contain this uid.

/view?filename={FILENAME} Access the images from history via this.

/extensions paths to the javascript files for each extension.

/embeddings Presumably a list of all embeddings. These aren't available as inputs in any Node, unlike models (see OTHERS).

/object_info List of all types of nodes that are available. (Not the actual node instances, just the node types.) This exposes a lot of important enumerable values in the form of input types.

/queue What's happening or queued at the moment. Surprisingly comprehensive.

/system_stats os, python version, torch devices (like gpus) and vram used.

POST

/prompt https://github.com/comfyanonymous/ComfyUI/blob/master/script_examples/basic_api_example.py

The json data payload must be stored under the name "prompt". In JS, it would look like so :

const res = await axios.post('http://127.0.0.1:8188/prompt',{
	prompt:json_data_object	
});

You can get an example of the json_data_object by enabling Dev Mode in the ComfyUI settings, and then clicking the newly added export button.

/interrupt Interrupts the execution of the running prompt and starts the next one in the queue.

/queue /history In POST, these routes should accept the payloads clear:true and delete:number[] .

/upload/image /upload/mask This uploads an image and returns a short path under which the image can be referred to in comfyui. Takes payload 'image' in body.

SYNTAX

The following typescript definitions are incomplete and may be wrong entirely. Please leave a comment if you spot errors or if you have improvements.

// This is what apis like "prompt" expect.
export type JSON_ComfyGraph = {
    [uid: string|number]: JSON_ComfyNode
}

// A node instance.
export type JSON_ComfyNode = {
    // The type of the node as found in 'GET /object_info' by key
    class_type: string;
    // All incoming connections
    inputs: {
        [input_name: string]: JSON_ValueRef;
    }
}

// Any value that can is passed into a ComfyNode
// This can be either an atomic value, like a string or a number, or a reference to another node's output.
export type JSON_ValueRef = [target_node_uid: string, output_socket_index: number] | string | number | boolean


// All possible input/output types. This type is incomplete.
// string[] represents an enum dropdown.
export type JSON_ComfyPrimitive = "MODEL" | "CLIP" | "STYLE_MODEL" | "CLIP_VISION_OUTPUT" | "CONTROL_NET" | "UPSCALE_MODEL" | "CLIP_VISION" | "VAE" | "LATENT" | "INT" | "FLOAT" | "IMAGE" | "TEXT" | "CONDITIONING" | "EXTRA_PNGINFO" | "MASK" | string[];

// Some types accept options like restrictions or special ui operations. So they are wrapped in an array of type + options
export type JSON_ComfyPrimitiveDeclaration = 
// An empty array means the name of the output/input is the same as the type. Case insensitive. Eg an output named 'latent' defaults to type "LATENT".
[] 
| [JSON_ComfyPrimitive]
| [
    "INT" | "FLOAT",
    {
        default?: number,
        min?: number,
        max?: number,
        step?: number,
        round?: number
    }
]
| [
    "TEXT"|string[],
    {
        multiline?:boolean,
        default?:string
    }
]
| [
    "IMAGE"|string[],
    {
        image_upload?:boolean,
        default?:string
    }
]

// This is what apis like "object_info" return.
export type JSON_ComfyNodeTypes = {
    [key: string]: {
        // This is what creates the submenus in the context menu.
        category: string,
        // Verbal description of the node for use in the ui.
        description: string,
        // The name which should be displayed in the editor.
        display_name: string,
        input: {
            // List of all input types
            required: {
                [name: string]: JSON_ComfyPrimitiveDeclaration
            },
            hidden: {
                [name: string]: JSON_ComfyPrimitiveDeclaration
            }
        },
        // Unique name
        name: string,
        // Unique names of all the outputs
        output: string[],
        // Is the output a list?
        output_is_list: boolean[],
        // Labels of all the outputs
        output_name: string[],
        // Is this node an output? Useful for evaluating if a graph is valid.
        output_node: boolean;
    }
}

// This is what apis like "history" return.
export type JSON_History = {
    [uid: string]: JSON_HistoryEntry;
}

export type JSON_HistoryEntry = {
    prompt: [id_maybe: number, uid:string, list_of_used_nodes:JSON_ComfyGraph, no_idea:any, no_idea2:[string]];
    outputs: {
        images: JSON_ImageReference[];
    }[];
}

export type JSON_ImageReference = {
    filename: string;
    subfolder: string;
    type: "output";
}

OTHER

You can retrieve the list of all available models from the CheckpointLoader object. GET /object_info Then CheckpointLoader->input->required->ckpt_name->0 .

Thanks, this should be added to wiki)

from comfyui.

I also want to use workflow struct my special API,because I can't process batch images now.It will make the workflow more powerful.

from comfyui.

That's a good question, and I'd like to implement this functionality by calling the ComfyUI workflow directly from an external application, which eliminates the hassle of using the stable diffusion api.

from comfyui.

I would also love to have an API. If possible also an OpenApi/Swagger definition would be awesome, so you could generate a client in different languages.

from comfyui.

+1

from comfyui.

Hi, Thanks for adding the "API Format" option it is super helpful !

from comfyui.

I'm Stucking in here! #1083 (comment)

Can you give me a tought?

from comfyui.

I took a look at how the websocket service works, and created a simple class to use the already existing API. It facilitated my workflow a lot and maybe some of you can find it useful.

https://github.com/fidecastro/ComfyUI-serverless/

It's a great work

from comfyui.

Could anyone give me any functional example of the use of the API?
What is complete and that has an example embedded as those of the Save in Api Format?
I can't run the Basic Example:/

from comfyui.

si actualizas a lo último puedes "solar las opciones de modo dev" en la configuración (envejecer junto al "Queue Size: ") esto permitirá un botón en la interfaz de usuario para guardar flujos de trabajo en formato api.

A continuación puede utilizar estos api jsons en estos ejemplos de script: https://github.com/comfyanonymous/ComfyU/tree/master/script-examples

Could anyone give me any functional example of the use of the API?
What is complete and that has an example embedded as those of the Save in Api Format?
I can't run the Basic Example:/

from comfyui.

For sample usage you may check this small project I've created.
https://github.com/rsandagon/comfyui-batch-image-generation

For websocket handling, check here

For API endpoint, check here

You need to have the same client_id for both API and websocket to get proper websocket updates on the workflow.

from comfyui.

you can also look at https://github.com/rvion/CushyStudio

It is probably the biggest custom frontend around.

I maintain the most comprehensive set of typings possible, along a higher level type-safe typescript ComfyUI SDK to build workflows with all possible goodies you can imagine.

CushyStudio also generates a whole type-safe SDK for your specific ComfyUI setup, with every custom node, and even model you have installed

• https://gist.github.com/rvion/f68635427c7d5e9b0846121e4d9e87a7#file-global-d-ts-L6789
• https://gist.github.com/rvion/f68635427c7d5e9b0846121e4d9e87a7#file-global-d-ts-L5324
• https://gist.github.com/rvion/f68635427c7d5e9b0846121e4d9e87a7#file-global-d-ts-L5190

writing workflows and running them is super smooth.

Cushy also feature a scripting engine with hot reload, and a custom gradio-like framework (simpler and type-safe) to build custom apps with UI on top of the ComfyUI SDK.

Only reason it's not popular yet is that it's not released yet (so no one knows about it), and that it probably has a few quirks here and there. but maybe you'll have fun trying it out. feel free to reach out on discord if you need help:)

from comfyui.

Cushy also includes higher level API / typings for comfy manager, and host management too, (and other non-comfy things that works well with ComfyUI, like a full programmatic image building API to build masks, etc)

from comfyui.

Finally I found a page, which explained how to use it in Python and it worked for me
https://medium.com/@yushantripleseven/comfyui-websockets-api-part-1-618175802d5a

https://medium.com/@yushantripleseven/comfyui-using-the-api-part-2-daac17fd2727

from comfyui.

@comfyanonymous is there a way to turn off nodes other than setting strength it doesn't seem like the api version of a workflow allows mode modification

from comfyui.