Giter Club home page Giter Club logo

comfyui_noise's Introduction

ComfyUI Noise

This repo contains 6 nodes for ComfyUI that allows for more control and flexibility over the noise. This allows e.g. for workflows with small variations to generations or finding the accompanying noise to some input image and prompt.

Nodes

Noisy Latent Image:

This node lets you generate noise, you can find this node under latent>noise and it the following settings:

  • source: where to generate the noise, currently supports GPU and CPU.
  • seed: the noise seed.
  • width: image width.
  • height: image height.
  • batch_size: batch size.

Duplicate Batch Index:

The functionality of this node has been moved to core, please use: Latent>Batch>Repeat Latent Batch and Latent>Batch>Latent From Batch instead.

This node lets you duplicate a certain sample in the batch, this can be used to duplicate e.g. encoded images but also noise generated from the node listed above. You can find this node under latent and it has the following settings:

  • latents: the latents.
  • batch_index: which sample in the latents to duplicate.
  • batch_size: the new batch size, (i.e. how many times to duplicate the sample).

Slerp Latents:

This node lets you mix two latents together. Both of the input latents must share the same dimensions or the node will ignore the mix factor and instead output the top slot. When it comes to other things attached to the latents such as e.g. masks, only those of the top slot are passed on. You can find this node under latent and it comes with the following inputs:

  • latents1: first batch of latents.
  • latents2: second batch of latents. This input is optional.
  • mask: determines where in the latents to slerp. This input is optional
  • factor: how much of the second batch of latents should be slerped into the first.

Get Sigma:

This node can be used to calculate the amount of noise a sampler expects when it starts denoising. You can find this node under latent>noise and it comes with the following inputs and settings:

  • model: The model for which to calculate the sigma.
  • sampler_name: the name of the sampler for which to calculate the sigma.
  • scheduler: the type of schedule used in the sampler
  • steps: the total number of steps in the schedule
  • start_at_step: the start step of the sampler, i.e. how much noise it expects in the input image
  • end_at_step: the current end step of the previous sampler, i.e. how much noise already is in the image.

Most of the time you'd simply want to keep start_at_step at zero, and end_at_step at steps, but if you'd want to re-inject some noise in between two samplers, e.g. one sampler that denoises from 0 to 15, and a second that denoises from 10 to 20, you'd want to use a start_at_step 10 and an end_at_step of 15. So that the image we get, which is at step 15, can be noised back down to step 10, so the second sampler can bring it to 20. Take note that the Advanced Ksampler has a settings for add_noise and return_with_leftover_noise which when working with these nodes we both want to have disabled.

Inject Noise:

This node lets you actually inject the noise into an image latent, you can find this node under latent>noise and it comes with the following inputs:

  • latents: The latents to inject the noise into.
  • noise: The noise. This input is optional
  • mask: determines where to inject noise. This input is optional
  • strength: The strength of the noise. Note that we can use the node above to calculate for us an appropriate strength value.

Unsampler:

This node does the reverse of a sampler. It calculates the noise that would generate the image given the model and the prompt. You can find this node under sampling and it takes the following inputs and settings:

  • model: The model to target.
  • steps: number of steps to noise.
  • end_step: to what step to travel back to.
  • cfg: classifier free guidance scale.
  • sampler_name: The name of the sampling technique to use.
  • scheduler: The type of schedule to use.
  • normalize: whether to normalize the noise before output. Useful when passing it on to an Inject Noise node which expects normalizes noise.
  • positive: Positive prompt.
  • negative: Negative prompt.
  • latent_image: The image to renoise.

When trying to reconstruct the target image as faithful as possible this works best if both the unsampler and sampler use a cfg scale close to 1.0 and similar number of steps. But it is fun and worth it to play around with these settings to get a better intuition of the results. This node let's you do similar things the A1111 img2img alternative script does

Examples

here are some examples that show how to use the nodes above. Workflows to these examples can be found in the example_workflow folder.

generating variations

screenshot of a workflow that demos generating small variations to a given seed

To create small variations to a given generation we can do the following: We generate the noise of the seed that we're interested using a Noisy Latent Image node, we then create an entire batch of these with a Duplicate Batch Index node. Note that if we were doing this for img2img we can use this same node to duplicate the image latents. Next we generate some more noise, but this time we generate a batch of noise rather than a single sample. We then Slerp this newly created noise into the other one with a Slerp Latents node. To figure out the required strength for injecting this noise we use a Get Sigma node. And finally we inject the slerped noise into a batch of empty latents with a Inject Noise node. Take note that we use an advanced Ksampler with the add_noise setting disabled

"unsampling"

screenshot of a workflow that demos generating small variations to a given seed

To get the noise that recreates a certain image, we first load an image. Then we use the Unsampler node with a low cfg value. To check if this is working we then take the resulting noise and feed it back into an advanced ksampler with the add_noise setting disabled, and a cfg of 1.0.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.