ComfyUI Workflows: Face-Consistent Generation with FaceID, ControlNet and a Trained LoRA

ComfyUI graphs that keep one face consistent across images using a LoRA, IP-Adapter FaceID Plus V2 and an OpenPose ControlNet, plus a Colab notebook that trains the LoRA with kohya_ss.

ComfyUI graph with LoRA, FaceID and ControlNet, and the LoRA training notebook — The real workflow graph: every node, setting and link. LoRA names, photo file names and the prompt were replaced.

The problem

Stable Diffusion changes a face a little on every image. For a character, a model or a brand face that has to look the same across a set, that drift is the whole problem.

The graph

A trained LoRA carries the identity into the model and the text encoder.
IP-Adapter FaceID Plus V2 takes a single reference photo and holds the face on top of the LoRA.
An OpenPose ControlNet takes the pose from a second photo. Its strength is its own setting, so a weak pose hint can be turned down without touching the face.
A KSampler with fixed settings, then VAE decode and save. Variants of the graph cover SD 1.5 and SDXL, with and without the pose hint.

Training the LoRA

A Colab notebook runs kohya_ss sd-scripts with Accelerate in fp16. Captioned images come from Google Drive, and each epoch's LoRA is saved back to it. It starts at 512 pixels with network dim 16 and a cosine schedule, and the notes cover what to change when free Colab runs out of memory.

The same LoRA pipeline was later built into a full product: see FaceSearch.

Message me on the platform where you found this and I will answer there.