What it does
Send a face photo and a prompt. A worker uploads the photo to ComfyUI, queues a FaceID graph (IPAdapter FaceID, optional LoRA, KSampler), waits while the GPU works, downloads the portrait and stores it. The client polls the job and fetches the image.
Where it came from
In my FaceSearch SaaS (face search, LoRA training, virtual try-on), image generation ran as a Celery task: submit to the AI engine, then poll until done, all inside one function. A separate Python script built the ComfyUI FaceID graph and waited on its history. If the worker died while waiting, the progress and the link to the running generation were lost, and the next attempt started over with a fresh submit: the same image on the GPU twice.
How genflow keeps a job alive
- Steps with checkpoints. Upload, submit, wait, download. Progress is saved in Postgres after each step, and a new attempt starts after the last saved one.
- Leases. Workers claim jobs with FOR UPDATE SKIP LOCKED and renew a lease while ComfyUI works. If a worker dies, the lease runs out and another worker takes over. Every write checks the lease owner, so a frozen worker that wakes up cannot overwrite the new owner's progress.
- A submit that is safe to repeat. The ComfyUI prompt id is the job id. Before queueing, the worker asks ComfyUI whether that id is already queued, running or finished, and only queues it if ComfyUI has never seen it. A resumed job re-attaches to the run already on the GPU. If ComfyUI itself restarted and lost the prompt, the same id is queued again, a limited number of times.
Also built in
- Retries with exponential backoff and jitter, with a cap on attempts.
- Errors sorted into temporary (ComfyUI busy or unreachable: retry) and permanent (bad graph, missing model, CUDA out of memory: fail at once).
- Backpressure: past a set queue depth the API answers 429 with Retry-After, and each worker process limits how many prompts it has on the GPU.
- Cancel, graceful shutdown, and API and workers that can run on separate machines.
Tested
Integration tests run against a real Postgres and a fake ComfyUI that speaks the same API and can fail on purpose. A worker crashes right after queueing; a second worker cannot touch the job while the first lease is valid, then resumes it, and the prompt reaches ComfyUI exactly once. ComfyUI drops its queue mid-run and the job still finishes. Two server errors are retried; an out-of-memory error fails on the first attempt; cancel stops a running job; the queue refuses work past its limit. Sixty jobs on eight workers each ran exactly once. Everything passes under the Go race detector, three runs in a row.
Stack
Go 1.27, Postgres with pgx, the ComfyUI HTTP API, Docker Compose for a GPU-free demo. Code: github.com/shabeebhasan/genflow-go.
What it does not claim
The tests use a fake ComfyUI, not a GPU. The graph mirrors the FaceID graph my Python script builds for ComfyUI, but this project has not been run on a GPU yet. It is my own build, not a client system in production.