ComfyUI Wrapper API
We have included 3 workflows in the code, shown below. Each workflow is a class inherited from the SDXLBasicWorkflow class. This means it is trivial to add new workflows as well as new model support, simply by adding a new class in the workflows file. We also have the flexibility to adjust all the hyperparameters in each workflow to get the best possible result. Everything from the model used to the positive and negative prompts to the scheduler can be adjusted on a workflow by workflow basis.
In the screenshot below you can see the parameters we found to be optimal in our testing.
Workflows
Our workflows each use their own SDXL-based checkpoints, each with custom parameters such as samplers and resolution based on what leads to the best results from our testing. A simple comparison of the workflows is shown in the table below.
Prompt Optimization
According to Oscar Mañas et. al in arXiv:2403.17804, using a large language model to optimize text to image prompt lead to an increase of result consistency by up to 25%, while not suffering degradation in image quality. We employ a similar approach in our prompt optimization and have observed notable improvements in generated images in our testing.
Below are some examples of input -> optimized prompts from our optimizer.
A cat with wings
A majestic cat with large, feathery wings outstretched, soft textures and gentle shading, warm sunlight casting a subtle glow, whimsical illustration style, dreamy atmosphere, close-up view with delicate details and expressive eyes
A man
A man stands confidently, neutral expression, simple background, soft natural light, realistic portrait, straight-on composition, shallow depth of field, subtle texture and tone in a photorealistic style
A purple potion with a landscape inside
A delicate glass vial contains a rich, swirling purple potion, a tiny intricate landscape visible inside, rolling hills and towering trees suspended in the liquid, warm golden light emanates from within, whimsical illustration style, close-up view, soft focus
Automated workflow switching
Each workflow is optimized for a specific image style (Realistic, Fantasy, Anime).
Large language models like Llama have a high level understanding of words and their associations, this allows us to make use of them to do smart, automatic workflow switching depending on the prompt.
If the user mentions the style they want in their prompt it will select that style to be used.
If no style is mentioned, it uses context from the prompt to select the most relevant workflow. If the prompt describes a realistic scene, the realistic workflow will be chosen. If the prompt contains an anime character, the anime workflow is chosen.
Below are some examples of input -> selected workflow from our optimizer.
A titan from attack on titan
Anime
A flying pig
Fantasy
A realistic photo of a flying pig
Realistic
A man
Realistic
Spike Spiegel
Anime
Last updated