CatVTON-FLUX: Combining CATVTON and Flux Fill Repair Models for Virtual Try-On Solutions

General Introduction

CatVTON-FLUX is an advanced virtual try-on solution that combines CATVTON (Concatenation Is All You Need for Virtual Try-On with Diffusion Models) and Flux Populated repair models for realistic and accurate garment transfers. Developed by nftblackmagic and inspired by In-Context LoRA for prompt engineering, the project aims to provide a high-quality virtual fitting experience through the use of diffusion modeling and LoRA (In-Context LoRA for prompt engineering) techniques. The platform allows users to virtually try on Hugging Face and experience the latest model weights and performance improvements.

Online experience: https://huggingface.co/spaces/xiaozaa/catvton-flux-try-on

Function List

• Virtual Fitting: Realistic garment transfers using CATVTON and Flux models.
• LoRA Weights Support: Provides LoRA weights to improve model performance.
• Gradio Demo: Demonstrate virtual try-on results with the Gradio platform.
• Hugging Face Support: Run CATVTON-FLUX-TRY-ON on the Hugging Face.
• Model Weights Update: Regularly update model weights to improve performance and detail representation.

Using Help

Installation process

1. Ensure that the runtime environment has >= 40GB of VRAM (80GB GPU recommended).
2. Create and activate a Python virtual environment:

   conda create -n flux python=3.10
conda activate flux

3. Install the required dependencies:

   pip install -r requirements.txt
huggingface-cli login

Usage Process

Run the LoRA version of the fitting

1. Prepare input images, masks, and costume images.
2. Run the following command to try it on:

   python tryon_inference_lora.py \
--image ./example/person/00008_00.jpg \
--mask ./example/person/00008_00_mask.png \
--garment ./example/garment/00034_00.jpg \
--seed 4096 \
--output_tryon test_lora.png \
--steps 30

Run a trial without LoRA version

1. Prepare input images, masks, and costume images.
2. Run the following command to try it on:

   python tryon_inference.py \
--image ./example/person/00008_00.jpg \
--mask ./example/person/00008_00_mask.png \
--garment ./example/garment/00034_00.jpg \
--seed 42 \
--output_tryon test.png \
--steps 30

Launching the Gradio Demo

1. Run the following command to start the Gradio demo with LoRA weights:

   python app.py

2. Run the following command to start the Gradio demo without LoRA weights:

   python app_no_lora.py

Detailed Operation Procedure

1. Preparing the input image: Ensure that the input image, mask and garment image are the same size, the recommended size is 576x768.
2. Running the test penetration command: Select the LoRA version or the LoRA-free version of the Try On command as required, adjusting parameters such as seed and steps for optimal results.
3. View Results: The fitting results will be saved in the specified output path, and the user can view and evaluate the fitting results.
4. Gradio Interface Interaction: Through the Gradio interface, users can upload their own images and garments to experience a virtual fitting in real time.