It has been revealed that FLUX.2 is the recent series of image generation models from Black Forest Labs, preceded by the Flux.1 series. It is an entirely new model with a new architecture and pre-training done from scratch! In this post, we discuss the key changes introduced in FLUX.2, performing inference with it under various setups, and LoRA fine-tuning.
🚨 FLUX.2 is not meant to be a drop-in replacement of FLUX.1, but a fresh image generation and editing model.
FLUX.2 can be used for both image-guided and text-guided image generation. Furthermore, it can take multiple images as reference inputs, while producing the final output image. Below, we briefly discuss the key changes introduced in FLUX.2.
In a recent development, first, instead of two text encoders as in Flux.1, it uses a single text encoder — Mistral Small 3.1. Using a single text encoder greatly simplifies the process of computing prompt embeddings. The pipeline allows for a max_sequence_length of 512. Instead of using a single-layer output for the prompt embedding, FLUX.2 stacks outputs from intermediate layers, which have been known to be more beneficial.
FLUX.2 follows the same general multimodel diffusion transformer (MM-DiT) + parallel DiT architecture as Flux.1. As a refresher, MM-DiT blocks first process the image latents and conditioning text in separate streams, only joining the two together for the attention operation, and are thus referred to as “double-stream” blocks. The parallel blocks then operate on the concatenated image and text streams and can be regarded as “single-stream” blocks.
The key DiT changes from Flux.1 to FLUX.2 are as follows:.
Time and guidance information (in the form of AdaLayerNorm-Zero modulation parameters) is shared across all double-stream and single-stream transformer blocks, respectively, rather than having individual modulation parameters for each block as in Flux.1.
None of the layers in the model use bias parameters. In particular, neither the attention nor feedforward (FF) sub-blocks of either transformer block use bias parameters in any of their layers.
In Flux.1, the single-stream transformer blocks fused the attention output projection with the FF output projection. FLUX.2 single-stream blocks also fuse the attention QKV projections with the FF input projection, creating a fully parallel transformer block:.
Note that compared to the ViT-22B block depicted above, FLUX.2 uses a SwiGLU-style MLP activation rather than a GELU activation (and also doesn’t use bias parameters).
Industry observers note that A larger proportion of the transformer blocks in FLUX.2 are single-stream blocks (8 double-stream blocks to 48 single-stream blocks, compared to 19/38 for Flux.1). This also means that single-stream blocks make up a larger proportion of the DiT parameters: Flux.1[dev]-12B has ~54% of its total parameters in the double-stream blocks, whereas FLUX.2[dev]-32B has ~24% of its parameters in the double-stream blocks (and ~73% in the single-stream blocks).
Recent findings suggest that FLUX.2 uses a larger DiT and Mistral3 Small as its text encoder. When used together without any kind of offloading, the inference takes more than 80GB VRAM. In the following sections, we show how to perform inference with FLUX.2 in more accessible ways, under various system-level constraints.
Before you try out the following code snippets, make sure you have installed diffusers from main and have run hf auth login.
Industry observers note that the above code snippet was tested on an H100, and it isn’t sufficient to run inference on it without CPU offloading. With CPU offloading enabled, this setup takes ~62GB to run.
People who have access to Hopper-series GPUs can take advantage of Flash Attention 3 to speed up inference:.
You can check out the supported attention backends (we have many!) here.
Using bitsandbytes, we can load the transformer and text encoder models in 4-bit, allowing owners of 24GB GPUs to use the model locally. You can run this snippet on a GPU with ~20 GB of free VRAM.
Notice that we're using a repository that contains the NF4-quantized versions of the FLUX.2 DiT and the Mistral text encoder.
Due to the modular design of a Diffusers pipeline, we can isolate modules and work with them in sequence. We decouple the text encoder and deploy it to an Inference Endpoint. This helps us with freeing up the VRAM usage for the DiT and VAE only.
⚠️ To use the remote text encoder, you need to have a valid token. If you are already authenticated, no further action is needed.
It has been revealed that the example below uses a combination of local and remote inference. Additionally, we quantize the DiT with NF4 quantization through bitsandbytes.
You can run this snippet on a GPU with 18 GB of VRAM:.
Sources indicate that for GPUs with even lower VRAM, we have group_offloading, which allows GPUs with as little as 8GB of free VRAM to use this model. However, you'll need 32GB of free RAM. Alternatively, if you're willing to sacrifice some speed, you can set low_cpu_mem_usage=True to reduce the RAM requirement to just 10GB.
You can check out other supported quantization backends here and other memory-saving techniques here.
To check how different quantizations affect an image, you can play with the playground below or access it as standlone in the FLUX.2 Quantization experiments Space.
According to recent reports, FLUX.2 supports using multiple images as inputs, allowing you to use up to 10 images. However, keep in mind that each additional image will require more VRAM. You can reference the images by index (e.g., image 1, image 2) or by natural language (e.g., the kangaroo, the turtle). For optimal results, the best approach is to use a combination of both methods.
It has been revealed that FLUX.2 supports advanced prompting techniques like structured JSON prompting, precise hex color control, and multi-reference image editing. Aside for the added control, this also allows for flexibility in changing specific attributes while maintaining others overall the same.For example, let's start with this json as the base schema (taken from the official FLUX.2 prompting guide):.
Building up on that, let's turn it into a prompt for a shot of a good old fashion walkman on a carpet (simply pass this prompt to your chosen diffusers inference example from above):.
Now, let's change the color of the carpet to a specific teal-blue shade (#367588) and add wired headphones plugged into the walkman:.
The carpet color now matches the hex code provided, and the headphones have been with small changes to the overall scene.
Check out the official prompting guide for more examples and details.
It has been revealed that being both a text-to-image and an image-to-image model, FLUX.2 makes the perfect fine-tuning candidate for many use-cases! However, as inference alone takes more than 80GB of VRAM, LoRA fine-tuning is even more challenging to run on consumer GPUs. To squeeze in as much memory saving as we can, we utilize some of the inference optimizations described above for training as well, together with shared memory saving techniques, to substantially reduce memory consumption. To train it, you can use either the diffusers code below or Ostris' AI Toolkit.
We provide both text-to-image and image-to-image training scripts, for the purpose of this blog will focus on a text-to-image training example.
Sources indicate that many of these techniques complement each other and can be used together to reduce memory consumption further. However, some techniques may be mutually exclusive, so be sure to check before launching a training run.
Please make sure to check out the README for prerequisites before starting training.
It has been revealed that for this example, we’ll use multimodalart/1920-raider-waite-tarot-public-domain dataset with the following configuration using FP8 training. Feel free to experiment more with the hyper-parameters and share your results 🤗.
The left image was generated using the pre-trained FLUX.2 model, and the right image was produced the LoRA.
In case your hardware isn’t compatible with FP8 training, you can use QLoRA with bitsandbytes. You first need to define a config.json file like so:.
And then pass its path to --bnb_quantization_config_path:.
Sources indicate that amazing! great work! 👏is there a support for multi-gpus? (device_map=auto).
You should be able to incorporate that in different forms. Check this out:https://huggingface.co/docs/diffusers/main/en/training/distributed_inference.
Hi, can you tell me a bit about your motivations behind omitting all bias parameters in network architecture? Thanks!
That's a question for the Black Forest Labs team, not us.
Industry observers note that amazing work! Can you tell me when the depth-maps model will be released?Has anyone already tried giving a depth map as a normal image? How does the model behave?
https://github.com/huggingface/blog/blob/main/flux-2.md?plain=1#L283 causes.
Probably an installation error?
pip install git+https://github.com/huggingface/diffusers -U should help you with this.
Hi, How can I deploy a text encoder privately?
Hi, you can read this nice repo with the process that @ariG23498 made.
https://github.com/ayttop/xflux2gpu/blob/main/xxxxxx%20(1).ipynb.
https://huggingface.co/docs/diffusers/main/en/api/parallel.
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference.
import torchfrom transformers import Mistral3ForConditionalGeneration.
from diffusers import Flux2Pipeline, Flux2Transformer2DModel.
repo_id = "diffusers/FLUX.2-dev-bnb-4bit"device = "cuda:0"torch_dtype = torch.bfloat16.
transformer = Flux2Transformer2DModel.from_pretrained( repo_id, subfolder="transformer", torch_dtype=torch_dtype, device_map="cpu")text_encoder = Mistral3ForConditionalGeneration.from_pretrained( repo_id, subfolder="text_encoder", dtype=torch_dtype, device_map="cpu").
pipe = Flux2Pipeline.from_pretrained( repo_id, transformer=transformer, text_encoder=text_encoder, torch_dtype=torch_dtype)pipe.enable_model_cpu_offload().
According to recent reports, prompt = "Realistic macro photograph of a hermit crab using a soda can as its shell, partially emerging from the can, captured with sharp detail and natural colors, on a sunlit beach with soft shadows and a shallow depth of field, with blurred ocean waves in the background. The can has the text BFL Diffusers on it and it has a color gradient that start with #FF5733 at the top and transitions to #33FF57 at the bottom.".
image = pipe( prompt=prompt, generator=torch.Generator(device=device).manual_seed(42), num_inference_steps=50, # 28 is a good trade-off guidance_scale=4,).images[0].
image.save("flux2_t2i_nf4.png").
Flax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.Flax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers./usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_validators.py:206: UserWarning: The local_dir_use_symlinks argument is deprecated and ignored in hf_hub_download. Downloading to a local directory does not use symlinks anymore. warnings.warn(Download complete: 0.00/0.00 [00:00() 22 prompt = "Realistic macro photograph of a hermit crab using a soda can as its shell, partially emerging from the can, captured with sharp detail and natural colors, on a sunlit beach with soft shadows and a shallow depth of field, with blurred ocean waves in the background. The can has the text BFL Diffusers on it and it has a color gradient that start with #FF5733 at the top and transitions to #33FF57 at the bottom." 23---> 24 image = pipe( 25 prompt=prompt, 26 generator=torch.Generator(device=device).manual_seed(42),.
36 frames/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py in decorate_context(*args, **kwargs) 122 # pyrefly: ignore [bad-context-manager] 123 with ctx_factory():--> 124 return func(*args, **kwargs) 125 126 return decorate_context.
/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline_flux2.py in call(self, image, prompt, height, width, num_inference_steps, sigmas, guidance_scale, num_images_per_prompt, generator, latents, prompt_embeds, output_type, return_dict, attention_kwargs, callback_on_step_end, callback_on_step_end_tensor_inputs, max_sequence_length, text_encoder_out_layers, caption_upsample_temperature) 869 prompt, images=image, temperature=caption_upsample_temperature, device=device 870 )--> 871 prompt_embeds, text_ids = self.encode_prompt( 872 prompt=prompt, 873 prompt_embeds=prompt_embeds,.
/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline_flux2.py in encode_prompt(self, prompt, device, num_images_per_prompt, prompt_embeds, max_sequence_length, text_encoder_out_layers) 586 587 if prompt_embeds is None:--> 588 prompt_embeds = self._get_mistral_3_small_prompt_embeds( 589 text_encoder=self.text_encoder, 590 tokenizer=self.tokenizer,.
/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline_flux2.py in _get_mistral_3_small_prompt_embeds(text_encoder, tokenizer, prompt, dtype, device, max_sequence_length, system_message, hidden_states_layers) 337 338 # Forward pass through the model--> 339 output = text_encoder( 340 input_ids=input_ids, 341 attention_mask=attention_mask,.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs) 1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc] 1775 else:-> 1776 return self._call_impl(*args, **kwargs) 1777 1778 # torchrec tests the code consistency with the following code.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs) 1785 or _global_backward_pre_hooks or _global_backward_hooks 1786 or _global_forward_hooks or _global_forward_pre_hooks):-> 1787 return forward_call(*args, **kwargs) 1788 1789 result = None.
/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py in new_forward(module, *args, **kwargs) 190 output = module._old_forward(*args, **kwargs) 191 else:--> 192 output = module._old_forward(*args, **kwargs) 193 return module._hf_hook.post_forward(module, output) 194.
/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, *args, **kwargs) 1000 outputs = func(self, *args, **kwargs) 1001 else:-> 1002 outputs = func(self, *args, **kwargs) 1003 except TypeError as original_exception: 1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.
/usr/local/lib/python3.12/dist-packages/transformers/models/mistral3/modeling_mistral3.py in forward(self, input_ids, pixel_values, attention_mask, position_ids, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict, cache_position, logits_to_keep, image_sizes, **kwargs) 444 return_dict = return_dict if return_dict is not None else self.config.use_return_dict 445--> 446 outputs = self.model( 447 input_ids=input_ids, 448 pixel_values=pixel_values,.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs) 1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc] 1775 else:-> 1776 return self._call_impl(*args, **kwargs) 1777 1778 # torchrec tests the code consistency with the following code.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs) 1785 or _global_backward_pre_hooks or _global_backward_hooks 1786 or _global_forward_hooks or _global_forward_pre_hooks):-> 1787 return forward_call(*args, **kwargs) 1788 1789 result = None.
/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, *args, **kwargs) 1000 outputs = func(self, *args, **kwargs) 1001 else:-> 1002 outputs = func(self, *args, **kwargs) 1003 except TypeError as original_exception: 1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.
/usr/local/lib/python3.12/dist-packages/transformers/models/mistral3/modeling_mistral3.py in forward(self, input_ids, pixel_values, attention_mask, position_ids, past_key_values, inputs_embeds, vision_feature_layer, use_cache, output_attentions, output_hidden_states, return_dict, cache_position, image_sizes, **kwargs) 323 inputs_embeds = inputs_embeds.masked_scatter(special_image_mask, image_features) 324--> 325 outputs = self.language_model( 326 attention_mask=attention_mask, 327 position_ids=position_ids,.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs) 1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc] 1775 else:-> 1776 return self._call_impl(*args, **kwargs) 1777 1778 # torchrec tests the code consistency with the following code.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs) 1785 or _global_backward_pre_hooks or _global_backward_hooks 1786 or _global_forward_hooks or _global_forward_pre_hooks):-> 1787 return forward_call(*args, **kwargs) 1788 1789 result = None.
/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, *args, **kwargs) 1000 outputs = func(self, *args, **kwargs) 1001 else:-> 1002 outputs = func(self, *args, **kwargs) 1003 except TypeError as original_exception: 1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.
/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling_mistral.py in forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, use_cache, cache_position, **kwargs) 395 396 for decoder_layer in self.layers[: self.config.num_hidden_layers]:--> 397 hidden_states = decoder_layer( 398 hidden_states, 399 attention_mask=causal_mask,.
/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py in call(self, *args, **kwargs) 91 92 return self._gradient_checkpointing_func(partial(super().call, **kwargs), *args)---> 93 return super().call(*args, **kwargs) 94 95.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs) 1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc] 1775 else:-> 1776 return self._call_impl(*args, **kwargs) 1777 1778 # torchrec tests the code consistency with the following code.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs) 1785 or _global_backward_pre_hooks or _global_backward_hooks 1786 or _global_forward_hooks or _global_forward_pre_hooks):-> 1787 return forward_call(*args, **kwargs) 1788 1789 result = None.
/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapped_forward(*args, **kwargs) 953 if key == "hidden_states" and len(collected_outputs[key]) == 0: 954 collected_outputs[key] += (args[0],)--> 955 output = orig_forward(*args, **kwargs) 956 if not isinstance(output, tuple): 957 collected_outputs[key] += (output,).
/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling_mistral.py in forward(self, hidden_states, attention_mask, position_ids, past_key_values, use_cache, cache_position, position_embeddings, **kwargs) 228 hidden_states = self.input_layernorm(hidden_states) 229 # Self Attention--> 230 hidden_states, _ = self.self_attn( 231 hidden_states=hidden_states, 232 attention_mask=attention_mask,.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs) 1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc] 1775 else:-> 1776 return self._call_impl(*args, **kwargs) 1777 1778 # torchrec tests the code consistency with the following code.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs) 1785 or _global_backward_pre_hooks or _global_backward_hooks 1786 or _global_forward_hooks or _global_forward_pre_hooks):-> 1787 return forward_call(*args, **kwargs) 1788 1789 result = None.
/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling_mistral.py in forward(self, hidden_states, position_embeddings, attention_mask, past_key_values, cache_position, **kwargs) 151 hidden_shape = (*input_shape, -1, self.head_dim) 152--> 153 query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2) 154 key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2) 155 value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2).
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs) 1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc] 1775 else:-> 1776 return self._call_impl(*args, **kwargs) 1777 1778 # torchrec tests the code consistency with the following code.
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs) 1785 or _global_backward_pre_hooks or _global_backward_hooks 1786 or _global_forward_hooks or _global_forward_pre_hooks):-> 1787 return forward_call(*args, **kwargs) 1788 1789 result = None.
/usr/local/lib/python3.12/dist-packages/bitsandbytes/nn/modules.py in forward(self, x) 554 weight = self.weight if getattr(quant_state, "packing_format_for_cpu", False) else self.weight.t() 555--> 556 return bnb.matmul_4bit(x, weight, bias=bias, quant_state=quant_state).to(inp_dtype) 557 558.
/usr/local/lib/python3.12/dist-packages/bitsandbytes/autograd/_functions.py in matmul_4bit(A, B, quant_state, out, bias) 399 return out 400 else:--> 401 return MatMul4Bit.apply(A, B, out, bias, quant_state).
/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py in apply(cls, *args, **kwargs) 581 # See NOTE: [functorch vjp and autograd interaction] 582 args = _functorch.utils.unwrap_dead_wrappers(args)--> 583 return super().apply(*args, **kwargs) # type: ignore[misc] 584 585 if not is_setup_ctx_defined:.
Sources indicate that /usr/local/lib/python3.12/dist-packages/bitsandbytes/autograd/_functions.py in forward(ctx, A, B, out, bias, quant_state) 313 # 1. Dequantize 314 # 2. MatmulnN--> 315 output = torch.nn.functional.linear(A, F.dequantize_4bit(B, quant_state).to(A.dtype).t(), bias) 316 317 # 3. Save state.
/usr/local/lib/python3.12/dist-packages/bitsandbytes/functional.py in dequantize_4bit(A, quant_state, absmax, out, blocksize, quant_type) 1048 ) 1049 else:-> 1050 out = torch.ops.bitsandbytes.dequantize_4bit.default( 1051 A, 1052 absmax,.
/usr/local/lib/python3.12/dist-packages/torch/_ops.py in call(self, *args, **kwargs) 817 # that are named "self". This way, all the aten ops can be called by kwargs. 818 def call(self, /, *args: _P.args, **kwargs: _P.kwargs) -> _T:--> 819 return self._op(*args, **kwargs) 820 821 # Use positional-only argument to avoid naming collision with aten ops arguments.
/usr/local/lib/python3.12/dist-packages/torch/_compile.py in inner(*args, **kwargs) 52 fn.__dynamo_disable = disable_fn # type: ignore[attr-defined] 53---> 54 return disable_fn(*args, **kwargs) 55 56 return inner.
/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py in _fn(*args, **kwargs) 1179 ): 1180 return fn(*args, **kwargs)-> 1181 return fn(*args, **kwargs) 1182 finally: 1183 set_eval_frame(None).
/usr/local/lib/python3.12/dist-packages/torch/library.py in func_no_dynamo(*args, **kwargs) 740 @torch ._disable_dynamo 741 def func_no_dynamo(*args, **kwargs):--> 742 return func(*args, **kwargs) 743 744 for key in keys:.
/usr/local/lib/python3.12/dist-packages/bitsandbytes/backends/cuda/ops.py in _(A, absmax, blocksize, quant_type, shape, dtype) 361 dtype: torch.dtype, 362 ) -> torch.Tensor:--> 363 out = torch.empty(shape, dtype=dtype, device=A.device) 364 _dequantize_4bit_impl(A, absmax, blocksize, quant_type, dtype, out=out) 365 return out.
In a recent development, outOfMemoryError: CUDA out of memory. Tried to allocate 40.00 MiB. GPU 0 has a total capacity of 14.56 GiB of which 17.81 MiB is free. Including non-PyTorch memory, this process has 14.54 GiB memory in use. Of the allocated memory 14.40 GiB is allocated by PyTorch, and 15.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables).
https://github.com/huggingface/diffusers/blob/main/docs/source/en/using-diffusers/loading.md.