Gemma 4 Technical Report

Gemma Team et al.

Audio by Paper2Audio.

We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3 billion to 31 billion parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12 billion model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across stem, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.

1. Introduction

The rapid evolution of large language models has driven the need for open-weight models with strong multimodal understanding, reasoning, and computational efficiency. Building upon the foundations of its predecessors, we introduce Gemma 4, the most capable and efficient generation in the Gemma model family to date. Gemma 4 offers natively multimodal architectures, capable of seamlessly processing text, images, and audio while achieving frontier-level performance on highly complex reasoning tasks. The Gemma 4 family is built to serve a variety of on-device hardware.

The model suite includes both dense architectures (2.3 billion, 4.5 billion, 12 billion, and 31 billion parameters) and a Mixture-of-Experts variant with 3.8 billion activated and 26 billion total parameters. We introduce several architectural and methodological innovations:

• Thinking mode for advanced reasoning: We introduce a thinking mode to Gemma 4 models. By outputting a reasoning trace before the response, models demonstrate improved capabilities in reasoning-heavy domains such as mathematics and coding.

• Long-context efficiency: Extended contexts lead to a memory explosion in the K.V cache. We conserve a 5:1 ratio of local sliding window to global self-attention (4:1 for the 2.3 billion model) and use p-RoPE as positional encoding. Combined with K.V cache sharing and the reuse of keys as values in global layers, these optimizations reduce the global K.V cache footprint by up to 37.5%.

• Compute efficiency: We release an autoregressive multi-token prediction (M.T.P) drafter head designed for speculative decoding to improve the decoding speed of our models.

• Memory efficiency: We provide quantized versions of our models trained with quantization-aware training to reduce their parameter memory footprint and latency with minimal impact on quality.

• Encoder-free architecture: Gemma 4 models have frozen vision and audio encoders. We introduce a unified encoder-free architecture for the 12 billion model, which projects raw 40ms audio chunks and image patches into the L.L.M embedding space, alleviating the need for separate encoders and reducing memory fragmentation.

In this technical report, we outline the different model architectures across model sizes as well as the pre-training and post-training recipe of Gemma 4. Through comprehensive benchmarks and human evaluations such as Arena, we demonstrate that Gemma 4 operates at a level comparable to larger, frontier open-source models across text, image, and audio modalities. We release the Gemma 4 models under an Apache 2.0 license, empowering developers and researchers everywhere to build upon, customize, and extend these capabilities.

Table 1 summary: Parameter counts for the Gemma 4 model family, which uses a vocabulary of 262 thousand entries. The model sizes scale from E2B up to 31B. The smallest models, E2B and E4B, both utilize a 305 million parameter audio encoder and a 150 million parameter vision encoder, with E4B having larger einsums at 3,940 million compared to 1,870 million for E2B. The largest model, 31B, features the highest parameter counts for einsums at 29,290 million, the embedder at 1,410 million, and the drafter at 500 million. The 26B-A4B model is a Mixture of Experts model with 24,500 million total einsum parameters, of which 2,800 million are active.

2. Model Architecture

Gemma 4 models follow a decoder-only Transformer architecture. Our models have pre-norm and post-norm with R.M.S.Norm, and Q.K.Norm.

Dense and MoE: The Gemma 4 family of models comprises dense architectures, with effective 2.3 billion (E.2.B), effective 4.5 billion (E.4.B), 12 billion and 31 billion parameters, as well as an MoE model with 3.8 billion activated parameters for 26 billion total parameters (26 billion-A4B). E.2.B and E.4.B use per-layer embeddings as in Gemma 3n, making them 2.3 billion and 4.5 billion effective out of 5 billion and 8 billion total parameters respectively.

Long-context efficiency: Our local to global attention ratio patterns follow, that is, 4-to-1 local attention blocks for E.2.B and 5-to-1 for the rest. We improve memory efficiency by re-using keys as values in the global attention layers (except in E.2.B and E.4.B), that is, values = keys. We encode position with p-RoPE with p = 0.25 on global attention layers and with RoPE on local attention layers, effectively reducing the global K.V cache by 37.5%. The RoPE frequencies are set to 1 million and 10k on global and local attention layers, respectively. Finally, we share the K.V cache with ratios of 20/35 and 18/42 for the E.2.B and E.4.B model.

2.1. Vision modality

E.2.B and E.4.B Gemma models come with a 150 million vision encoder, while larger models use a 550 million encoder (except for the unified 12 billion). Both are Vision Transformers with a patch size of 16, whose architectural differences are detailed in Table 10 in Appendix. Our vision encoders support variable aspect ratios (see Figure 2 and Algorithm 1) and incorporate both axial 2.D-RoPE with non-causal attention and 2.D absolute positional embeddings. We restrict the maximum number of tokens, N max to the values 70, 140, 280, 560 and 1120 (see Algorithm 1 for implementation details).

Table 10 summary: Vision encoder architecture specifications for two model sizes. The larger 550M parameter model uses a model dimension of 1152, an MLP dimension of 4304, 16 heads, and 27 layers. The smaller 150M parameter model uses a model dimension of 768, an MLP dimension of 3072, 12 heads, and 16 layers.

Figure 2 summary: A diagram illustrating an image resizing and pooling process. An original image of an otter in a spacesuit is resized to a smaller version and divided into a grid of pooled patches. According to the caption, the image is resized to a 2 by 4 grid of pooled patches to keep the sequence length below a target of 10, with 72 original patches pooled by a 3 by 3 kernel to produce 8 soft tokens for a language model. The process demonstrates how high-resolution images are compressed into a small number of tokens for efficient LLM processing.

2.2. Audio modality

E.2.B and E.4.B Gemma models use a 305 million audio encoder that processes audio in 40ms chunks with Mel filterbank inputs. The encoder architecture is based on the Universal Speech Model, consisting of two downsampling convolution layers followed by twelve Conformer layers. While the architecture remains similar to that of Gemma 3n, we reduce the number of parameters by 55% (from 680 million to 305 million). We do not use vector quantization; the L.L.M ingests the con tinuous representations produced by the audio encoder. As with the vision encoder, we keep weights frozen during pre-training.

2.3. Encoder-free architecture

Gemma 4 12 billion is trained from scratch based on a new, unified, and encoder-free model paradigm, replacing the separate vision and audio encoders with lightweight projection modules. For the vision modality, Gemma 4 12 billion takes in 48 times 48 times 3 R.G.B patches, but replaces the 550 million vision encoder by a single large matmul (35 million parameters). Spatial awareness is maintained by adding 2.D coordinate-based positional embeddings directly to the patch representations before a final LayerNorm layer.

For audio, the 305 million U.S.M-based conformer encoder is entirely discarded. Raw audio is segmented into 40ms chunks at 16 kilohertz, resulting in 640-dimensional vectors per chunk. These are projected directly into the L.L.M embedding space. Since audio is a temporal sequence, it does not require additional positional encoding.

2.4. Pre-training

We follow a similar pre-training as Gemma 3.

Training data. Our pre-training dataset is a large-scale, diverse collection of data from a wide range of domains and modalities, including web documents, code, images, and audio (for E.2.B, E.4.B and 12 billion), with a cutoff date of January 2025.

Tokenizer. We use the same tokenizer as Gemini Team (2025) that is, a SentencePiece tok enizer with split digits, preserved whitespace, and byte-level encodings. The vocabulary has 262k entries.

Filtering. We filter data to decontaminate benchmarks, and to reduce the risk of unwanted or unsafe utterances and the risk of recitation.

2.5. Quantization-Aware Training

We provide quantized models and encoders in different formats along with the raw checkpoints. Based on the most popular open source quantization inference engines (e.g. llama.cpp) as well as efficient hardware support, we focus on two sets of weight representations:

- mobile quantization: per-channel low bitwidth weight (mix of int2 and int4) and activation quantization (int8).

- Q.4 0 quantization: blockwise quantization, often referred to as Q.4 0.

In Table 3, we report the memory filled by raw and quantized models with and without a K.V cache for a sequence of 32k tokens. Furthermore, to enable stable inference in fp16, we introduce a scalar scale at each block in order to bound the activation ranges to fit fp16.

: Table 3 summary: Quantized checkpoints significantly reduce the memory footprint of weights compared to bf16 across all model sizes. For example, the 31B model's memory footprint drops from 64.0 Gb in bf16 to 19.2 Gb with Q4_0 quantization. The smallest model, E2B, decreases from 4.6 Gb to 0.8 Gb using mobile quantization. Adding an int8 KV cache at a 32k context size increases the memory footprint by a small amount, ranging from +0.05 Gb for E2B to +1.10 Gb for the 31B model.

We also apply Q.A.T to the image and audio encoders. On the 150 million image encoder, quantizing activations and weights to 8-bit precision (W.8.A.8) yields a 2× reduction in total forward-pass memory footprint (from 400 M.B to 200 M.B, including on-device compilation overhead) and a 44% reduction in on-device latency relative to Gemma 3n on newer hardware. On the audio encoder, we further reduce activation precision to 8 bits and weight precision to {2, 4, 8} bits, varying by layer cluster. Overall, we achieve a 78% reduction in on-disk footprint, from 390 M.B in Gemma 3n to 87 M.B in this version.

2.6. Multi-Token Prediction Drafter

We train a small autoregressive M.T.P drafter head with our models, used for speculative decoding. In our M.T.P procedure, the model's last layer activations from the previous step and token embeddings are fed into the M.T.P head. The M.T.P head generates future tokens sequentially using a separate embedder and a 4-layer Transformer block that cross-attends to the K.V's of the main model (Figure 1), thus eliminating the need for M.T.P prefill and supporting any draft length. The Transformer block has model dimension 256 for E.2.B and E.4.B, 1024 for 26 billion-A4B and 31 billion, three local, and one global attention layers.

Figure 1 summary: A schematic diagram of an autoregressive Multi-Token Prediction (MTP) drafter integrated with a main model. The main model, shown in gray, processes an input embedding through sequential layers to produce a final activation. The MTP drafter, shown in blue, uses these activations and attention mechanisms to predict subsequent tokens across multiple time steps (t3 and t4) using a series of MTP layers and projection blocks. This architecture allows the drafter to generate multiple future tokens efficiently by leveraging the internal representations of the main model.

Efficient M.T.P Decoding. For the E.2.B and E.4.B drafters, we reduce the decoding overhead by replacing the projection operation to the entire vocabulary by a top-k operation on clusters of tokens. As a result, final matrix multiplication is reduced from d times 262 , 000 to d times 4096 while preserving a similar acceptance rate.

2.7. Compute Infrastructure

We train our models with T.P.U.v5p and T.P.U.v6e as outlined in Table 2. Each model configuration is optimized to minimize training step time. For our larger models, we leverage Slice-Granularity Elasticity, which allows continuous training with fewer “slices” of T.P.U chips when there is a localized failure. This reconfiguration reduces the delay caused by interruptions from many minutes to a few seconds.

Table 2 summary: Pre-training infrastructure configurations for five models, showing a scaling of compute resources and sharding strategies. The 12B model uses the most hardware with 12,288 v5p TPU chips and 48 replica shards. Other models use v6e TPU chips, with chip counts ranging from 4,096 for the E2B model to 10,240 for the 31B model. Across all models, data sharding remains constant at 16, while sequence sharding is 8 for the E2B model and 16 for all others. Replica sharding varies by model, ranging from 24 for the E4B and 26B-A4B models up to 40 for the 31B model.

The optimizer state is sharded using an implementation of ZeRO-3. For multi-pod training, we perform a data replica reduction over the data center network, using the Pathways approach of Barham et al. (2022). We use the single controller programming paradigm of jax and Pathways, along with the G.S.P.M.D partitioner and the MegaScale X.L.A compiler.

3. Instruction Tuning

Pre-trained models are turned into instruction-tuned models with a similar post-training approach as in Gemma 3. A significant difference is the addition of a thinking mode, where the model can output a reasoning trace before answering.

Data filtering. We carefully optimize the data used in post-training to maximize model performance. mance. We filter examples that show certain personal information, unsafe or toxic model outputs, mistaken self-identification data, and duplicated examples. Including subsets of data that encourage better in-context attribution, hedging, and refusals to minimize hallucinations also improves performance on factuality metrics, without degrading model performance on other metrics.

P.T versus I.T formatting. All models share the same tokenizer, with some control tokens dedicated to I.T formatting. A key difference is that P.T models output an <eos> token at the end of generation, while I.T models output <turn|> at the end of the generation. An example is given for I.T in Table 11. Fine-tuning either model type thus requires adding their respective end tokens. We detail how to activate thinking and how models handle function calling in Table 11.

Table 11 summary: Formatting specifications for Gemma IT models. The formatting uses specific tags to denote different contexts, such as <| think|> for the thinking toggle, <| tool>declaration: ...<tool|> for function declarations, and <| tool_call>call: ...<tool_call|> for function calls. Interaction turns are marked with <| turn>system, <| turn>user, and <| turn>model, with <turn|> indicating the end of a turn. Additionally, thinking traces are formatted as <| channel>thought ...<channel|>.

4. Evaluation of final models

In this section, we evaluate the I.T models over a series of automated benchmarks and human evaluations across a variety of domains, as well as static benchmarks such as M.M.L.U Pro.

4.1. Human evaluation

We report the performance of our 31 billion and 26 billion-A4B models on Arena in blind side-by-side evaluations by human raters against other state-of-the-art models. We report Elo scores in Table 4. Gemma 4 31 billion is the top open model in the dense category, and both Gemma 4 31 billion and 26 billion-A4B show performance equal to much larger open models.

Table 4 summary: Gemma 4 31B is the leading dense open-weight model on the Arena Text leaderboard, ranking 43rd overall with an Elo score of 1451. It rivals significantly larger Mixture of Experts models, such as GLM 5, which ranks 37th with an Elo of 1457 despite having 744 billion total parameters. Other Gemma models include Gemma 4 26B-A4B, which ranks 61st with an Elo of 1438, and Gemma 3 27B, which ranks 157th with an Elo of 1366. For scale, the top closed model, Claude Fable 5, holds the 1st rank with an Elo of 1508.

4.2. Static benchmarks

In Table 5, we show the performance of our final models across a variety of benchmarks compared to Gemma 3 27 billion. Gemma 4 31 billion is closest in size and significantly better across the board, while E.2.B roughly matches Gemma 3 27 billion performance with 10x less parameters. Table 6 shows the performance of Gemma 4 models on vision benchmarks, with E.4.B equaling or outperforming Gemma 3 27 billion on all evals. Tables 7 and 8 display the multilingual audio transcription and translation performance of E.2.B & E.4.B and of 12 billion respectively. Table 9 shows a leap on long-context capabilities between Gemma 3 27 billion and Gemma 4 models, with E.4.B outperforming Gemma 3 27 billion.

Table 5 summary: Gemma 4 models generally outperform Gemma 3 27B non-thinking across all benchmarks, with performance scaling upward as model size increases. The Gemma 4 31B model achieves the highest scores overall, including 85.2 on MMLU Pro, 89.2 on AIME 2026 no tools, and a Codeforces Elo of 2150. In comparison, the Gemma 3 27B non-thinking model scores 67.6 on MMLU Pro, 20.8 on AIME 2026 no tools, and 110 on Codeforces Elo. Significant gains are seen in the Gemma 4 series for complex reasoning tasks; for example, on the HLE benchmark, Gemma 4 31B scores 19.5, while the smaller Gemma 4 12B scores 5.2. The Gemma 4 31B also shows a substantial lead in the MRCR v2 8-needle, 128k test at 66.4, compared to 13.5 for the Gemma 3 27B non-thinking model.

Table 6 summary: Gemma 4 models demonstrate performance gains across vision benchmarks as model size increases, with the 31B model achieving the highest scores. For the 31B model, key results include 76.9 on MMMU Pro, 85.6 on MATH-Vision, and 92.0 on InfographicVQA. In contrast, the smallest Gemma 4 model, E2B, scores 44.2 on MMMU Pro, 52.4 on MATH-Vision, and 63.9 on InfographicVQA. On the OmniDocBench 1.5 benchmark, where lower scores are better, the 31B model achieves the best result of 0.131, compared to 0.290 for E2B and 0.365 for the non-thinking Gemma 3 27B model.

Table 7 summary: Gemma 4 models outperform Gemma 3n models in both translation and transcription tasks while using a significantly smaller audio encoder footprint of 87 megabytes compared to 390 megabytes. In the CoVoST translation task, Gemma 4 E4B achieves the highest average score of 38.2, while Gemma 4 E2B averages 35.4, representing relative improvements of 10 percent and 12 percent respectively over corresponding Gemma 3n models. For FLEURS ASR transcription, Gemma 4 E4B reaches the lowest average Word Error Rate of 0.075 and Gemma 4 E2B reaches 0.090, marking relative improvements of 12 percent and 17 percent over the Gemma 3n variants.

Table 8 summary: Gemma 4 12B achieves competitive audio-text performance across multiple languages without a dedicated audio encoder. In FLEURS ASR tasks, Word Error Rates range from 0.030 for Italian to 0.081 for French, while Character Error Rates for Korean and Japanese are 0.057 and 0.080 respectively. For CoVoST translation from various languages into English, CorpusBLEU scores range from 26.4 for Japanese to 50.5 for Russian.

Table 9 summary: Gemma 4 models generally outperform Gemma 3 27B across multiple long context benchmarks. On the RULER benchmark at 128k context length, Gemma 4 31B achieves 96.4 percent accuracy, significantly higher than the 66.0 percent of Gemma 3 27B. This trend continues in LOFT Text Retrieval at 128k, where Gemma 4 31B reaches 79.5 percent Recall@k compared to 8.6 percent for Gemma 3 27B. In GraphWalks, Gemma 4 31B scores 82.3 F1, while Gemma 3 27B scores 32.8. For MTOB translation tasks at approximately 128k context, Gemma 4 31B leads with 52.9 chrF for English to kgv and 48.6 chrF for kgv to English, compared to 41.0 and 31.2 for Gemma 3 27B, respectively. Within the Gemma 4 family, performance typically scales with model size, with the 31B and 26B-A4B models consistently performing better than the smaller E4B and E2B variants.

5. Responsibility, Safety, Security

As open models become central to enterprise infrastructure, provenance and security are paramount. Gemma 4 undergoes the same rigorous safety evaluations as Gemini models. Responsibility, safety, and security are of utmost importance in the development workflow, ensuring that these language models are designed from the ground up for responsible A.I development.

5.1. Governance & Assessment

Our approach to assessing the benefits and risks of Gemma 4 reflects the foundation established in prior models, updated to account for its expanded multimodal capabilities. We maintain the belief that openness in A.I can spread the benefits of these technologies across society, but this must be continuously evaluated against the risk of malicious uses that can cause individual and institutional harm.

Gemma 4 models were developed in partnership with internal safety and responsible A.I teams. Releasing these models required careful scrutiny of the evolving risks associated with L.L.M's and an understanding of how models are deployed in the wild. While an open model shares innovation across the A.I ecosystem, we remain committed to providing educational resources to users and monitoring downstream model usage.

5.2. Safety Policies and Train-Time Mitigations

A key pillar of Gemma's safety approach is aligning our fine-tuned models with Google's A.I principles and safety policies. These policies aim to prevent our generative models from producing harmful content, specifically:

• Content related to child sexual abuse material

- (C.S.A.M) and exploitation;

- Dangerous content, for example, promoting suicide, or instructing in activities that could cause real-world harm;

• Sexually explicit content;

- Hate speech, for example, dehumanizing members of protected groups;

- Harassment, for example, encouraging violence against people.

To mitigate these risks, Gemma 4 models underwent careful input data pre-processing and scrutiny. The training data was specifically filtered for the removal of certain personal information and other sensitive data to guard against privacy violations. Post-training evaluations and train-time mitigations were also implemented to align the model with our safety policies.

5.3. Safety Evaluations

We conduct rigorous automated and human evaluations to understand the potential harms our models might cause. For all areas of safety testing, we saw major improvements in every category of content safety relative to previous Gemma models. Overall, Gemma 4 models significantly out- perform Gemma 3 and 3n models in improving safety, while keeping unjustified refusals low.

Importantly, all testing was conducted without safety filters to accurately evaluate the model's inherent capabilities and behaviors. For both text-to-text and image-to-text modalities, and across all model sizes, the models produced minimal policy violations. We balance development speed with targeted safety testing, upholding the commitments laid out in our Frontier Safety Framework.

5.4. Ethical Considerations and Risk Mitigation

The development of L.L.M's introduces specific ethical considerations. In making Gemma 4, we focused heavily on:

- Bias and Fairness: L.L.M's trained on large-scale text and image data can reflect embedded sociocultural biases. We encourage developers to perform continuous monitoring (using evaluation metrics and human review) and explore debiasing techniques during model fine-tuning.

- Misinformation and Misuse: L.L.M's can be misused to generate false or misleading text. We provide technical limitations, developer education, and guidelines for responsible use within the Responsible Generative A.I Toolkit to mitigate malicious applications.

• Privacy Considerations: While our training datasets were filtered to remove certain personal information and other sensitive data, developers are strongly encouraged to adhere to local privacy regulations and implement privacy-preserving techniques in their applications.

5.5. Our Approach to Responsible Open Models

Designing safe, secure, and responsible applications requires a system-level approach that mitigates risks associated with specific use cases and environments. We provide guidelines, mechanisms, and safeguards for content safety, and encourage developers to implement appropriate configurations based on their product policies. We will continue to adopt safety mitigations proportionate to potential risks, sharing these models with the community only when confident that the benefits significantly outweigh foreseeable risks.

6. Discussion and Conclusion

In this technical report, we presented Gemma 4, an open-weight model family featuring multimodal dense and MoE architectures designed for varied hardware environments. Gemma 4 models come with a thinking mode in which they generate reasoning traces prior to responding, improving overall performance. We introduced a unified, encoder-free architecture that processes raw audio and image patches. We also alleviated long-context memory limitations via better local-to-global attention ratios, positional encoding, and K.V cache sharing. We increased the overall compute efficiency via Q.A.T and memory efficiency via M.T.P drafters. Gemma 4 models demonstrate a leap in performance compared to Gemma 3 across benchmarks, and human evaluations demonstrate that Gemma 4 performs comparably to significantly larger open models, providing a scalable foundation for edge deployment and reasoning while supporting open research.

Table summary: A list of names organized alphabetically across three columns. Examples include Vidya Venkiteswaran, Hao Xu, Sajjad Zafar, Francesco Visin, and David Zhou.

You have reached the end of the main document. Additional summarized content follows

Table 12 summary: Performance on vision benchmarks for Gemma 4 models at a resolution of 280, showing a general trend of higher scores as model size increases. The 31B model achieves the highest scores across most benchmarks, including 75.8 on MMMU Pro, 83.4 on MATH-Vision, 60.7 on MedXPertQA MM, and 82.8 on InfographicVQA. Performance declines across these four metrics as the models scale down through 26B-A4B, 12B, E4B, and E2B, with the E2B model scoring the lowest, such as 43.2 on MMMU Pro and 22.5 on MedXPertQA MM. For OmniDocBench 1.5, where a lower score is better, the 31B model performs best at 0.201, while the E2B model has the highest error rate at 0.496.