Gemini and Google Models

Why This Matters

Google researchers introduced the transformer architecture, and Google owns the full Gemini stack: research lab, TPU hardware, compiler and serving systems, product surfaces, Gemini API, Vertex AI, and open-weight Gemma releases. That vertical integration matters because model quality is not only a neural-network question. It is also a systems question: training hardware, data pipelines, multimodal serving, long-context retrieval, safety gates, product telemetry, and API stability all shape what users actually experience.

This page is a dated reference snapshot. As of May 29, 2026, the public Gemini API surface has several layers that should not be confused: stable 2.5 models, stable and preview Gemini 3-family endpoints, separate media models, and agent/tool models such as Deep Research and Antigravity. The operational source of truth is the Google AI for Developers model list and deprecation table, not launch-blog naming.

Current Snapshot: May 29, 2026

Definition

Preview model

A preview Gemini model is an API endpoint Google makes available before it is treated as a stable model string. Preview models can be useful, but they may change, be renamed, or be shut down with notice. The current model list explicitly separates stable, preview, latest, experimental, deprecated, and shut-down statuses; this matters more for product code than the family name.

Definition

Stable model

A stable Gemini model points to a specific production model. Stable does not mean permanent. It means the endpoint is safer for production code than a preview alias, because behavior and replacement timelines are documented more conservatively.

Pre-Gemini: PaLM and PaLM 2

PaLM (April 2022)

PaLM was a 540B-parameter dense language model trained with Google's Pathways infrastructure. It matters historically because it sits at the point where scale, chain-of-thought prompting, and broad benchmark evaluation became tightly coupled in public language-model research.

The emergence story should be read carefully. PaLM showed large jumps on some tasks, but later work argued that several "emergent abilities" can be artifacts of nonlinear metrics or thresholded evaluations rather than literal phase transitions in model cognition.

PaLM 2 (May 2023)

PaLM 2 improved multilingual, reasoning, and coding performance over PaLM. Google did not disclose exact parameter counts. The safer public interpretation is that PaLM 2 emphasized better data mixture and compute allocation rather than a larger announced parameter count.

Gemini 1.0 (December 2023)

Gemini 1.0 shipped as Nano, Pro, and Ultra:

• Nano: on-device models for mobile settings.
• Pro: default mid-tier model for Google products and developers.
• Ultra: largest 1.0 model, reserved for the highest-capability tier at launch.

The important design claim was native multimodality. Gemini 1.0 was trained on interleaved text, image, audio, and video data, rather than only attaching a vision encoder to a text-only model after pretraining. That does not prove superior vision performance on every task; it does explain why Google framed Gemini as a family built around cross-modal reasoning from the start.

Gemini 1.5: Long Context Becomes a Product Feature

Gemini 1.5 Pro introduced the public long-context story. Google described Gemini 1.5 as a mixture-of-experts model and announced a standard 128K-token context window, with up to 1M tokens for selected developers and enterprise users at launch. Google later described production execution at up to 1M tokens and research tests beyond that.

The conceptual shift was not merely "more tokens." A million-token context changes the application shape: full code repositories, long legal records, long video/audio inputs, and multi-document research tasks can be placed in one prompt. The hard part is not advertising the window. The hard part is retrieving the right evidence inside that window reliably and cheaply.

Proposition

Quadratic Attention Cost for Long Sequences

Statement

Standard self-attention computes a full $L \times L$ attention matrix, requiring $O(L^2 d)$ FLOPs and $O(L^2)$ memory per layer. For $L = 1{,}000{,}000$ tokens, the attention matrix has $10^{12}$ entries per layer.

Intuition

Every token can attend to every other token. Doubling the context length roughly quadruples the attention-matrix work. A million-token context is therefore not a normal transformer setting; it requires model, systems, and serving tricks.

Proof Sketch

Self-attention computes $\operatorname{softmax}(QK^\top / \sqrt{d_k})V$. The matrix $QK^\top$ has shape $L \times L$. Computing and storing that matrix scales quadratically in $L$.

Why It Matters

Long-context Gemini models are useful because they change the user workflow, but context length alone is not intelligence. The model still has to locate evidence, preserve instruction hierarchy, avoid distraction from irrelevant middle-context material, and pay the latency/cost bill.

Failure Mode

If a model supports a million-token prompt but retrieval quality collapses for evidence placed in the middle, the advertised window overstates the useful window. Always test long-context models with positional retrieval tasks that match the real workload.

report a correction →

Gemini 2.0 and 2.5

Gemini 2.0 made Flash the center of the initial release. Google framed 2.0 around tool use, low-latency serving, native multimodal output, and "thinking" experiments such as Gemini 2.0 Flash Thinking.

Gemini 2.5 made thinking models the main line. Google released Gemini 2.5 Pro Experimental on March 25, 2025 and described it as a model that reasons through internal steps before responding. Google also stated that 2.5 Pro shipped with a 1M-token context window, with a 2M-token window planned at launch.

As of the May 2026 review, the Gemini API docs still list stable 2.5 endpoints:

The right lesson is practical: model family pages and API docs can disagree in spirit if one is a launch narrative and the other is an operational interface. For production code, the API docs and deprecation table matter more.

Gemini 3 Generation

Gemini 3 Pro launched on November 18, 2025 across the Gemini app, AI Mode in Search, Google AI Studio, Vertex AI, Gemini Enterprise, Gemini CLI, and Google Antigravity. The Gemini 3 Pro model card says Gemini 3 Pro is a sparse mixture-of-experts transformer with native multimodal support for text, vision, and audio inputs, a context window up to 1M tokens, and text output up to 64K tokens.

The May 2026 Gemini API model page lists the Gemini 3 generation as a mixed stable/preview surface:

Google's model list also separates model families from agent products: Deep Research, Deep Research Max, and Antigravity appear as tool/agent models, not just chat completions. That is a product architecture shift: the model is becoming one component in longer research, coding, browsing, and environment-interaction workflows.

Gemma: The Open-Weight Branch

Gemma is Google's open-weight branch built from Gemini research, not the same thing as frontier Gemini API models.

There is no official Google Gemma 4 line in the reviewed Google AI for Developers Gemma release notes. If a third-party table uses "Gemma 4," treat it as unverified until it appears in Google's own Gemma docs or release notes.

Use Gemma when the deployment constraints are local control, privacy, cost, fine-tuning, licensing, or edge hardware. Use closed Gemini API models when the task needs the strongest Google-hosted model and you can accept external API dependency.

Google's Infrastructure Advantage

TPUs. Gemini is trained and served on Google's TPU stack. That gives Google unusually tight control over the hardware, compiler, distributed training system, serving path, and model architecture.

Data and products. Search, YouTube, Android, Workspace, Google Books, and other Google products create a large surface for data, evaluation, user feedback, and deployment. This is not automatically a quality guarantee. It does mean that Gemini has a distribution and integration path that most labs cannot match.

API and product coupling. Gemini is not only a model family. It is a product layer across Search, Gemini app, NotebookLM, Android Studio, Vertex AI, Google AI Studio, and Google Cloud. That matters because user-facing capabilities often come from model plus tools plus retrieval plus product constraints.

What Not To Overclaim

• Do not rank Gemini from stale benchmark tables. Public rankings move quickly. Use current benchmark pages only for dated comparisons, and state the date.
• Do not treat context length as solved retrieval. A 1M-token window is useful, but lost-in-the-middle behavior, instruction conflicts, and irrelevant evidence still matter.
• Do not confuse Gemma with Gemini. Gemma is open-weight and local-friendly; Gemini is Google's closed hosted frontier line.
• Do not treat preview endpoints as stable defaults. The Gemini model docs explicitly distinguish stable, preview, latest, experimental, deprecated, and shut-down strings.
• Do not infer hidden parameter counts. Google does not disclose most Gemini frontier parameter counts. The factual page should say "undisclosed" rather than repeat rumors.

Common Confusions

Watch Out

Native multimodality does not automatically mean better vision

Jointly training on text, images, audio, and video can help cross-modal reasoning, but it does not guarantee superior performance on every visual task. Evaluation still depends on the benchmark, prompting setup, image resolution, and whether the task needs OCR, spatial reasoning, world knowledge, or tool use.

Watch Out

A preview model can be stronger and riskier at the same time

A preview endpoint may be the best public Gemini model for a task, but it can also have more churn. For a product that needs reproducible behavior, a stable 2.5 model may be safer until a 3.x model becomes stable.

Watch Out

Long context is not the same as memory

Context is information supplied inside the current request. Memory is information retained across requests by a product or agent system. Gemini can have a large context window, but persistent user memory depends on the product layer and user controls.

Exercises

ExerciseCore

Problem

The Gemini API docs list Gemini 3-family text endpoints with million-token input windows. What practical problem does this solve that a 128K-token model cannot solve as directly?

Hint

Reveal Solution

ExerciseAdvanced

Problem

Why is gemini-3.1-pro-preview not automatically the correct production choice even if it is the current Pro preview?

Hint

Reveal Solution

ExerciseResearch

Problem

Design a fair evaluation for choosing between Gemini 2.5 Pro, a current Gemini 3-family hosted endpoint, and Gemma 3n for a private-codebase assistant.

Hint

Reveal Solution

References

Canonical and technical:

• Vaswani et al., "Attention Is All You Need" (2017)
• Chowdhery et al., "PaLM: Scaling Language Modeling with Pathways" (2022)
• Google DeepMind, "Gemini: A Family of Highly Capable Multimodal Models" (2023)
• Google, "Introducing Gemini 1.5, Google's next-generation AI model" (Feb 15, 2024)
• Google DeepMind, "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context" (2024)
• Google DeepMind, Gemini 3 Pro Model Card (updated Dec 2025)

Current product/API references:

• Google AI for Developers, Gemini models (accessed May 29, 2026; page last updated May 28, 2026)
• Google AI for Developers, Gemini API deprecations (accessed May 29, 2026)
• Google, "Gemini 2.5: Our most intelligent AI model" (Mar 25, 2025)
• Google, "Gemini 3: Introducing the latest Gemini AI model from Google" (Nov 18, 2025)
• Google, "Gemini 3 Flash: frontier intelligence built for speed" (Dec 17, 2025)
• Google, "Introducing Gemma 3" (Mar 12, 2025)
• Google AI for Developers, Gemma models overview (accessed May 29, 2026)
• Google AI for Developers, Gemma releases (accessed May 29, 2026)
• Schaeffer et al., "Are Emergent Abilities of Large Language Models a Mirage?" (2023)

Next Topics

• Model comparison table: compare Gemini against GPT, Claude, DeepSeek, Llama, Qwen, and Kimi with dated caveats
• Sparse attention and long context: why million-token windows require engineering beyond vanilla attention
• Llama and open-weight models: compare Gemma's open-weight branch with Meta's Llama line