WWDC 2026
Developer Sessions Deep Analysis

Table of Contents

1. Executive Summary: The New Apple AI Developer Stack
2. Core AI: The Inference Engine (Session 1: Ben)
3. Core AI On-Device Intelligence (Session 5: Carina)
4. Foundation Models Framework (Session 2: Christopher Webb)
5. LanguageModel Protocol Close Reading
6. Custom Segments & Server-Side Tools
7. Advanced App Intents: Siri Customization (Session 3: Antonio Cancio)
8. Content Discovery: Semantic Index & Structured Search
9. Onscreen Awareness: Connecting UI to Intelligence
10. Entity Annotations: Notifications, NowPlaying, Alarms
11. SpotlightSearchTool (Session 4: Jennifer)
12. Pipeline Stages & Custom Computation
13. Evaluations Framework: Result Coverage & Sample Generation
14. Cross-Cutting Themes: Privacy, Multiplatform, Performance
15. Synthesis: Apple's Developer AI Strategy for 2026 and Beyond

Executive Summary: The New Apple AI Developer Stack

Five sessions. One unified vision. A complete AI platform for every developer.

WWDC 2026 marks a watershed moment for Apple's AI developer platform. Across five interconnected sessions, Apple unveiled a comprehensive stack that spans the entire model lifecycle, from Python-based authoring and conversion, through Swift-native inference on device, to protocol-driven model interchange that makes Claude, Gemini, and community models first-class citizens alongside Apple's own. The message is unmistakable: AI development on Apple platforms is no longer gated behind specialized ML engineering teams. Every iOS, macOS, visionOS, and watchOS developer can now integrate state-of-the-art intelligence into their apps using familiar tools and consistent APIs.

Core AI marks the next evolution of on-device AI execution across Apple platforms. It's built from the ground up for modern workloads, and delivers the high-performance inference you need to build advanced AI features. Core AI is the inference framework powering on-device Apple Intelligence. And now, it's available for you to use.

Ben, Core AI Engineer — Introduction to Core AI

What makes WWDC 2026's announcements truly transformative is not any single technology in isolation, but the systematic integration across layers. A model converted from PyTorch using Core AI's Python toolchain can be packaged as a .aimodel file, loaded through the Core AI Swift framework, wrapped as a CoreAILanguageModel conforming to the Foundation Models LanguageModel protocol, and immediately accessible through LanguageModelSession with structured output, streaming, and tool-calling, all without writing a single line of networking or serialization code. This is the end-to-end promise of the new Apple AI developer stack.

Core AI: The Inference Engine

From PyTorch to production on Apple Silicon, the complete conversion, integration, and optimization pipeline.

Whether you want your app to identify who's talking in a live meeting with a small speaker diarization model, your users to point their camera at anything, ask a question, and instantly get an answer with a larger vision language model, or let them hand off complex, multi-step tasks to a powerful agentic assistant powered by a 70 billion parameter LLM — Core AI has you covered. With all of it running locally on Apple devices, with no server and no cost per token.

Ben, Core AI Engineer

🔧 Model Conversion Pipeline: PyTorch → Core AI

Ben's session introduced the complete conversion workflow using the coreai-torch Python package. The process begins with loading a trained PyTorch checkpoint, exporting it via torch.export with dynamic shape annotations, running decompositions against Core AI's decomposition table, and finally converting through the TorchConverter. A critical step before leaving the Python environment is numeric verification, running identical inputs through both the PyTorch module and the Core AI inference function, then asserting a sufficiently small delta.

🔄 Swift Runtime: AIModel, InferenceFunction, NDArray

The Swift API surface was designed with progressive disclosure, simple cases require minimal code, but deeper layers of control are available for performance-critical applications. The core types form a clean hierarchy:

The Core AI framework is a new Swift API surface for loading and running Core AI models. It offers a progressively disclosing set of APIs, which makes it simple to get things up and running, while also having deeper layers of flexibility for supporting performance-critical applications. Also, it uses modern Swift language features like non-escapable types, to offer memory-safe APIs while not sacrificing performance.

Ben, Core AI Engineer

⚡ The Snake Game Case Study

Ben built a two-player Snake game where one snake is controlled by a SnakeTransformer model running through Core AI. The model receives features including normalized wall distances, food position, current direction, opponent distance, and opponent direction, then predicts the best movement direction via logit sampling. This seemingly simple demo exercises the same APIs that scale to 70-billion-parameter LLMs. The game initially suffered from progressive slowdown as sequence length grew, a textbook manifestation of transformer quadratic complexity.

🧠 KV Caching via States: The Performance Breakthrough

The key optimization was introducing key/value caches as Core AI states. States are model inputs that are both read and updated in-place during inference. By registering key and value cache tensors as PyTorch buffers (via register_buffer), exporting them as mutable buffers, and specifying them in the state_names argument of the converter, the model avoids recomputing embeddings for the entire history on each call. After the first full input, subsequent steps only pass the newest board state features:

The result: steady inference speed regardless of game length, confirmed by Instruments showing dramatically reduced latency growth.

🔬 The Core AI Developer Toolchain

There is also a convenient Core AI debug gauge which shows you streaming Core AI activity while your app is running in Xcode. This is a great place to spot performance issues before jumping into instruments.

Ben, Core AI Engineer

📦 Model Specialization: The Silent First-Launch Cost

When a .aimodel file ships with an app, it exists as a source representation, portable across devices, but not yet optimized. On first load, the model must be specialized for the specific device and OS version. This involves two transformations: (1) core compilation (segmenting, planning, optimizing compute graphs) and (2) executable artifact generation tied to specific compute units (CPU, GPU, ANE). Compilation accounts for the vast majority of specialization latency. Future loads hit a cache and are instant.

Best practices for managing this, as detailed by Ben:

• Never specialize during user-interactive flows. Gate features on cache presence; show preparatory UI while specializing.
• Use AIModelCache to programmatically check for cached models, request explicit specialization, manage entries, and share caches across app groups.
• Use coreai-build to pre-compile models ahead-of-time, slashing on-device specialization to a fraction of the original cost.
• SpecializationOptions provide fine-grained control over optimization targets.

🔌 The Core AI Models Repository Ecosystem

Beyond the core framework, Apple ships a Core AI Models repository that significantly accelerates development. The repo contains three major assets: a catalog of popular models with ready-to-run conversion scripts (SAM 3, Qwen, and a growing collection), Python utilities for model export and optimization, and Swift packages that provide higher-level runtime libraries. These libraries abstract away model-specific preprocessing and postprocessing, text encoding, mask extraction, tokenizer setup, behind clean Swift APIs like CoreAIImageSegmenter and CoreAILanguageModel. The repo also provides an API for creating a Core AI Language Model that directly plugs into the Foundation Models framework, allowing custom models and custom token sampling strategies to work through the same LanguageModelSession API as Apple's built-in models.

The Core AI Models repository is a great place to find what you need. It has a collection of popular models, each just a single command away from being converted and optimized for your app. AI skills that are experts in Core AI model authoring, optimization, and conversion. And a Swift package with libraries for specific families of models that give you higher-level APIs that already have many of those low-level inference optimizations built in.

Ben, Core AI Engineer

🎯 Advanced Inference Optimizations

Core AI On-Device Intelligence: SAM 3 + Qwen in Production

How Carina built a language-learning app with two models working in tandem, entirely on device.

With Core AI, you can build app experiences where user's data never leaves their device. There's no server to manage, no cost per token, and no latency to the cloud.

Carina, Core AI Team — On-Device Intelligence

🏗️ The Two-Model Architecture

Carina's language-learning app for Mandarin Chinese demonstrates a powerful pattern: decompose complex AI tasks into specialized models rather than reaching for a single monolithic solution. The app uses:

SAM 3 is a vision-transformer-based model for promptable image segmentation. It's a powerful model that does exactly what my app needs. A student points their camera at something and SAM 3 isolates the object according to their prompt precisely. It provides a clean cutout for the card graphic.

Carina, Core AI Team

📐 SAM 3 Close Reading: The 623MB Vision-Transformer

SAM 3 exposes three distinct functions, each visible in Xcode's model inspector:

• imageEncode. Converts a camera frame tensor into a dense feature embedding. Input: tensor with specific shape/dtype. Output: feature embedding.
• detect: Takes image features + text prompt, outputs raw masks, bounding boxes, and confidence scores.
• maskDecode: Refines masks into final segmentation output.

Rather than requiring developers to write preprocessing and postprocessing logic for each function, the Core AI Models repository provides runtime Swift libraries that abstract tensor wrangling behind clean APIs like CoreAIImageSegmenter.

🌐 Qwen: 119 Languages, On-Device Reasoning

Carina selected Qwen specifically for its combination of multilingual breadth (119 languages and dialects), reasoning capability (contextual examples, not just translations), structured output support (typed fields via @Generable), and compact variants suitable for on-device deployment. The 0.6B version fits alongside SAM 3 on iPhone, while the 8B version unlocks richer capabilities on Mac, including pinyin correctness verification and curriculum sequencing.

I did some quick tests and Qwen stood out — it supports one hundred nineteen languages and dialects, and it is a reasoning model, which means it can generate contextual examples, not just translations. A great starting point for vocab card generation.

Carina, Core AI Team

🚀 The FoundationModels Bridge

A critical architectural insight: Carina's custom Qwen model is accessed through the same LanguageModelSession API that Apple's own on-device model uses. By importing FoundationModels and creating a CoreAILanguageModel, the app gains all the framework's features, streaming responses, structured output via @Generable macros, and guided generation, without any custom networking or serialization code:

📦 Deployment Strategy: Background Assets + coreai-build

Carina's deployment strategy showcases mature thinking about production AI features:

1. Don't bundle >1GB of models with every app update, use Background Assets to download models only when users opt into the feature.
2. Dedicated first-run experience, explain the feature while models download and specialize behind the scenes.
3. coreai-build, pre-compile models ahead-of-time on the developer machine, producing architecture-specific compiled variants as separate Background Assets.
4. Runtime architecture detection, app detects device architecture and requests the appropriate pre-compiled asset, cutting specialization time dramatically.

🔄 Multiplatform Reuse

The same Core AI code, models, and APIs run identically on iOS and macOS. Carina's Mac version reuses all the original code, adds a batch processing layer, and swaps the 0.6B Qwen for the 8B variant, gaining richer prompts, multiple example sentences, pinyin generation, curriculum sequencing, and even pinyin self-verification, all through the same API calls with a more capable model underneath.

Foundation Models Framework: The Protocol That Unifies Everything

Christopher Webb unveils an open protocol that makes every LLM. Apple's, yours, or the community's, a first-class citizen.

We're opening up the framework to work with nearly any LLM, local or server-based. This allows anyone, from large companies to individual developers, to easily build their own model-integration on top of the framework.

Christopher Webb, Machine Learning Research

🏛️ The Four Model Tiers

Apple now offers developers four distinct model tiers, each accessed through the identical LanguageModel protocol and LanguageModelSession API:

🤝 Anthropic + Google: Official Swift Packages

Perhaps the most significant ecosystem announcement: Anthropic and Google will release official Swift packages that extend the Foundation Models framework with Claude and Gemini models. These third-party packages conform to the same LanguageModel protocol, meaning developers call them identically to Apple's own models:

Anthropic and Google will soon extend the Foundation Models framework with Swift packages of their own, making state-of-the-art Claude and Gemini models available to all Swift developers. Which ever model you use, Apple's, yours, or the community's, you call them the same way, because every model conforms to the Language Model protocol.

Christopher Webb, Machine Learning Research

📦 Open Source + Multiplatform Reach

The Foundation Models framework is being released as open source, with official support for iOS, macOS, visionOS, watchOS, and Linux. This means third-party packages can target server-side Swift deployments in addition to Apple's consumer platforms. Package distribution uses Swift Package Manager's decentralized model, a git tag on a public repository is all that's needed to publish a LanguageModel package that any developer can integrate by pasting a URL into Xcode.

🔑 Package Authoring: Four Steps to Integration

Christopher outlined a clear four-step process for model providers to bring their models into the Foundation Models ecosystem:

1. Packaging. Create a Swift Package with Package.swift; consider platform support; be mindful of dependency weight (every dependency is bytes shipped to users); publish via git tag.
2. Protocol Implementation. Define LanguageModel (capabilities + configuration) and LanguageModelExecutor (the engine that runs inference, manages resources, streams tokens).
3. Authentication: For server-based models: prefer token providers or sign-in flows over raw API key strings; persist tokens securely via Keychain; use App Attest for device attestation and tamper detection.
4. Customization: Extend with response metadata, custom segments (new modalities), and server-side tools.

LanguageModel Protocol Close Reading

The two-type protocol architecture that makes model interchange seamless.

The protocol is the bridge between your model, and the Foundation Models framework. The protocol has two key pieces. The first is LanguageModel. It describes the model to the framework. It declares what the model can do, through capabilities, and provides the configuration the framework needs to set up the model's EXECutor. The second piece is LanguageModelExecutor where the work happens.

Christopher Webb, Machine Learning Research

🔗 The Configuration-as-Key Pattern

The protocol's elegance emerges from a simple but powerful design: Configuration is Hashable, and each LanguageModelSession maintains an Executor store that maps unique configurations to exactly one Executor. When a model is used, the framework hashes its configuration, checks the store, and either reuses an existing Executor or creates a new one. This means:

• Two models producing the same configuration share the same Executor, reducing resource duplication.
• Different configurations get their own Executors, enabling per-model customization.
• When the session deallocates, all Executors are automatically released: deinit runs, weights are freed, connections close, no manual teardown code needed.

📋 The Six Transcript Entry Types

Your model defines its own roles. Your EXECutor's job is to map between the two, no matter the shape your model takes.

Christopher Webb, Machine Learning Research

⚙️ ContextOptions + GenerationOptions

Every request carries developer intent through two option types:

• ContextOptions. What goes into the prompt: reasoning level, response schema, tool enablement.
• GenerationOptions. How to decode: sampling strategy, temperature, maximum response length.

The Executor receives both on every respond() call and passes them through to the underlying inference engine. This separation allows the same model to produce dramatically different behavior, from creative free-form to strictly structured JSON, based purely on option configuration.

🔥 The prewarm Function

Before the first request arrives, developers can call prewarm() on a session to trigger expensive setup ahead of time, loading model weights, opening server connections, initializing KV caches. For server-backed models where setup is trivial, prewarm is a no-op. For local models with gigabytes of weights, it's the difference between a snappy first response and a loading spinner. Critically, prewarm is not guaranteed to run, the Executor must handle both the pre-warmed and cold-start paths gracefully, loading weights exactly once regardless of the path taken.

📊 The Response Handshake Protocol

The Executor's response stream follows a deliberate ordered handshake:

1. Metadata update. Model and request IDs for logging/debugging.
2. Usage update: Prompt token counts for accounting, sent upfront so developers don't wait through the entire stream for cost information.
3. Text deltas: Token-by-token streaming as the model produces them, giving users word-by-word display.

All responses are implemented as streaming; the one-shot API simply collects deltas internally.

Custom Segments & Server-Side Tools

Extending the protocol for new modalities and server-powered capabilities.

🧩 Custom Segments: Protocol Extensibility

Custom segments allow model providers to define entirely new data types that flow through the transcript alongside text. They must conform to PromptRepresentable, meaning developers can pass them directly in prompts just like strings. When a model responds, the Executor emits results back as custom segment updates through the same channel, the segment ID controls whether it's a new segment or an update to an existing one being streamed. This pattern supports audio, video, and any future modality without developers leaving LanguageModelSession.

Custom segment types let you extend the protocol. When a new modality comes along — audio, video, whatever's next — developers have a typed, structured way to send that data to your model.

Christopher Webb, Machine Learning Research

🛠️ Server-Side Tools: Three Surfacing Levels

Server-side tools are capabilities that the model invokes and the server executes, web search, code execution, image generation. The developer constructs the model with the tools they want, and they arrive on every request through the same capability declaration mechanism. Christopher described three levels of surfacing tool results:

Through one channel, the events you forward, the metadata you attach, and the custom segments you design, server-side tools shape what apps using your package can show their users.

Christopher Webb, Machine Learning Research

🔐 Authentication Best Practices

Christopher devoted significant time to authentication design, emphasizing that package authors must make it easy for developers to do the right thing. Key principles:

• Avoid raw API key strings in initializers, they encourage insecure patterns.
• Offer token providers or sign-in flows instead.
• Persist access tokens securely using Keychain.
• For cloud-based packages, use App Attest for device attestation: verifying devices, catching tampered builds, signing payloads, and leveraging Apple's fraud signal to keep bad traffic off your service.

⚠️ Error Handling: Built-in vs. Custom

Foundation Models ships LanguageModelError with predefined cases for common failure modes: context window overflows, rate limits, refusals, and more. Package authors should prefer these built-in errors when applicable, any developer familiar with the framework already knows how to handle them. Custom errors should be reserved for failures that are genuinely specific to a particular service (subscription tiers, account states, feature gates).

The Executor's responsibility extends to handling impossible requests gracefully. When a developer sets contradictory options, a strict token limit combined with a schema that requires more output than the limit allows, the Executor cannot satisfy both constraints. In such cases, throwing a LanguageModelError is the correct response. The framework's error cases cover context window overflows, rate limits, content refusals, and several other common scenarios that span across model providers and deployment types.

Custom errors are powerful, and sometimes you need them. But each one is a new case developers must learn, catch, and handle in their app. Try to use a built-in LanguageModelError when it fits, and save the custom ones for failures only your service can produce.

Christopher Webb, Machine Learning Research

🔄 Transcript Differencing: Stateful EXECutor Design

A critical implementation detail for stateful EXECutors: the framework provides the full transcript on every call to respond(), not just new entries. The EXECutor must compare the new transcript against the one it saved from the previous call. In the common case, where new entries are simply appended (a new prompt after the last response), the EXECutor can preserve existing state (KV caches, persistent connections) and only process what's new. But when the comparison detects removed or modified entries (e.g., the developer trimmed older entries to save context), the EXECutor must invalidate back to the divergence point. The framework leaves the comparison logic entirely to the EXECutor implementer: what counts as a match and how to handle changes are custom decisions that can be optimized for specific model architectures and service characteristics.

Advanced App Intents: Siri Customization

Antonio Cancio shows how to make Siri feel polished, personal, and unmistakably yours.

Today, you'll learn techniques to take the experience of your app in Siri and Apple Intelligence beyond the basics, so it feels polished, personal, and unmistakably yours.

Antonio Cancio, App Intents Engineer

💬 IntentDialog: Custom Responses That Match Your Brand

Antonio introduced IntentDialog as the mechanism for customizing what Siri says after executing an intent. By conforming to ProvidesDialog, an intent's perform() method returns an IntentResult containing an IntentDialog with both full and supporting strings. On visual devices, Siri shows the shorter supporting string with UI; on voice-only devices like AirPods, Siri reads the full dialog, so the full string must describe what happened on its own.

❓ Dialog Requests: Clarifying Questions During Execution

For intents that need to ask questions mid-execution, the perform() method can issue a dialog request before returning the final result. Common patterns include requesting a missing optional parameter value, asking the user to choose from a list of items, or requesting confirmation. Antonio's example: when creating a timer in CosmoTunes while another is already running, the intent requests a label for the new timer to avoid confusion.

A well-placed clarifying question lets people finish the action they meant to take.

Antonio Cancio, App Intents Engineer

🎨 Entity DisplayRepresentation + Custom View Snippets

Two visual customization mechanisms work together:

• Entity DisplayRepresentation. Defines how an entity looks across the system: title (required), subtitle and image (optional). Used in Siri responses, disambiguation lists, Spotlight, Shortcuts, and confirmation dialogs.
• Custom View Snippets. Per-intent SwiftUI views returned via ShowsSnippetView. Antonio's playlist snippet shows details in the app's familiar brand colors.

🤝 Interaction Donations: Teaching Siri About UI Behavior

Siri automatically learns from actions taken through Siri and Shortcuts, but it cannot learn from UI interactions without developer help. Interaction donations bridge this gap. Each donation is a schema-conforming App Intent representing a real user action in the app's UI, stored in a temporary transcript that gives Siri context for smarter decisions.

In UnicornChat, when a user sends a message from the compose view, the app donates a SendMessageIntent via IntentDonationManager. After repeated donations, Siri learns which app to use for a given contact. The pattern extends to ongoing activities, a Maps app donating NavigationSession interactions lets Siri know about active navigation when the user asks to add a stop.

When people interact with your app through Siri or Shortcuts, the system already knows about it. But Apple Intelligence can't learn from actions people take through your app's UI without your help. That's where donations come in.

Antonio Cancio, App Intents Engineer

✅ Confirmation Flow + OwnershipProvidingEntity

Siri can automatically confirm intents that may have meaningful side effects, canceling events, modifying shared content, updating public data. By default, Siri assumes entities are private and may skip confirmations for them. To tell Siri an entity has been shared or made public, conform it to the new OwnershipProvidingEntity protocol. Only add this protocol to entities that people can actually share in your app. The ownership state must be kept up to date whenever the system requests the entity.

Giving people a chance to confirm actions when appropriate builds trust in your app's experience with Siri.

Antonio Cancio, App Intents Engineer

Content Discovery: Semantic Index & Structured Search

Three paths for Siri to find your app's content, and when to use each.

There are three paths I want to cover: the semantic index, structured search, and in-app search.

Antonio Cancio, App Intents Engineer

📇 Path 1: Semantic Index (IndexedEntity + CSSearchableIndex)

For locally available, relatively static content (like playlists in CosmoTunes), indexing entities in Spotlight via CSSearchableIndex.indexAppEntities() is the preferred approach. This populates the Spotlight semantic index, enabling Apple Intelligence and Siri to understand entities based on meaning, not just exact keyword matches. The index also powers Spotlight search UI and, depending on the App Intents domain, provides semantic search capabilities that go beyond simple text matching.

Critical maintenance requirements:

• Index new entities as users add content.
• Update existing entries when key properties change (especially DisplayRepresentation fields).
• Delete entries when content is removed.
• Support reindexing via IndexedEntityQuery for when Spotlight requests a full refresh.

🔍 Path 2: Structured Search (IntentValueQuery)

When content is too large to index, lives on a server, or changes too frequently, IntentValueQuery provides an on-demand alternative. Unlike EntityQuery, it receives a structured search input from the system and can return multiple entity types. Antonio demonstrated this with AudioSearch in CosmoTunes:

🔎 Path 3: In-App Search (.system.searchInApp)

For apps that want to route search results through their own custom UI, the .system.searchInApp schema (formerly .system.search in iOS 17) lets Siri call an intent with the same search string, and the intent's perform() method finds and displays results in the app. This works regardless of entity indexing and alongside any other domain adoption.

Onscreen Awareness: Connecting UI to Intelligence

How your app teaches Siri what's visible on screen, and what users can do with it.

Onscreen awareness is how your app connects what's visible on screen to structured information and actions the system understands. Siri can then resolve references like 'Play the third one' or 'that conversation' without the person naming them explicitly.

Antonio Cancio, App Intents Engineer

🖼️ Four Annotation APIs

Apple provides four tiers of onscreen awareness annotations, each suited to different UI patterns:

Collection annotations help me avoid the overhead of attaching an annotation to every single row. Instead, the system fetches identifiers lazily, as it needs them. Collection annotations also let Siri discover entities that have been selected and scrolled off screen.

Antonio Cancio, App Intents Engineer

⚡ Performance: DisplayRepresentation Querying

A critical performance optimization: when Siri evaluates onscreen entities against a request (e.g., "play the third one"), it needs to quickly determine relevance. By implementing displayRepresentations on entity queries, Siri can retrieve just the text representation, skipping the overhead of fetching full entities from the database. Without this optimization, Siri may fall back to asking for clarification, causing users to abandon the request.

🔄 Cross-Framework Support

All onscreen awareness APIs are available across SwiftUI, UIKit, and AppKit. UIKit-specific APIs include AppEntityAnnotatable, UICollectionViewAppIntentsDataSource, and appEntityUIElementProvider. These annotations also power contextual menu items in UIKit apps.

Entity Annotations: Notifications, NowPlaying, Alarms

Making your app's entities the universal language of the system.

With this added context, your app entities act as a universal language. They let Siri understand not just what's on screen, but how other system integrations and time-sensitive events relate to your content.

Antonio Cancio, App Intents Engineer

🔔 Three Integration Points, One Pattern

Entity annotations follow a consistent pattern across all three system integrations: assign persistent EntityIdentifier values to an appEntityIdentifiers property on the relevant system object. Transient entities cannot be used, they lack persistent identifiers needed for system-wide references.

When a UnicornChat notification is announced on AirPods I can say: 'Reply, ok, I'll swing by the unicorn supply store and pick those up.'

Antonio Cancio, App Intents Engineer

For NowPlaying specifically, the identifier ordering matters: entities should be listed from most specific to least specific (e.g., song → artist → playlist). This ordering helps Siri prioritize context when interpreting ambiguous requests like "play the live version."

SpotlightSearchTool: LLM-Powered Search for Every App

Jennifer unveils a tool that lets language models directly search your app's Core Spotlight index.

Today, we're introducing SpotlightSearchTool. It's a tool that adopts the tool protocol, to let a language model directly search your app's content in Core Spotlight for contextual response generation.

Jennifer, Spotlight Engineering

🧰 How SpotlightSearchTool Works

SpotlightSearchTool implements the Tool protocol from Foundation Models, enabling language models to generate search queries against an app's Core Spotlight index. The trajectory follows a tool-calling pattern: the model decides it needs to search, generates a query, Spotlight executes it and returns result descriptions, and the model reasons over those results to generate its final response. The tool is available on iOS, iPadOS, macOS, and visionOS.

Prerequisites for adoption:

1. App must donate searchable content to Core Spotlight (or index entities for Apple Intelligence).
2. Configure the tool for the desired search kind (default index-based, or custom sources like FileSource).
3. Choose a model : SystemLanguageModel or any custom model via Model Provider APIs.
4. Add the tool instance to LanguageModelSession.

📋 Recovering Full Metadata: searchableItems(forIdentifiers:)

Some Spotlight index metadata (text content, HTML) is stored in highly-compact representations that can be searched but not recovered in a language-model-readable format. To bridge this gap, SpotlightSearchTool adds a new method to the index delegate: searchableItems(forIdentifiers:). This returns the complete CSSearchableItem for each identifier, allowing the model to efficiently reason over potentially millions of results. Apps can also attach additional metadata at this point, data that's useful for model reasoning but doesn't need to be in the search index.

We've added a method to the delegate to recover the full CSSearchableItem by its unique identifier. This allows the model to efficiently manage responses over potentially millions of results.

Jennifer, Spotlight Engineering

🖥️ Display Patterns: Search Replies

Results are accessible through two surfaces: the session response (concise description over result set, ideal for assistant-style interfaces) and search replies (an async sequence of events, each potentially containing a batch of CSSearchableItem results). Search replies use queryToken values to manage conversation stream updates, ensuring the UI stays current. Since a model may call SpotlightSearchTool multiple times before generating its final response, tokens prevent incorrect UI refreshes.

🎛️ Customization: Guidance Profiles & Reference Resolution

SpotlightSearchTool provides its entire set of search capabilities to the model by default, semantic search over text, structured search over metadata (dates, persons, locations, and more). However, not every app needs every capability. The hiking trails app doesn't donate person relationships, so guiding the model on searching for authors and recipients wastes precious context window space on limited-context models. GuidanceProfile lets developers selectively enable guidance on specific search capabilities and even specify the exact list of metadata attributes the model should consider, with a dynamic guide level set when creating the tool.

For reference resolution, the contactResolver helps ground ambiguous references. When a user mentions "other participants on the trail," the model needs to know who the user is. By providing a contact resolver that returns identity-related contact information, the app enables SpotlightSearchTool to match against person metadata in the search index, bridging natural language references to structured index data.

Pipeline Stages & Custom Computation

When search alone isn't enough, teaching Spotlight to compute, not just retrieve.

A pipeline search brings together queries to the index, plus computation over a result set, for maximal efficiency. Pipeline stages allow the tool to perform efficient computation, or transformation, over a search result set on behalf of the model.

Jennifer, Spotlight Engineering

🔗 The Pipeline Pattern

For complex queries that involve aggregation or computation over search results. "How many trails have I hiked this year, and for each month, how many miles have I gone on average?", the model can request a pipeline search: a sequence of search and computation stages executed by Spotlight. The model generates a search for completed hikes, a counting stage that builds a table by month, and an averaging stage over those counts. This offloads computation from the model's limited context window to Spotlight's efficient data processing.

🏗️ Custom Pipeline Stages

Apps can register their own pipeline stages. Stages are Generable types, the model generates them on-demand based on the user's prompt, using @Guide property annotations to inform the model about which results to prefer.

Jennifer's example: a happiness score stage that computes sentiment over hike notes. The stage operates on CSSearchableItem inputs, runs a sentiment analysis model (or custom logic like 5-star rating checks) over the notes attribute, and returns scored versions as output. The model then generates responses solely from the top-scoring results. Stage registration is a single line: add the stage to the tool's configuration.

And each reply comes with a handy LLM-generated label describing the content, giving your app the most flexibility for its user interface.

Jennifer, Spotlight Engineering

📊 Pipeline Output Display

When a pipeline stage produces results, the model may send back a search reply containing the stage's output data as partial results, aggregate counts, tables, free-form text, or computed numeric values, each with an LLM-generated label. This gives apps maximum UI flexibility: they can display any or all of these data types in whatever form suits their design.

Evaluations Framework: Result Coverage & Sample Generation

Systematic testing of LLM-powered app experiences, from data generation to metric-driven validation.

Not only can we quickly build evaluations to see how well the model is calling the tool, and how meaningful the response; we can also rapidly iterate on our app's searchable content paired with different guidance profiles on SpotlightSearchTool itself.

Jennifer, Spotlight Engineering

🧪 Building an Evaluation Suite

The Evaluations framework provides APIs for end-to-end evaluation: dataset generation, evaluation runs with custom metrics, and reporting. Jennifer focused on result coverage as the primary metric, given a dataset indexed in Core Spotlight, how well does the model generate responses based on the items we expect it to find?

📊 The Evaluation Flow

1. Define hiking trail data, metadata that the app is expected to donate to Core Spotlight.
2. Build seed samples, adopting ModelSampleProtocol. Each sample includes: natural language query, expected output response, expected tool-call trajectory, and a set of unique identifiers for items the tool should return.
3. Expand via Sample Generation APIs, from a command-line tool, expand a small seed set into many variations for broad coverage of how users might phrase questions.
4. Run evaluation, load trail items and samples from generated datasets; donate items to Core Spotlight; configure SpotlightSearchTool; run evaluation.
5. Measure result coverage, how many expected items appeared in the final response.

We're not writing search queries anymore. We're providing the content, and letting intelligence do the rest.

Jennifer, Spotlight Engineering

Cross-Cutting Themes: Privacy, Multiplatform, Performance

The principles that unify all five sessions.

🔒 Privacy as Architecture

Privacy is not a feature bolted onto Apple's AI stack, it's a structural property. On-device inference via Core AI means user data never leaves the device. Private Cloud Compute extends Apple's privacy guarantees to server-scale inference with a 32K token context and reasoning capabilities. Christopher explicitly called out the privacy responsibility for third-party package authors: developers and users must understand whether a model runs on-device or in the cloud, as the privacy characteristics differ fundamentally.

Whether you're choosing a package or shipping one, make sure everyone in the chain understands the privacy implications of the model behind it. On-device and cloud-based models have very different privacy characteristics, and your users deserve to know which they're getting.

Christopher Webb, Machine Learning Research

🌍 True Multiplatform

⚡ Performance Patterns

Across all five sessions, consistent performance patterns emerge:

• Pre-warm and pre-specialize. Use prewarm() for model setup, coreai-build for ahead-of-time compilation, and AIModelCache for specialization management.
• Cache aggressively. KV caches (Core AI states), Executor stores (Configuration Hashable), and specialization caches (device-specific).
• Lazy-fetch where possible. Collection annotations fetch identifiers lazily; display representation querying avoids full entity database fetches.
• Batch and pipeline. Pipeline stages offload computation from model context windows; batch processing maximizes throughput on Mac.
• Use progressive APIs. Simple paths for common cases; lower-level APIs (optimal memory layouts, pre-allocated outputs, async values) for hot inference loops.

🔄 The Iterative Development Model

Ben emphasized that Core AI is designed to support fast, iterative cycles. The Python→PyTorch→Core AI conversion pipeline reuses familiar tools. The numeric verification step catches conversion errors before they reach Xcode. Instruments and the Core AI Debugger provide progressive depth, from the Debug Gauge's streaming overview to the Debugger's Python source trace. The Evaluations framework extends this iteration to LLM-powered features with automated, metric-driven testing.

Core AI is designed to support that iteration and make it as fast and frictionless as possible.

Ben, Core AI Engineer

📱 Platform Performance Gains & Reach

The iOS 27 and macOS 27 releases bring substantial performance improvements across the board. Apple reported app launch times 30% faster, Photos operations 70% faster, AirDrop transfers 80% faster, and iPad performance improvements of up to 5× in specific workflows. Critically, iPhone 11 and later support iOS 27 thanks to a new CPU scheduler backport, dramatically expanding the addressable market for Core AI-powered features. With over 1,000 App Store submissions processed per hour and 20 Apple Developer Academies worldwide, the developer ecosystem is primed for widespread AI adoption. The Foundation Models framework's open-source release and Linux support further extend the reach, enabling server-side Swift deployments alongside consumer platforms.

🤖 The Agentic Future

Several sessions point toward agentic AI as the next horizon. Ben referenced "agentic assistant powered by a 70 billion parameter LLM" running locally. Christopher mentioned "Build agentic app experiences with the Foundation Models framework" as a dedicated session showing multi-step, tool-using workflows. The combination of Core AI inference, Foundation Models tool-calling, SpotlightSearchTool pipeline stages, and App Intents system integration creates the infrastructure for apps that don't just respond to queries but autonomously execute complex, multi-step tasks, all with privacy-preserving on-device execution where possible, and Private Cloud Compute for server-scale reasoning when needed.

Synthesis: Apple's Developer AI Strategy for 2026 and Beyond

What these five sessions reveal about Apple's long-term AI platform vision.

🧩 The Complete Developer AI Lifecycle

WWDC 2026's developer sessions paint a picture of a platform that has matured from individual AI capabilities into a coherent, end-to-end developer experience. The lifecycle now spans:

🎯 Strategic Implications

🔮 What Comes Next

These sessions position developers for a future where AI is not a specialized discipline but a standard tool in every app developer's kit. The key enablers are already in place: a unified inference runtime (Core AI), a protocol-driven model abstraction (Foundation Models), deep system integration (App Intents + SpotlightSearchTool), and systematic evaluation (Evaluations framework). The platform effect is beginning to compound, each component strengthens the others.

We can't wait to see what sorts of experiences you build.

Ben, Core AI Engineer — closing Introduction to Core AI

We hope to see a thriving ecosystem of LanguageModel packages, giving Swift developers the freedom to choose the model that's right for their app. We can't wait to see what you build.

Christopher Webb, Machine Learning Research — closing Foundation Models Framework

With the 27 releases, Apple Intelligence is transforming what Siri can do, and App Intents puts that transformative power directly in your hands. You now have everything you need to build delightful, elevated experiences that feel like a natural extension of the system.

Antonio Cancio, App Intents Engineer — closing Advanced App Intents

We're not writing search queries anymore. We're providing the content, and letting intelligence do the rest.

Jennifer, Spotlight Engineering — closing SpotlightSearchTool

The models are ready. The tools are ready. With Core AI you have everything you need to bring powerful, private intelligence to every Apple platform. Now, let's go build something powerful on device!

Carina, Core AI Team — closing On-Device Intelligence