[Feature] Add Microsoft.Maui.Essentials.AI - Apple Intelligence

[mattleibow]

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Description

This PR introduces Microsoft.Maui.Essentials.AI, a new library that provides cross-platform AI capabilities through the Microsoft.Extensions.AI abstractions. This enables .NET MAUI developers to leverage on-device AI features while maintaining a consistent programming model across different AI backends.

What's Added

Core Library (Microsoft.Maui.Essentials.AI)

Apple Intelligence Chat Client (iOS 26+, macOS 26+, Mac Catalyst 26+, tvOS 26+)

AppleIntelligenceChatClient - Full IChatClient implementation for Apple's FoundationModels framework:

Feature	Description
Synchronous Responses	GetResponseAsync() for complete responses
Streaming Responses	GetStreamingResponseAsync() returns IAsyncEnumerable<ChatResponseUpdate> for real-time UI updates
Function/Tool Calling	Register AIFunction tools that the model can invoke; automatic JSON argument parsing and result handling
Structured JSON Output	Use ChatResponseFormatJson with JSON Schema to enforce typed responses matching your C# models
Generation Options	Configure Temperature, TopK, Seed, and MaxOutputTokens
Cancellation Support	Full CancellationToken support with proper native task cancellation
Middleware Compatible	Works with IChatClient pipeline builders (.AsBuilder().UseLogging().Build())

Embedding Generator (iOS 13+, macOS 10.15+, Mac Catalyst 13.1+, tvOS 13+)

NLEmbeddingGenerator - IEmbeddingGenerator<string, Embedding<float>> implementation using Apple's NaturalLanguage framework:

Feature	Description
Sentence Embeddings	Generate vector embeddings for text using NLEmbedding
Language Support	Specify language via NLLanguage or use default (English)
Thread Safety	Internal semaphore ensures thread-safe access to underlying NLEmbedding
Wrap Existing	Constructor accepts existing NLEmbedding instance

Extension Method:

• NLEmbedding.AsIEmbeddingGenerator() - Wrap existing NLEmbedding as IEmbeddingGenerator<string, Embedding<float>>

Internal Stream Processing (not public API):

• JsonStreamChunker - Converts complete JSON snapshots into incremental streaming deltas for smooth UI updates
• PlainTextStreamChunker - Handles plain text streaming with simple delta computation

---

Capabilities

Feature	Status	Notes
Chat Completion	✅ Supported	Full non-streaming via GetResponseAsync()
Streaming Responses	✅ Supported	IAsyncEnumerable<ChatResponseUpdate> via Channels
Text Embeddings	✅ Supported	NLEmbeddingGenerator via NaturalLanguage framework
Function/Tool Calling	✅ Supported	AIFunction tools with JSON argument parsing and result handling
Structured JSON Output	✅ Supported	ChatResponseFormatJson with JSON Schema via JsonSchemaDecoder
System Prompts	✅ Supported	Via ChatMessage with ChatRole.System or ChatOptions.Instructions
Multi-turn Conversation	✅ Supported	Full message history preserved
Image Input	❌ Not Supported	FoundationModels is text-only
Temperature	✅ Supported	Via ChatOptions.Temperature
TopK	✅ Supported	Via ChatOptions.TopK
TopP	❌ Not Supported	Not exposed by FoundationModels
Seed	✅ Supported	Via ChatOptions.Seed for reproducible outputs
Max Output Tokens	✅ Supported	Via ChatOptions.MaxOutputTokens
Cancellation	✅ Supported	Full CancellationToken with native Swift task cancellation

---

Architecture: Swift Native Layer & C# Integration

The Apple Intelligence integration uses a two-layer architecture to bridge .NET with Apple's Swift-only FoundationModels framework:

┌─────────────────────────────────────────────────────────────────┐
│                    .NET MAUI Application                        │
│  IChatClient / IEmbeddingGenerator<string, Embedding<float>>    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                   C# Layer (Essentials.AI)                      │
│  AppleIntelligenceChatClient : IChatClient                      │
│  • Implements Microsoft.Extensions.AI abstractions              │
│  • Converts ChatMessage ↔ ChatMessageNative                     │
│  • Handles streaming via System.Threading.Channels              │
│  • Wraps AIFunction tools as AIFunctionToolAdapter              │
│  NLEmbeddingGenerator : IEmbeddingGenerator                     │
│  • Uses NaturalLanguage framework (Obj-C API, no Swift needed)  │
└─────────────────────────────────────────────────────────────────┘
                              │ (ObjC Binding)
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                Swift Layer (EssentialsAI)                       │
│  ChatClientNative                                               │
│  • Wraps FoundationModels.LanguageModelSession                  │
│  • Manages Swift async Tasks with cancellation                  │
│  ToolNative : FoundationModels.Tool                             │
│  • Bridges C# AIFunction → Apple's Tool protocol                │
│  JsonSchemaDecoder                                              │
│  • Parses JSON Schema → GenerationSchema                        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│              Apple FoundationModels Framework (iOS/macOS 26+)   │
└─────────────────────────────────────────────────────────────────┘

Why Swift is Required for Chat Client:

• Apple's FoundationModels framework is Swift-only with no Objective-C API
• Uses Swift-specific features: async/await, Sendable, result builders
• Swift code compiles into a static library bound via Xamarin binding definitions

Layer Responsibilities:

Layer	Responsibility
C# (AppleIntelligenceChatClient)	IChatClient implementation, type conversion, streaming orchestration
C# (NLEmbeddingGenerator)	IEmbeddingGenerator implementation using NaturalLanguage framework directly
Swift (ChatClientNative)	Wraps FoundationModels, manages async Tasks, native callbacks
Swift (ToolNative)	Bridges AIFunction → Apple's Tool protocol
Swift (JsonSchemaDecoder)	JSON Schema → GenerationSchema for structured output

Note: Embedding generators (NLEmbeddingGenerator) use Apple's NaturalLanguage framework which has Objective-C APIs, so no Swift layer is needed.

---

Why

1. Unified AI API: Write AI features using IChatClient/IEmbeddingGenerator abstractions—same code works with cloud AI (OpenAI, Azure) or on-device (Apple Intelligence)

2. On-Device Privacy: Apple Intelligence runs entirely on-device, keeping user data private

3. Future Platform Support: Architecture supports additional platforms (Android MLKit/Gemini Nano) as on-device AI expands

---

Samples

Sample	Description
Essentials.AI.Sample	.NET MAUI sample demonstrating chat client configuration and usage
AppleNative/EssentialsAISample	Native Objective-C sample demonstrating Swift framework interop

---

Test Coverage

Category	Test Files	Description
Unit Tests	56	JsonStreamChunker, PlainTextStreamChunker, BufferedChatClient, streaming roundtrip scenarios
Device Tests	2	AppleIntelligenceChatClient, NLEmbeddingGenerator on-device validation
Benchmarks	3	Streaming JSON deserialization performance

Device test coverage:

• Chat client: instantiation, messaging, streaming, JSON schema output, cancellation
• Embeddings: concurrency, similarity, disposal, language support

---

Public API Surface

Namespace: Microsoft.Maui.Essentials.AI

// Apple Intelligence Chat Client (iOS/macOS/MacCatalyst/tvOS 26+)
public sealed class AppleIntelligenceChatClient : IChatClient
{
    public AppleIntelligenceChatClient();
    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default);
    public IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default);
}

// Sentence Embedding Generator (iOS 13+, macOS 10.15+, MacCatalyst 13.1+, tvOS 13+)
public class NLEmbeddingGenerator : IEmbeddingGenerator<string, Embedding<float>>, IDisposable
{
    public NLEmbeddingGenerator();                           // Default: English
    public NLEmbeddingGenerator(NLLanguage language);        // Specify language
    public NLEmbeddingGenerator(NLEmbedding embedding);      // Wrap existing
    public Task<GeneratedEmbeddings<Embedding<float>>> GenerateAsync(
        IEnumerable<string> values,
        EmbeddingGenerationOptions? options = null,
        CancellationToken cancellationToken = default);
    public void Dispose();
}

Namespace: Microsoft.Extensions.AI (extension methods)

public static class NLEmbeddingExtensions
{
    public static IEmbeddingGenerator<string, Embedding<float>> AsIEmbeddingGenerator(
        this NLEmbedding embedding);
}

---

Usage Examples

// Chat with Apple Intelligence
var chatClient = new AppleIntelligenceChatClient();

// Simple response
var response = await chatClient.GetResponseAsync([
    new ChatMessage(ChatRole.User, "Hello!")
]);

// Streaming response
await foreach (var update in chatClient.GetStreamingResponseAsync(messages))
{
    Console.Write(update.Text); // Progressive output
}

// Structured JSON output
var options = new ChatOptions 
{ 
    ResponseFormat = ChatResponseFormat.ForJsonSchema<MyModel>() 
};
var structured = await chatClient.GetResponseAsync(messages, options);

// With middleware pipeline
var client = new AppleIntelligenceChatClient()
    .AsBuilder()
    .UseLogging(loggerFactory)
    .Build();

// Sentence Embeddings
var generator = new NLEmbeddingGenerator(NLLanguage.English);
var embeddings = await generator.GenerateAsync(["text to embed"]);

// From existing NLEmbedding
var nlEmbedding = NLEmbedding.GetSentenceEmbedding(NLLanguage.Spanish);
var generator = nlEmbedding.AsIEmbeddingGenerator();

---

Not in Scope (Future Work)

• Android platform implementation (MLKit/Gemini Nano)
• Windows platform implementation
• Additional embedding model types

[Copilot]

Pull request overview

This PR introduces Microsoft.Maui.Essentials.AI, a new cross-platform AI library providing Apple Intelligence integration through Microsoft.Extensions.AI abstractions. The PR adds comprehensive test infrastructure including unit tests for JSON/plain text streaming, test data files for various AI scenarios, and model classes for structured output testing.

Changes:

• Uncommented Debug mode PublicAPI generation in PublicAPI.targets
• Added extensive unit test suite for JsonStreamChunker and PlainTextStreamChunker (internal streaming classes)
• Added test data files (emotional responses, itineraries) in .txt and .jsonl formats
• Added model classes for Weather, Itinerary, Landmark, and PointOfInterest
• Added empty PublicAPI.Unshipped.txt for netstandard

Reviewed changes

Copilot reviewed 128 out of 132 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/PublicAPI.targets	Enabled Debug mode PublicAPI generation
PlainTextStreamChunkerTests/*.cs	Unit tests for plain text delta computation (basic, progressive, edge cases)
JsonStreamChunkerTests/*.cs	Unit tests for JSON streaming (basic, properties, nested, integration)
TestHelpers/DataStreamsHelper.cs	Helper for loading test data files
TestData/**/*.txt	Expected streaming output for various AI scenarios
TestData/**/*.jsonl	Progressive JSON snapshots for streaming tests
Models/*.cs	Test models for Weather, Itinerary, Landmark, PointOfInterest
PublicAPI.Unshipped.txt	Empty netstandard API tracking file

Files not reviewed (1)

• src/AI/src/AppleNative/EssentialsAI.xcodeproj/project.xcworkspace/contents.xcworkspacedata: Language not supported

[rolfbjarne]

I don't really know enough this to review the PR properly - so maybe just add the [Experimental] attribute in any new managed code until it's gotten some real world testing?

FWIW it's a rather big PR, GitHub even says so:

Due to the large number of changes in this pull request, only one file is being shown at a time.