Return tokens to emitted symbols on EmitResult

[tmat]

Background and Motivation

Scenario:
Host application creates a C# compilation, calls Emit and loads the resulting PE image as Assembly.
The host then wants to access a runtime entity that corresponds to a source symbol of the compilation.

E.g. the host specifies that the source code shall define a top-level static local function of certain name and signature. Once the Assembly is loaded it wants to invoke the runtime method that implements the local function. Or the host specifies that the source code shall define a type implementing a certain interface. It then needs to find the corresponding Type, instantiate it and call its method via the interface. Etc.

Roslyn does not provide APIs that would make it easy to resolve runtime entities for a given ISymbol.

The new API would enable cleaner and simpler implementation of Semantic Search feature: https://github.com/dotnet/roslyn/blob/main/src/Features/Core/Portable/SemanticSearch/AbstractSemanticSearchService.cs#L327

Proposed API

A new overload of Emit method that takes and additional tokenRequests parameter.

namespace Microsoft.CodeAnalysis;

public class Compilation
{
+    public EmitResult Emit(
+         Stream peStream,
+         Stream? pdbStream = null,
+         Stream? xmlDocumentationStream = null,
+         Stream? win32Resources = null,
+         IEnumerable<ResourceDescription>? manifestResources = null,
+         EmitOptions? options = null,
+         IMethodSymbol? debugEntryPoint = null,
+         Stream? sourceLinkStream = null,
+         IEnumerable<EmbeddedText>? embeddedTexts = null,
+         Stream? metadataPEStream = null,
+         IEnumerable<ISymbol>? metadataTokenRequests = null,
+         CancellationToken cancellationToken = default(CancellationToken))
}

public class EmitResult
{
+      /// <summary>
+      /// If <see cref="Success"/> is true, returns metadata tokens emitted for the symbols specified in
+      /// metadataTokenRequests parameter of the Emit method. 
+      /// If the specified symbol does not have a token in the emitted image the corresponding item in this array will be 0.
+      /// </summary>
+      public ImmutableArray<int> RequestedMetadataTokens { get; }
}

If the specified ISymbol is a type, method or local function, field, property, event, parameter, generic parameter symbol defined in the source of the compilation the token will be a definition token: TypeDef, MethodDef, FieldDef, Property, Event, Param, GenericParam, respectively.
If the specified ISymbol is a type symbol that is referenced from the emitted metadata or IL the token will be a reference token (AssemblyRef, TypeRef, MemberRef, TypeSpec, MethodSpec).
Otherwise (local variable, using alias, PE symbol that is not referenced, etc.), the token is 0.

Usage Examples

var result = compilation.EmitResult(peStream)

using var peStream = new MemoryStream();

var emitResult = queryCompilation.Emit(peStream, options: emitOptions, tokenRequests: [functionSymbol], cancellationToken: cancellationToken);

if (!emitResult.Success)
{
     return;
}

peStream.Position = 0;
var assembly = AssemblyLoadContext.Default.LoadFromStream(peStream);    
var functionToken = emitResult.RequestedMetadataTokens[0];
var info = assembly.ManifestModule.ResolveMethod(functionToken);
  
info.Invoke(...);

Alternative Designs

We can limit the implementation to definition tokens only.

Risks

[tmat]

@333fred FYI

[333fred]

API Review

• Could we just not use local functions here to make it easier?
  • Would require more wrapping of C# that we don't want to introduce.
• Tokens aren't really understood by the compiler, and it seems odd that we'd return them.
• This seems like it would be a significant chunk of work
• There are other cases that would need to be considered, such as anonymous function symbols.

Conclusion: Needs work. We'll work offline to determine the future of the proposal.

[tmat]

Tokens aren't really understood by the compiler, and it seems odd that we'd return them.

We do have ISymbol.MetadataToken public API that returns token of a PE symbol, or 0 if the symbol isn't a PE symbol with a token.

The proposed API is essentially similar to the existing API, it would instead work on symbols defined in source. I agree we should not try to implement it for symbols that are emitted as implementation details. A rule of thumb could be to support it for definitions that are named in the code and have tokens (named type, method or local function, field, property, event, parameter, generic parameter symbol).

The Semantic Search scenario that uses local function is just one hosting scenario. In general hosting scenario a host application uses Roslyn compilation to emit assembly, then loads the assembly and then interacts with the loaded entities via Reflection. The mapping between ISymbol and the emitted metadata might get complicated. E.g. when the host needs to distinguish 2 method overloads, or needs to find a symbol whose name is mangled (local function, explicit interface implementation, etc.).

Note also that EmitDifferenceResult, which derives from EmitResult and is used for EnC returns handles (tokens) to the methods and types it emitted.