Support generating non-source files

[geeknoid]

We have a few use cases where we'd like to leverage source generators to produce non-source files. Basically, we want to use source generators to generate secondary build outputs. For example:

• We annotate data models with data classification attributes to denote PII and we produce a CSV file that enumerates the types and members in the project and their data classification. You can then take this CSV file and use it when doing privacy audits.

• We use source generators to produce a bunch of code around metrics. We'd like to emit a schema file describing the effective shape of the metrics being produced by the code. This would probably be in the form of an output JSON file.

We have a few other scenarios in the wings which could leverage this approach.

Unfortunately, source generation is currently limited to C# sources. Trying to emit files with other extensions doesn't work. As a result, our source generator currently contains this horrible hack:

        public void Execute(GeneratorExecutionContext context)
        {
            ... generate the CSV file ...

            context.CancellationToken.ThrowIfCancellationRequested();

            /// Adding the workaround to generate reports via File writes since <see cref="GeneratorExecutionContext.AddSource(string, CodeAnalysis.Text.SourceText)"/>
            /// has an underlying check for `.cs` files and automatically adds the `.cs` suffix if the provided filename has some other filetype.
            /// Refer <see href="https://github.com/dotnet/roslyn/blob/v3.8.0/src/Compilers/Core/Portable/SourceGeneration/AdditionalSourcesCollection.cs#L63">AppendExtensionIfRequired</see>.
            _directory ??= FileUtilHelpers.GetOutputDirectoryForGeneratedFile(typeof(Generator), context.Compilation.Assembly, context.Compilation.Options.OptimizationLevel);
            _ = Directory.CreateDirectory(_directory);

            // Write properties to CSV file.
            File.WriteAllText(Path.Combine(_directory, _propertiesFileName), properties);

            // Write log methods to CSV file.
            File.WriteAllText(Path.Combine(_directory, _logMethodsFileName), logMethods);
      }

Could we get 1st class support for producing more than .cs files?

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ryzngard]

Are these outputs also included in the final compilation somehow, or are they just independent artifacts of the build?

[geeknoid]

They are secondary outputs of the build process, just like the .xml doc file produced for XML doc comments. If C# didn't support XML doc comments, you could in fact implement them fully as a source generator.

Ideally, the output files would be produced in the same directory as the final output assembly, rather than being buried somewhere in the obj directory. But I can live with the obj directory output.

[lsoft]

probably, something related? #52677

[vpenades]

I agree with this proposal. in my case I would need something as simple as this:

void AddContent(string fileName, byte[] content); // add file as a content item

Notice that in my case, I also need that the output file to be considered as a Content item, so it also needs to handle CopyToOutputDirectory, pack to nuget as a content file, etc... and not limited to text files

In the grand scale of things, I would like to have is a framework similar to source generator, but for content files... basically a "content generator". I am not sure if this can be achieved by simply adding an "AddContent" method, or it would require a completely separated, and more specialised framework.

A practical use case scenario from which we could benefit: We have a localization framework that parses the source code looking for specific localization attributes. Then it queries our database for localized strings, and writes an XML file containing the attribute keys and the localized strings. This is now running as a command line tool, which is nasty.

[AaronRobinsonMSFT]

This is also of interest to the Interop team and the DllImport source generator.

We would like to emit artifacts that could be used in other languages and likely used in a post-compile MSBuild Target. Roslyn could give us a Stream instance to use or we could call an API that takes a Stream. Either approach works for the data stream. The name of the output artifact file would also be important. The specific path isn't interesting coming out of the compiler, since we would move/copy it to another location, but the name would be helpful.

From within MSBuild, it would be helpful to either have a convention of a new set of Items (e.g., <GeneratorName>Artifacts) that we could then consume. I would caution against requesting an Item name from the generator for the typical name conflict issues.

/cc @dotnet/interop-contrib @chsienki @jaredpar @kg

[chsienki]

@AaronRobinsonMSFT This is basically the design we landed on for the feature :)

We actually have a (somewhat out of date) PR to implement this on the analyzer side #49046 but it's been stuck in limbo due to lack of prioritization.

Feel free to bug @jaredpar if this is something you think is important to the success of the DllImport generator.

[stephentoub]

As a +1, I'd also like this for the RegexGenerator. I'd like to be able to output non-compiled files that a developer could browse to in solution explorer to be able to provide additional diagnostic information about the regular expressions being used. Imagine, for example, if in addition to outputting RegexGenerator.g.cs that included the source code, it could also output a .dgml file containing a graph of the DFA for the regex, and a developer could simply double-click that .dgml file to have it opened in VS to visualize the shape of their regex. Today the best I can do here seems to be emitting that dgml content as a comment embedded in the .cs file file, at which point a developer needs to find it, copy/paste it to a separate file, and then open that file they created (and I need to somehow escape the content in a way that it doesn't break the .cs compilation but is still valid dgml).