Add trace events to scenarios where a certificate is loaded and used by an application to monitor for expiration

[joperezr]

It is unfortunately not uncommon for applications (particularly web applications) to fail whenever relying on certificates that expire. This can happen both for certificates that are specific to the application itself, or certificates of other services that our application is trying to call into. If the .NET platform started emitting some sort of telemetry each time a certificate is used, and amongst other things, included the expiration date of that certificate, then components can be built on top of that telemetry which will be able to configure filters and then set up monitoring alerts so the application developers can get notified ahead of time and prevent an issue that might take the application down.

It is important to try to define what "using" a certificate means and which scenarios are the ones that developers would care about, since certificates get used a lot by applications these days on scenarios that some might not care about, for instance, certificates will get loaded each time the application makes an https request. This means that defining the scenarios that a developer might care about could help reducing potential "noise".

Here are some initial goals that we would like to enable for application developers with this:

• Provide the ability to observe certificates used by dotnet runtime in the context of an application, including usage metadata. Cert use-case examples include service/app identity authentication (i.e. TLS, mTLS), data encryption/decryption, and signing. Including certificates where the app only has the public key.
• The observation hook/event stream must allow listeners to obtain certificate metadata to filter certificates it does not care about. e.g. customer data certificates.
• Ability to infer usage from metadata.

cc: @InterpolationStation @GrabYourPitchforks @ericstj @geeknoid @tarekgh

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

It is unfortunately not uncommon for applications (particularly web applications) to crash whenever relying on certificates that expire. This can happen both for certificates that are specific to the application itself, or certificates of other services that our application is trying to call into. If the .NET platform started emitting some sort of telemetry each time a certificate is used, and amongst other things, included the expiration date of that certificate, then components can be built on top of that telemetry which will be able to configure filters and then set up monitoring alerts so the application developers can get notified ahead of time and prevent an issue that might take the application down.

It is important to try to define what "using" a certificate means and which scenarios are the ones that developers would care about, since certificates get used a lot by applications these days on scenarios that some might not care about, for instance, certificates will get loaded each time the application makes an https request. This means that defining the scenarios that a developer might care about could help reducing potential "noise".

Here are some initial goals that we would like to enable for application developers with this:

• Provide the ability to observe certificates used by dotnet runtime in the context of an application, including usage metadata. Cert use-case examples include service/app identity authentication (i.e. TLS, mTLS), data encryption/decryption, and signing. Including certificates where the app only has the public key.
• The observation hook/event stream must allow listeners to obtain certificate metadata to filter certificates it does not care about. e.g. customer data certificates.
• Ability to infer usage from metadata.

cc: @InterpolationStation @GrabYourPitchforks @ericstj @geeknoid @tarekgh

Author:	joperezr
Assignees:	-
Labels:	area-System.Security, untriaged, needs-area-label
Milestone:	-

[tarekgh]

@joperezr I think we decided to use EventSource there as the code in the corelib. It is worth calling that out.

[bartonjs]

I don't think any of this telemetry belongs with certificates; it's all going to be particular use cases.

It probably makes sense for SslStream to log what certificate is being used to start a server authentication context, and perhaps the expiration thereof, and similarly when using a certificate for client authentication. But that would be very different than loading a certificate out of the AuthentiCode signature of an executable, or any other situation.

Since it's all context-specific, I think this issue should be closed, and anywhere that it seems relevant to have such logging (e.g. SslStream) should have individual focused items.

[ericstj]

Let's keep this issue open to track the cross-cutting ask and discussion. I believe that @geeknoid and @InterpolationStation also had scenarios where the application loads certificates and uses them itself. They would like some telemetry about those as well. I'm not sure if all those would be covered by passing to framework API or not.

[geeknoid]

We need a single chokepoint to unilaterally capture all cert accesses going on in the system such that we can report robust telemetry with no gaps. The idea is that we would have filters so that only telemetry relevant for a specific scenario gets emitted.

[vcsjones]

We need a single chokepoint to unilaterally capture all cert accesses going on in the system

Some random thoughts.

First, I don't think there is a "single" chokepoint in .NET because the OS may access certificates on behalf of .NET. Say for example, OCSP responses. .NET doesn't ever actually see the OCSP responder's certificate on Windows, to my knowledge. Expired OCSP responders happen, and has caused at least one outage for a Microsoft operated site, as far as I am aware.

Now, I might be playing "gotcha" here, but I want to make sure it is understood that no amount of telemetry in .NET can capture literally every single certificate access. The OS is doing some lifting in some places.

Maybe you're not interested in expired OCSP responders (or other things the OS does on .NETs behalf). I would say that is reasonable, but if you aren't interested in that scenario, what scenarios are you interested in? And that is where the telemetry should go - not in S.S.C.X509Certificates.

The idea is that we would have filters so that only telemetry relevant for a specific scenario gets emitted.

What do you need in the trace event to understand if it is relevant or not? Let's say telemetry is added to X509Certificate2 and friends - where does the data that you are filtering on come from? From a certificate's perspective, it has no idea why it is being used - it's just a public key with extra stuff. Maybe that stuff is part of what you want to filter on. Or maybe it isn't. For example, server certificates and client certificates can be indistinguishable from each other because most CAs issue certificates with both EKUs. So the certificate has no idea if it is being used in a client scenario or a server scenario.

[geeknoid]

The reality of the situation is that services are constantly experiencing outages due to cert expiration. Do you have any recommendations on how the libraries can provide support to avoid this problem from occurring? We found that producing telemetry is a relatively simple and effective way to enable systematic detection of potentially outdated secrets. The problem with our current solution is that it requires a dedicated API call that customers must call explicitly, which is easy to forget to do.

I'm open to other ideas.

[ericstj]

Perhaps it would be helpful to enumerate all the certificate use cases and see if a .Net solution covers the majority on all platforms? What do you think @InterpolationStation @geeknoid?