[apple] TvOS/MacCatalyst/iOSSimulator X509Certificates failures during AppleCrypto interop

[matouskozak]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=721722
Build error leg or test failing: Microsoft.Bcl.Cryptography.Tests, System.Security.Cryptography.X509Certificates.Tests
Affected CI: tvos_arm64_release_allsubsets_mono (runtime-extra-platforms)
Range of commits: 42ade70...5a0eb6e

Stack trace example:

[06:04:15.2485240] 2024-06-28 03:04:15.317 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[FAIL] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadWrappingCertificate_PEM_WithSurroundingText
[06:04:15.2485650] Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.
[06:04:15.2520200] 2024-06-28 03:04:15.318 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafeCreateHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.2520870] 2024-06-28 03:04:15.318 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafePasswordHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.2521240]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadX509(ReadOnlySpan`1 data)
[06:04:15.2521910] 2024-06-28 03:04:15.319 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePal(ReadOnlySpan`1 data)
[06:04:15.2522170]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePalFromFile(String path)
[06:04:15.2522250]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificateFromFile(String path)
[06:04:15.2523200] 2024-06-28 03:04:15.319 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate(Byte[] bytes, String path)
[06:04:15.2524270] 2024-06-28 03:04:15.319 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificateNoFile(Byte[] bytes)
[06:04:15.2524440]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests.LoadWrappingCertificate_PEM_WithSurroundingText()
[06:04:15.2524520]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
[06:04:15.2525260] 2024-06-28 03:04:15.319 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
[06:04:15.2632460] 2024-06-28 03:04:15.331 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[FAIL] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate_WithTrailingData
[06:04:15.2632820] Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.
[06:04:15.2632910] 2024-06-28 03:04:15.332 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafeCreateHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.2633130] 2024-06-28 03:04:15.332 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafePasswordHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.2633230]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadX509(ReadOnlySpan`1 data)
[06:04:15.2644490] 2024-06-28 03:04:15.333 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePal(ReadOnlySpan`1 data)
[06:04:15.2644880]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePalFromFile(String path)
[06:04:15.2644960]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificateFromFile(String path)
[06:04:15.2645070] 2024-06-28 03:04:15.333 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate(Byte[] bytes, String path)
[06:04:15.2645150]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificateNoFile(Byte[] bytes)
[06:04:15.2645890] 2024-06-28 03:04:15.334 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests.LoadCertificateAtOffset(Byte[] bytes, Int32 offset)
[06:04:15.2646130]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests.LoadCertificate_WithTrailingData()
[06:04:15.2646200] 2024-06-28 03:04:15.334 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
[06:04:15.2646260]    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
[06:04:15.2770090] 2024-06-28 03:04:15.344 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[PASS] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate_DER_WithTrailingData
[06:04:15.2809240] 2024-06-28 03:04:15.350 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[FAIL] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate_PEM
[06:04:15.2809640] Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.
[06:04:15.2809740] 2024-06-28 03:04:15.350 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafeCreateHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.2836480] 2024-06-28 03:04:15.350 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafePasswordHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.2836890]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadX509(ReadOnlySpan`1 data)
[06:04:15.2837020]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePal(ReadOnlySpan`1 data)
[06:04:15.2837470] 2024-06-28 03:04:15.351 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePalFromFile(String path)
[06:04:15.2837690]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificateFromFile(String path)
[06:04:15.2838510] 2024-06-28 03:04:15.351 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate(Byte[] bytes, String path)
[06:04:15.2838720]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests.LoadCertificate_PEM()
[06:04:15.2839180] 2024-06-28 03:04:15.351 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
[06:04:15.2839240]    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
[06:04:15.2930870] 2024-06-28 03:04:15.361 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[PASS] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadSerializedStore_Fails
[06:04:15.2955610] 2024-06-28 03:04:15.364 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[PASS] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadPkcs7_PEM_Fails
[06:04:15.3006020] 2024-06-28 03:04:15.368 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[PASS] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadPkcs7_BER_Fails
[06:04:15.3118340] 2024-06-28 03:04:15.381 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[FAIL] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadWrappingCertificate_PEM_WithTrailingData
[06:04:15.3118780] Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.
[06:04:15.3118870] 2024-06-28 03:04:15.381 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafeCreateHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.3118950] 2024-06-28 03:04:15.381 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafePasswordHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.3119000]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadX509(ReadOnlySpan`1 data)
[06:04:15.3132990] 2024-06-28 03:04:15.382 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePal(ReadOnlySpan`1 data)
[06:04:15.3133350]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePalFromFile(String path)
[06:04:15.3133620]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificateFromFile(String path)
[06:04:15.3133820] 2024-06-28 03:04:15.382 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate(Byte[] bytes, String path)
[06:04:15.3133940] 2024-06-28 03:04:15.382 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificateNoFile(Byte[] bytes)
[06:04:15.3134000]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests.LoadWrappingCertificate_PEM_WithTrailingData()
[06:04:15.3134050]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
[06:04:15.3139960] 2024-06-28 03:04:15.383 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
[06:04:15.3353850] 2024-06-28 03:04:15.404 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[PASS] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadSerializedCert_Fails
[06:04:15.3870620] 2024-06-28 03:04:15.454 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[PASS] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadPfx_NoPasswordNeeded_Fails
[06:04:15.3895800] 2024-06-28 03:04:15.457 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[PASS] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadEmpty
[06:04:15.4924640] 2024-06-28 03:04:15.560 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[FAIL] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadNestedCertificates
[06:04:15.4925140] Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.
[06:04:15.4925490] 2024-06-28 03:04:15.560 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafeCreateHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.5006610] 2024-06-28 03:04:15.562 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafePasswordHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.5006930]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadX509(ReadOnlySpan`1 data)
[06:04:15.5007110] 2024-06-28 03:04:15.563 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePal(ReadOnlySpan`1 data)
[06:04:15.5007290]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePalFromFile(String path)
[06:04:15.5007340]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificateFromFile(String path)
[06:04:15.5007910] 2024-06-28 03:04:15.563 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate(Byte[] bytes, String path)
[06:04:15.5008000]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificateNoFile(Byte[] bytes)
[06:04:15.5008480] 2024-06-28 03:04:15.564 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests.LoadNestedCertificates()
[06:04:15.5008550]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
[06:04:15.5008660] 2024-06-28 03:04:15.564 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
[06:04:15.5021220] 2024-06-28 03:04:15.571 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[PASS] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate_DER
[06:04:15.5422310] 2024-06-28 03:04:15.610 Microsoft.Bcl.Cryptography.Tests[58164:98425848] 	[FAIL] System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadWrappingCertificate_PEM
[06:04:15.5423010] Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.
[06:04:15.5423160] 2024-06-28 03:04:15.610 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafeCreateHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.5432180] 2024-06-28 03:04:15.612 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at Interop.AppleCrypto.X509ImportCertificate(ReadOnlySpan`1 bytes, X509ContentType contentType, SafePasswordHandle importPassword, SafeSecIdentityHandle& identityHandle)
[06:04:15.5432490]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadX509(ReadOnlySpan`1 data)
[06:04:15.5469330] 2024-06-28 03:04:15.612 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePal(ReadOnlySpan`1 data)
[06:04:15.5469640]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificatePalFromFile(String path)
[06:04:15.5469940]    at System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificateFromFile(String path)
[06:04:15.5470360] 2024-06-28 03:04:15.613 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificate(Byte[] bytes, String path)
[06:04:15.5470460]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromFile.LoadCertificateNoFile(Byte[] bytes)
[06:04:15.5470950] 2024-06-28 03:04:15.614 Microsoft.Bcl.Cryptography.Tests[58164:98425848]    at System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests.LoadWrappingCertificate_PEM()
[06:04:15.5471160]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=721722
Error message validated: [Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2024 3:11:20 PM UTC

Report

Build	Definition	Test	Pull Request
728775	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#102403
727276	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
721722	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.X509CertificateLoaderTests_FromByteSpan.LoadNestedCertificates

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	2	3

[dotnet-policy-service]

Tagging subscribers to 'os-tvos': @vitek-karas, @kotlarmilos, @ivanpovazan, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[matouskozak]

There are two X509Certificates related commits in the range:

• Improve use of PAL's RawData for X509Certificate #103828 @vcsjones
• Create new X509CertificateLoader #102167 @bartonjs

Could any of them be related to this failure?

[vcsjones]

Very likely #102167 is the cause.

[matouskozak]

Very likely #102167 is the cause.

Thank you, @bartonjs could you take a look please?