`GITHUB_ENV` file intermittently corrupted on OSX

[anthony-c-martin]

Issue

Our GitHub CI is intermittently failing on macOS build machines when running dotnet build, with this cryptic error:

Error: Unable to process file command 'env' successfully.
Error: Value cannot be null. (Parameter 'name')

Example run here: https://github.com/Azure/bicep/runs/4073161120?check_suite_focus=true

Here's what this step looks like in the workflow definition: https://github.com/Azure/bicep/blob/d41b31884bdb1e8311e357711ebad9933d293aa2/.github/workflows/build.yml#L60-L61

Troubleshooting

To troubleshoot, I pushed a minimal build definition (OSX only, only run dotnet build) to my own fork of the repro and kicked off a number of builds, and added some logging for the contents of $GITHUB_ENV.

The step I'm using to troubleshoot is here: https://github.com/anthony-c-martin/bicep/blob/031f226ffea5c8e88af82f6bfa99bc6fa19d85f4/.github/workflows/build.yml#L45-L50

Good run

What I found was on a good run, the $GITHUB_ENV file contained a bunch of well-formatted nerdbank env variables - e.g.:

...
GitAssemblyInformationalVersion=0.4.1080+9f9a6b01ee
GitAssemblyInformationalVersion=0.4.1080+9f9a6b01ee
GitAssemblyInformationalVersion=0.4.1080+9f9a6b01ee
GitAssemblyInformationalVersion=0.4.1080+9f9a6b01ee
GitAssemblyInformationalVersion=0.4.1080+9f9a6b01ee
GitAssemblyInformationalVersion=0.4.1080+9f9a6b01ee
GitBuildVersion=0.4.1080.39583
GitBuildVersion=0.4.1080.39583
GitBuildVersion=0.4.1080.39583
...

Example run: https://github.com/anthony-c-martin/bicep/runs/4079159672?check_suite_focus=true

Bad run

On a bad run however, the $GITHUB_ENV file appears corrupted (note the missing key when setting an env var):

...
GitAssemblyInformationalVersion=0.4.1079+91a0cb236a
GitBuildVersion=0.4.1079.41105
GitBuildVersion=0.4.1079.41105
=0.4.1079+91a0cb236a
GitBuildVersion=0.4.1079.41105
GitAssemblyInformationalVersion=0.4.1079+91a0cb236a
...

Example run: https://github.com/anthony-c-martin/bicep/runs/4079159378?check_suite_focus=true

I suspect the lack of key when setting some env vars in the $GITHUB_ENV file is corresponding to the cryptic error coming back from GitHub (Error: Value cannot be null. (Parameter 'name'))

[AArnott]

Thanks for the detailed report and your efforts so far. My guess is there is a bug in GitHub Actions or how the runtime is implementing File.AppendAllText such that when run concurrently, file corruption follows. For resiliency against concurrency, we should probably add a named mutex around access to that file. I see we already built a retry around it, but that would only resolve the caught errors, rather than the uncaught corruption.

A couple of workaround ideas for you in the meantime:

1. Add /m:1 to your dotnet build command, which may make your build take longer, but should remove the concurrency and make it reliable.
2. Set cloudBuild: { setVersionVariables: false } to disable setting the env vars. If you need them, add a nbgv cloud step to your workflow.

If you try either or both of these workarounds, please leave feedback as to whether it addressed the issue as that will help confirm/deny my hypothesis as to the cause of this bug.

[anthony-c-martin]

I tested out both workarounds, and saw 7 successful builds in a row after each. Prior to the workarounds, I was seeing a ~50% failure rate, so it appears that both options work.

I don't believe we're using these env vars, so I'll probably go with option 2 above to workaround this for now. Thanks for the speedy feedback!

[georgey-jakks]

Not limited to MacOS. We are experiencing this same failure intermittently during a dotnet test on a GitHub action running Ubuntu 24.04 and Nerdbank.GitVersioning 3.6.133. Will try to remove concurrency, we use the env vars so don't want to squash them.

Command:

dotnet test **.sln --logger trx  --filter "**" -- xunit.parallelizeAssembly=true

[AArnott]

Thanks for sharing. Adding -m:1 after dotnet test will probably avoid the problem. Or if you already built, do yourself a favor and make it faster by adding --no-build so it doesn't build again.