Migrate CI pipeline from CircleCI to GitHub Actions

[Brandon-Kimberly]

I implemented the transition of the CI pipeline in the OpenTelemetry C++ repository from CircleCI to GitHub Actions. More specifically, I have implemented a GitHub Actions workflow with separate jobs to run all tests that were previously running in CircleCI. I’m submitting this PR to add the GitHub Actions workflows for the repository. I will be submitting another PR for the integration with codecov.io (http://codecov.io/) for coverage reports to be displayed in the repository.

This is a first cut at migrating the CI pipeline from CircleCI to GitHub Actions addressing issue #102 (#102). All CircleCI jobs and tests have been ported over to Actions. Note: I have left the CircleCI config file in the repository for now.

Update

All jobs are running correctly now and this PR, unless anybody has more feedback, should be good to go!

[linux-foundation-easycla]

The committers are authorized under a signed CLA.

• ✅ Brandon Kimberly (920e8ca, df4ed5c, 07dc7de, 5e1d9b3, 7d93ae7, ba894cb, 8832e48, 0d9aa44, 72c923b)
• ✅ Johannes Tax (c0ddb8f, aadd8e9, 8afef6a)
• ✅ Hudson Humphries (72feb9f)

[pyohannes]

Could you please rebase? It seems this contains lots of unrelated changes.

[Brandon-Kimberly]

Could you please rebase? It seems this contains lots of unrelated changes.

I'm sorry. Could you please elaborate on how I would do that? I tried to rebase before submitting but I guess I messed it up somehow.

[pyohannes]

Could you please elaborate on how I would do that?

You could do a hard reset of your branch to origin/master, then cherry-pick all the commits that you have made and do a force-push.

If you don't mind losing commit history, you can reset your branch to origin/master, copy in your modifications, commit and force-push.

[anuraaga]

Nice README updates :) Left some small comments on the ci config

[meastp]

@Brandon-Kimberly I'm looking at the windows failures... When I scroll through the output in CI / windows I can only see output for ./ci/setup_windows_cmake.ps1 and not ./ci/setup_windows_ci_environment.ps1 or ./ci/install_windows_protobuf.ps1 which should have some output if it did run.

Perhaps you could format the workflow file run commands on multiple lines to make it easier to read?

Also, perhaps there is a difference between the windows shell and ubuntu shell wrt to space-separated scripts on a single line?

Try this:

  windows:

    runs-on: windows-2019

    steps:
    - uses: actions/checkout@v2
    - name: setup
      run: |
        ./ci/setup_windows_cmake.ps1
        ./ci/setup_windows_ci_environment.ps1
        ./ci/install_windows_protobuf.ps1
      timeout-minutes: 15
    - name: run cmake test
      run: ./ci/do_ci.ps1 cmake.test
    - name: run otprotocol test
      run: ./ci/do_ci.sh cmake.exporter.otprotocol.test

[Brandon-Kimberly]

@Brandon-Kimberly What happened? I think this is a great feature - why did you close it and delete the commits (by force push)?

I accidentally messed up the rebasing. Fixing now.

[Brandon-Kimberly]

@Brandon-Kimberly I'm looking at the windows failures... When I scroll through the output in CI / windows I can only see output for ./ci/setup_windows_cmake.ps1 and not ./ci/setup_windows_ci_environment.ps1 or ./ci/install_windows_protobuf.ps1 which should have some output if it did run.

Perhaps you could format the workflow file run commands on multiple lines to make it easier to read?

Also, perhaps there is a difference between the windows shell and ubuntu shell wrt to space-separated scripts on a single line?

Try this:

  windows:

    runs-on: windows-2019

    steps:
    - uses: actions/checkout@v2
    - name: setup
      run: |
        ./ci/setup_windows_cmake.ps1
        ./ci/setup_windows_ci_environment.ps1
        ./ci/install_windows_protobuf.ps1
      timeout-minutes: 15
    - name: run cmake test
      run: ./ci/do_ci.ps1 cmake.test
    - name: run otprotocol test
      run: ./ci/do_ci.sh cmake.exporter.otprotocol.test

Wow you're right! Windows does only allow one executable per line. The others are ignored.

Putting the different run commands on separate lines fixed one of the Windows jobs. Unfortunately one still fails stating that "The term './ci/install_windows_protobuf.ps1' is not recognized as the name of a cmdlet, function, script file, or operable program."

[meastp]

@Brandon-Kimberly I'm looking at the windows failures... When I scroll through the output in CI / windows I can only see output for ./ci/setup_windows_cmake.ps1 and not ./ci/setup_windows_ci_environment.ps1 or ./ci/install_windows_protobuf.ps1 which should have some output if it did run.
Perhaps you could format the workflow file run commands on multiple lines to make it easier to read?
Also, perhaps there is a difference between the windows shell and ubuntu shell wrt to space-separated scripts on a single line?
Try this:

  windows:

    runs-on: windows-2019

    steps:
    - uses: actions/checkout@v2
    - name: setup
      run: |
        ./ci/setup_windows_cmake.ps1
        ./ci/setup_windows_ci_environment.ps1
        ./ci/install_windows_protobuf.ps1
      timeout-minutes: 15
    - name: run cmake test
      run: ./ci/do_ci.ps1 cmake.test
    - name: run otprotocol test
      run: ./ci/do_ci.sh cmake.exporter.otprotocol.test

Wow you're right! Windows does only allow one executable per line. The others are ignored.

Putting the different run commands on separate lines fixed one of the Windows jobs. Unfortunately one still fails stating that "The term './ci/install_windows_protobuf.ps1' is not recognized as the name of a cmdlet, function, script file, or operable program."

Great! Perhaps you need to explicitly state that you want powershell as the shell? EDIT: Nevermind (https://github.blog/changelog/2019-10-17-github-actions-default-shell-on-windows-runners-is-changing-to-powershell/)

EDIT2: It seems that shell: powershell (which is the default and therefore does not have to be explicitly used) and shell: pwsh (powershell core) are two different shells. You could also try the latter.

You can also try to use a Windows-y slash in the paths: e.g. ./ci/install_windows_protobuf.ps1 -> .\ci\install_windows_protobuf.ps1, although powershell is supposed to accept both / and \

@Brandon-Kimberly You could also split the three commands (or the last command, which is the one failing) into a separate run: task... Then work on unifying them into a single run-command after the PR is merged? :)

[Brandon-Kimberly]

Update: After trying the suggestions above, one of the Windows jobs now passes correctly!

However, the Windows OTProtocol test still fails on account of not being able to locate the file "\ci\install_windows_protobuf.ps1". I have tried to fix this by using '' instead of '/' and by specifying that the job uses powershell core. But the test is still failing. If anybody has any suggestions I would be very thankful!

Additionally, the format job always fails as well. The job runs the format tool correctly and says that some files do not abide by the formatting specifications. Running the format tool on my local machine changes a few files including ci/setup_windows_cmake.ps1, tools/git-cl.cmd, and tools/setup-devenv.cm. However, when trying to push those files to the remote repository I am met with this warning:

"warning: LF will be replaced by CRLF in tools/setup-devenv.cmd.
The file will have its original line endings in your working directory"

Again, I welcome any help on this issue.

[meastp]

@Brandon-Kimberly I finally found the issue on Windows: it is simply the ci/setup_windows_ci_environment.ps1 script that changes directory to vcpkg.

To fix this, change ci/setup_windows_ci_environment.ps1 from:

(remove cd vcpkg)

trap { $host.SetShouldExit(1) }

git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
$VCPKG_DIR=(Get-Item -Path ".\").FullName
./bootstrap-vcpkg.bat
./vcpkg integrate install
./vcpkg install benchmark:x64-windows
./vcpkg install gtest:x64-windows

to:

(add Push-Location and Pop-Location)

trap { $host.SetShouldExit(1) }

git clone https://github.com/Microsoft/vcpkg.git
Push-Location -Path vcpkg
$VCPKG_DIR=(Get-Item -Path ".\").FullName
./bootstrap-vcpkg.bat
./vcpkg integrate install
./vcpkg install benchmark:x64-windows
./vcpkg install gtest:x64-windows
Pop-Location

I think it is more consistent to use / in paths (in the github actions workflow file) on Windows as well as this was not the issue. Do what you please :)

[meastp]

@Brandon-Kimberly ok, I think I figured out the format issue as well

• As far as I can tell, the tools/format.sh script removes carriage return (CR) from all files. This is problematic for .cmd and .ps1 files in the ci directory, which according to the .gitattributes file should have crlf line endings.

(this is also why git commands give warnings as the one you saw)

• to fix this, I propose changing ./ci/do_ci.sh to running git add --renormalize . after tools/format.sh and before git ls-files --modified to normalise line endings according to .gitattributes.

from:

elif [[ "$1" == "format" ]]; then
  tools/format.sh
  CHANGED="$(git ls-files --modified)"

to:

elif [[ "$1" == "format" ]]; then
  tools/format.sh
  # normalize file endings according to .gitattributes
  git add --renormalize .
  CHANGED="$(git ls-files --modified)"

[Brandon-Kimberly]

@Brandon-Kimberly ok, I think I figured out the format issue as well

* As far as I can tell, the `tools/format.sh` script removes carriage return (CR) from all files. This is problematic for .cmd and .ps1 files in the ci directory, which according to the .gitattributes file should have crlf line endings.

(this is also why git commands give warnings as the one you saw)

* to fix this, I propose changing `./ci/do_ci.sh` to running `git add --renormalize .` after `tools/format.sh` and before `git ls-files --modified` to normalise line endings according to .gitattributes.

from:

elif [[ "$1" == "format" ]]; then
  tools/format.sh
  CHANGED="$(git ls-files --modified)"

to:

elif [[ "$1" == "format" ]]; then
  tools/format.sh
  # normalize file endings according to .gitattributes
  git add --renormalize .
  CHANGED="$(git ls-files --modified)"

Thanks this worked! I figured it had something to do with line endings but was unsure how to modify the format job to account for this.

[Brandon-Kimberly]

@Brandon-Kimberly I finally found the issue on Windows: it is simply the ci/setup_windows_ci_environment.ps1 script that changes directory to vcpkg.

To fix this, change ci/setup_windows_ci_environment.ps1 from:

(remove cd vcpkg)

trap { $host.SetShouldExit(1) }

git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
$VCPKG_DIR=(Get-Item -Path ".\").FullName
./bootstrap-vcpkg.bat
./vcpkg integrate install
./vcpkg install benchmark:x64-windows
./vcpkg install gtest:x64-windows

to:

(add Push-Location and Pop-Location)

trap { $host.SetShouldExit(1) }

git clone https://github.com/Microsoft/vcpkg.git
Push-Location -Path vcpkg
$VCPKG_DIR=(Get-Item -Path ".\").FullName
./bootstrap-vcpkg.bat
./vcpkg integrate install
./vcpkg install benchmark:x64-windows
./vcpkg install gtest:x64-windows
Pop-Location

I think it is more consistent to use / in paths (in the github actions workflow file) on Windows as well as this was not the issue. Do what you please :)

Thank you, this worked as well! I guess the problem was that the job ended up running in a different directory then I thought.

Thank you for all your help! 😄

[pyohannes]

Thanks, that looks really great!

[reyang]

Looks great to me! 🥇

[pyohannes]

Looks great, I think it's ready to go. I have one small nit that's not critical (yet).

[Brandon-Kimberly]

Branch is rebased and should be good to go!

[reyang]

LGTM.

[g-easy]

Oops, git add --renormalize broke format CI. We haven't been enforcing formatting since.

[g-easy]

Fix in #222