Document how to properly setup Codecov Github Action for OpenSource repositories

[ST-DDT]

Describe the bug

PRs from external contributors fail CI.

The PR is happening in a forked repo. Using tokenless upload.
Tokenless has reached GitHub rate limit.

Environment (please complete the following information):

• CodeCov GitHub-Action v4.1.0
• https://github.com/faker-js/faker/blob/a672d277b883a445a852f9b6db18818104e8de7e/.github/workflows/ci.yml#L212-L216

To Reproduce

Steps to reproduce the behavior:

1. Go to an OpenSource project with CodeCov GitHub Action
2. Open a PR as external contributor
3. The CI will very likely fail (~10 CI failures, sometimes multiple in a row)

External contributors have a near 100% chance at failing. Internal contributors have a near 0% chance at failing.

Expected behavior

The CI is expected to work.
Or alternatively the guide describing how to setup the action should provide hints on how to deal with the error.

Screenshots

info - 2024-03-12 15:43:46,354 -- ci service found: github-actions
info - 2024-03-12 15:43:46,690 -- The PR is happening in a forked repo. Using tokenless upload.
info - 2024-03-12 15:43:47,076 -- Process Commit creating complete
error - 2024-03-12 15:43:47,076 -- Commit creating failed: {"detail":"Tokenless has reached GitHub rate limit. Please upload using a token: https://docs.codecov.com/docs/adding-the-codecov-token. Expected available in 1455 seconds."}
Error: Codecov: Failed to properly create commit: The process '/home/runner/work/_actions/codecov/codecov-action/54bcd8715eee62d40e33596ef5e8f0f48dbbccab/dist/codecov' failed with exit code 1

Additional context

Created on request here:

• "Unable to locate build via Github Actions API" for the public repository #126 (comment)

Affected CI pipelines:

• https://github.com/faker-js/faker/actions/runs/8251702824/job/22569395448?pr=2740
• https://github.com/faker-js/faker/actions/runs/8251662161/job/22569253564?pr=2724

[kdarkhan]

It might be hard to find the workaround I came up with from the linked issue, adding it here if that becomes useful.

I ended up using 2 workflow approach which seems to work pretty stable with token based auth for forked repos.

This workflow uploads coverage report to Github artifacts and this workflow uses that report and the token to upload it to codecov.

[adamjstewart]

Any updates on this? We're seeing a 100% failure rate on most recent PRs. Is the open source community expected to transition to a different coverage provider, or are there future plans to once again support forks?

[rohan-at-sentry]

@ST-DDT thanks for raising.

For context (which you probably already realized from the screenshot - but may benefit others), v4 allows tokenless uploads for forks ; enabling open-source contributors to upload coverage.

Because this happens without a token, Codecov uploads from forks upload using Codecov's own token - you can see how quickly that can get consumed leading to a failure. That is most likely what is happening when you see

error - 2024-03-12 15:43:47,076 -- Commit creating failed: {"detail":"Tokenless has reached GitHub rate limit. Please upload using a token: https://docs.codecov.com/docs/adding-the-codecov-token. Expected available in 1455 seconds."}

In effect - uploading from a fork should be (from a reliability standpoint) no different from before v4 was released. Is it possible for you to confirm if the rate is worse since Feb of this year?

Also - I'm curious if the workaround here worked for you

[ST-DDT]

In effect - uploading from a fork should be (from a reliability standpoint) no different from before v4 was released. Is it possible for you to confirm if the rate is worse since Feb of this year?

I think the rate remained the same.
It just happened a lot at the time of reporting (and for an extended period of time).
Previously it was an undefimed error, now it is a specific error.

Also - I'm curious if the workaround here worked for you

No, we are currently in a feature freeze and thus have very few external contributions.
We generally have a low need for code coverage and thus consider removing it instead due to the additional work required.

[adamjstewart]

I tried to use the above workaround. Didn't try the full thing with hundreds of lines of JavaScript code, but it was too complicated to get working with the usual upload/download artifact actions. TorchGeo gets 99% external contributions, so we're very keen on getting this working.