GH-49676: [Python][Packaging] Avoid rebuilding all base images by not pulling base images on docker build#49678
GH-49676: [Python][Packaging] Avoid rebuilding all base images by not pulling base images on docker build#49678raulcd wants to merge 14 commits intoapache:mainfrom
Conversation
…by not pulling bases
|
@github-actions crossbow submit wheel-windows-* |
|
|
|
Revision: 663e4a6 Submitted crossbow builds: ursacomputing/crossbow @ actions-2d3c582858
|
… reduce layer size
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: a0edd40 Submitted crossbow builds: ursacomputing/crossbow @ actions-a1756da700
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: b45c287 Submitted crossbow builds: ursacomputing/crossbow @ actions-946cdff5a8
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 4b9fd9c Submitted crossbow builds: ursacomputing/crossbow @ actions-e9e5587b8a
|
… of layer size" This reverts commit 4b9fd9c.
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 5479a8d Submitted crossbow builds: ursacomputing/crossbow @ actions-310307e9a4
|
|
@kou any idea what might be the issue here? |
|
I'll take a look at this in a few hours! |
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: a7c0613 Submitted crossbow builds: ursacomputing/crossbow @ actions-66876a4e34
|
|
Sorry. I couldn't take a look at this in a few hours... I'll take a look at this in 24 hours... |
|
As per 🤖 we should implement the change below: "this adds a step that makes the job explicitly check for the docker service, start it if needed, and wait until docker version succeeds before any archery docker pull/build/run commands execute. So this turns an intermittent windows-2022 runner issue into a handled setup step, which should reduce spurious CI failures." --- a/dev/tasks/python-wheels/github.windows.yml
+++ b/dev/tasks/python-wheels/github.windows.yml
@@ -43,6 +43,32 @@
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_login_ghcr()|indent }}
{{ macros.github_install_archery()|indent }}
+
+ - name: Wait for Docker
+ shell: pwsh
+ run: |
+ $dockerService = Get-Service -Name docker -ErrorAction SilentlyContinue
+ if ($null -eq $dockerService) {
+ Write-Error "Docker service not found."
+ exit 1
+ }
+
+ if ($dockerService.Status -ne "Running") {
+ Write-Host "Starting Docker service..."
+ Start-Service -Name docker
+ }
+
+ for ($i = 1; $i -le 60; $i++) {
+ docker version *> $null
+ if ($LASTEXITCODE -eq 0) {
+ Write-Host "Docker is ready."
+ exit 0
+ }
+ Start-Sleep -Seconds 2
+ }
+
+ Write-Error "Docker failed to become ready."
+ exit 1
- name: Prepare
shell: bash |
|
I think that could solve the "intermittent failure" around docker being unable but the big issue is not that one. The real problem is when docker is running at building the flight dependencies from vcpkg step. If I isolate the different vcpkg features being built the image builds fine until the flight feature step. Flight pulls grpc/abseil/protobuf. The |
|
Docker seemed to have been upgraded on the runner images from 27.x/28.x to 29.1 which might be the issue: |
…lling gRPC with vcpkg
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 14c89a4 Submitted crossbow builds: ursacomputing/crossbow @ actions-d3af6272a6
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 7f24ad8 Submitted crossbow builds: ursacomputing/crossbow @ actions-574c6b6bb8
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 6bd3713 Submitted crossbow builds: ursacomputing/crossbow @ actions-15fa7be7ef
|
|
Hmm. It seems that we need to patch vcpkg to remove unconditional diff --git a/ci/vcpkg/vcpkg.patch b/ci/vcpkg/vcpkg.patch
index a4c8d52097..2627f8c562 100644
--- a/ci/vcpkg/vcpkg.patch
+++ b/ci/vcpkg/vcpkg.patch
@@ -1,3 +1,18 @@
+diff --git a/scripts/toolchains/windows.cmake b/scripts/toolchains/windows.cmake
+index 3cc90cc886..36af495687 100644
+--- a/scripts/toolchains/windows.cmake
++++ b/scripts/toolchains/windows.cmake
+@@ -86,8 +86,8 @@ if(NOT _VCPKG_WINDOWS_TOOLCHAIN)
+
+ set(CMAKE_CXX_FLAGS_DEBUG "${VCPKG_CRT_LINK_FLAG_PREFIX}d /Z7 /Ob0 /Od /RTC1 ${VCPKG_CXX_FLAGS_DEBUG}" CACHE STRING "")
+ set(CMAKE_C_FLAGS_DEBUG "${VCPKG_CRT_LINK_FLAG_PREFIX}d /Z7 /Ob0 /Od /RTC1 ${VCPKG_C_FLAGS_DEBUG}" CACHE STRING "")
+- set(CMAKE_CXX_FLAGS_RELEASE "${VCPKG_CRT_LINK_FLAG_PREFIX} /O2 /Oi /Gy /DNDEBUG /Z7 ${VCPKG_CXX_FLAGS_RELEASE}" CACHE STRING "")
+- set(CMAKE_C_FLAGS_RELEASE "${VCPKG_CRT_LINK_FLAG_PREFIX} /O2 /Oi /Gy /DNDEBUG /Z7 ${VCPKG_C_FLAGS_RELEASE}" CACHE STRING "")
++ set(CMAKE_CXX_FLAGS_RELEASE "${VCPKG_CRT_LINK_FLAG_PREFIX} /O2 /Oi /Gy /DNDEBUG ${VCPKG_CXX_FLAGS_RELEASE}" CACHE STRING "")
++ set(CMAKE_C_FLAGS_RELEASE "${VCPKG_CRT_LINK_FLAG_PREFIX} /O2 /Oi /Gy /DNDEBUG ${VCPKG_C_FLAGS_RELEASE}" CACHE STRING "")
+
+ string(APPEND CMAKE_STATIC_LINKER_FLAGS_RELEASE_INIT " /nologo ")
+ set(CMAKE_MODULE_LINKER_FLAGS_RELEASE "/nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF ${VCPKG_LINKER_FLAGS} ${VCPKG_LINKER_FLAGS_RELEASE}" CACHE STRING "")
diff --git a/scripts/cmake/vcpkg_execute_build_process.cmake b/scripts/cmake/vcpkg_execute_build_process.cmake
index 60fd5b587a..c8dc021af8 100644
--- a/scripts/cmake/vcpkg_execute_build_process.cmake |
|
@kou that's a really good one. Let me push a commit to try it |
…ng /Z7 flag from build
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 79da88c Submitted crossbow builds: ursacomputing/crossbow @ actions-8c0fd92f01
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 96931be Submitted crossbow builds: ursacomputing/crossbow @ actions-5806a993b9
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: a033258 Submitted crossbow builds: ursacomputing/crossbow @ actions-8d2d071202
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 14f1b27 Submitted crossbow builds: ursacomputing/crossbow @ actions-733eebe387
|
3 participants
TBD
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?
This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)