Skip to content

[KTX] Fixing port to properly patch zstd for platforms other than windows#24869

Merged
dan-shaw merged 10 commits into
microsoft:masterfrom
Honeybunch:ktx-zstd
Jun 20, 2022
Merged

[KTX] Fixing port to properly patch zstd for platforms other than windows#24869
dan-shaw merged 10 commits into
microsoft:masterfrom
Honeybunch:ktx-zstd

Conversation

@Honeybunch
Copy link
Copy Markdown
Contributor

Describe the pull request
I wanted to get the ktx[vulkan]:arm64-android build working and this is what I had to do to do it.

  • What does your PR fix?

I wasn't able to get ktx[vulkan]:arm64-android to build due to undefined symbols from zstd libraries
The patch for making ktx work with the vcpkg installed zstd was only going to work on windows, so I updated the patch to properly link zstd on all platforms.

Now ktx[vulkan]:arm64-android builds and I've compiled an app against it that works.

Considering that KTX-Software has CI up for an android build I think it's safe to say that the arm64-android port here should also work.

I'm also bumping the revision hash which brings in a fix from upstream that allows ktx to build and work for x64-mingw-dynamic and x64-mingw-static (at least in my tests).

  • Which triplets are supported/not supported? Have you updated the CI baseline?

I've tested this patch on
arm64-android
x64-mingw-static
x64-mingw-dynamic
x64-windows
x64-window-static
No reason to think it would break anything else
I think this also opens up testing for this to see if it works on arm64-osx

  • Does your PR follow the maintainer guide?

    Yes I hope so

  • If you have added/updated a port: Have you run ./vcpkg x-add-version --all and committed the result?

    Yep!

@JackBoosY JackBoosY added the category:port-bug The issue is with a library, which is something the port should already support label May 23, 2022
@JackBoosY
Copy link
Copy Markdown
Contributor

[43/99] C:\PROGRA~1\MICROS~1\2022\ENTERP~1\VC\Tools\MSVC\1432~1.313\bin\Hostx64\arm64\cl.exe   /TP -DASTCENC_AVX=0 -DASTCENC_F16C=0 -DASTCENC_NEON=1 -DASTCENC_POPCNT=0 -DASTCENC_SSE=0 -D_CRT_SECURE_NO_WARNINGS -ID:\buildtrees\ktx\src\8a8972a566-4be5f08999.clean\lib\astc-encoder\Source /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /MD /O2 /Oi /Gy /DNDEBUG /Z7  -MD /W4 /O2 /wd4324 /EHsc /fp:strict -std:c++14 /showIncludes /Folib\astc-encoder\Source\CMakeFiles\astcenc-neon-static.dir\astcenc_ideal_endpoints_and_weights.cpp.obj /Fdlib\astc-encoder\Source\CMakeFiles\astcenc-neon-static.dir\astcenc-neon-static.pdb /FS -c D:\buildtrees\ktx\src\8a8972a566-4be5f08999.clean\lib\astc-encoder\Source\astcenc_ideal_endpoints_and_weights.cpp
FAILED: lib/astc-encoder/Source/CMakeFiles/astcenc-neon-static.dir/astcenc_ideal_endpoints_and_weights.cpp.obj 
C:\PROGRA~1\MICROS~1\2022\ENTERP~1\VC\Tools\MSVC\1432~1.313\bin\Hostx64\arm64\cl.exe   /TP -DASTCENC_AVX=0 -DASTCENC_F16C=0 -DASTCENC_NEON=1 -DASTCENC_POPCNT=0 -DASTCENC_SSE=0 -D_CRT_SECURE_NO_WARNINGS -ID:\buildtrees\ktx\src\8a8972a566-4be5f08999.clean\lib\astc-encoder\Source /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /MD /O2 /Oi /Gy /DNDEBUG /Z7  -MD /W4 /O2 /wd4324 /EHsc /fp:strict -std:c++14 /showIncludes /Folib\astc-encoder\Source\CMakeFiles\astcenc-neon-static.dir\astcenc_ideal_endpoints_and_weights.cpp.obj /Fdlib\astc-encoder\Source\CMakeFiles\astcenc-neon-static.dir\astcenc-neon-static.pdb /FS -c D:\buildtrees\ktx\src\8a8972a566-4be5f08999.clean\lib\astc-encoder\Source\astcenc_ideal_endpoints_and_weights.cpp
cl : Command line warning D9025 : overriding '/W3' with '/W4'
D:\buildtrees\ktx\src\8a8972a566-4be5f08999.clean\lib\astc-encoder\Source\astcenc_vecmathlib_neon_4.h(877): warning C4244: 'return': conversion from 'int' to 'uint16_t', possible loss of data
D:\buildtrees\ktx\src\8a8972a566-4be5f08999.clean\lib\astc-encoder\Source\astcenc_vecmathlib_neon_4.h(219) : error C2057: expected constant expression
D:\buildtrees\ktx\src\8a8972a566-4be5f08999.clean\lib\astc-encoder\Source\astcenc_vecmathlib_neon_4.h(219) : operand 1: Illegal reg field

D:\buildtrees\ktx\src\8a8972a566-4be5f08999.clean\lib\astc-encoder\Source\astcenc_vecmathlib_neon_4.h(219) : fatal error C1001: Internal compiler error.
(compiler file 'D:\a\_work\1\s\src\vctools\Compiler\Utc\src\p2\arm64\disencode.cpp', line 2686)
 To work around this problem, try simplifying or changing the program near the locations listed above.
If possible please provide a repro here: https://developercommunity.visualstudio.com 
Please choose the Technical Support command on the Visual C++ 
 Help menu, or open the Technical Support help file for more information

Please only check this one:

D:\buildtrees\ktx\src\8a8972a566-4be5f08999.clean\lib\astc-encoder\Source\astcenc_vecmathlib_neon_4.h(219) : error C2057: expected constant expression

@JackBoosY JackBoosY added requires:author-response category:community-triplet A PR or issue related to community triplets not officially validated by the vcpkg team. labels May 23, 2022
Comment thread ports/ktx/0001-Use-vcpkg-zstd.patch Outdated
@Honeybunch
Copy link
Copy Markdown
Contributor Author

@JackBoosY There seems to be a bug in the MSVC arm64 implementation: https://developercommunity.visualstudio.com/t/inlining-turns-constant-into-register-operand-for/1394798

Also found in the astc encoder on windows arm64. Fix still doesn't seem to be deployed yet so maybe we should just force this port to reject only arm64 windows?

@JackBoosY
Copy link
Copy Markdown
Contributor

According to the feedback reply:

We will update this Developer Community item when there is a fix for this issue. In the meantime, you can work around this issue by passing the /d2ssa-cfg-sink- flag to the compiler.

I think you may need to add a patch to add flag /d2ssa-cfg-sink- to resolve this. Also please add comment to the place which apply this patch.

@Honeybunch
Copy link
Copy Markdown
Contributor Author

Oops I totally missed that part of the post there. I'm just going to try to upstream a patch to KTX rather than do that work as part of another .patch file

Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!

PRs must add only one version and must not modify any published versions

When making any changes to a library, the version or port-version in vcpkg.json or CONTROL must be modified.

error: checked-in files for ktx have changed but the version was not updated
version: 4.1.0-rc2
old SHA: 4439d89eb44c15008e4c66902b2ce5167799bb2a
new SHA: e8a49dcd008235323faeb7a266a0b04a8487a11c
Did you remember to update the version or port version?
Use --overwrite-version to bypass this check
***No files were updated***

@Honeybunch
Copy link
Copy Markdown
Contributor Author

Finally got KTX upstream to accept some changes so it should now build for arm64 windows and arm64 android. I also made a small tweak to the mingw toolchain since ktx built with mingw should also work but the mingw toolchain didn't like the gcc install I got from scoop

Comment thread scripts/toolchains/mingw.cmake Outdated
@Honeybunch
Copy link
Copy Markdown
Contributor Author

Not sure why x86_windows failed here. The port is marked as not being supported on x86 and the previous commit succeeded

@Honeybunch
Copy link
Copy Markdown
Contributor Author

Seems like something internal broke in CI?

2022-06-16T23:56:16.9123289Z error: curl has reported unexpected results to vcpkg and vcpkg cannot continue.
2022-06-16T23:56:16.9129981Z Please review the following text for sensitive information and open an issue on the Microsoft/vcpkg GitHub to help fix this problem!

https://dev.azure.com/vcpkg/c1ee48cb-0df2-4ab3-8384-b1df5a79fe53/_apis/build/builds/73801/logs/63

@dg0yt
Copy link
Copy Markdown
Contributor

dg0yt commented Jun 17, 2022

To re-trigger x86-windows:

git commit --allow-empty -m CI
git push

@JackBoosY JackBoosY added info:needs-maintainer-attention Lets the current 'on rotation' vcpkg maintainer know they need to look at this. and removed depends:upstream-changes Waiting on a change to the upstream project labels Jun 17, 2022
@dan-shaw dan-shaw merged commit d83d14d into microsoft:master Jun 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category:community-triplet A PR or issue related to community triplets not officially validated by the vcpkg team. category:port-bug The issue is with a library, which is something the port should already support info:needs-maintainer-attention Lets the current 'on rotation' vcpkg maintainer know they need to look at this.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants