cfile / ddio: use most of functions to std::filesystem::path

[winterheart]

Pull Request Type

• GitHub Workflow changes
• Documentation or Wiki changes
• Build and Dependency changes
• Runtime changes
  • Render changes
  • Audio changes
  • Input changes
  • Network changes
  • Other changes

Description

This PR updates cfile and partially ddio modules to use C++17's std::filesystem::path feature, which simplifies overall filesystem operations.

Some parts of code was migrated, and there some performance improvements was achieved. Internally cfile maintains some sort search paths. Old code was excessively added some non-existent paths, and on opening files these paths was pointlessly used. Also there some code optimization in cf_FindRealFileNameCaseInsenstive() was performed.

Added unittest framework for cfile.

Related Issues

Screenshots (if applicable)

Checklist

• I have tested my changes locally and verified that they work as intended.
• I have documented any new or modified functionality.
• I have reviewed the changes to ensure they do not introduce any unnecessary complexity or duplicate code.
• I understand that by submitting this pull request, I am agreeing to license my contributions under the project's license.

Additional Comments

[Lgt2x]

You can do something like that to fix mac os CI (but cleaner). strncpy is problematic Lgt2x@15a50ca#diff-be34739991a103c73033e6358b31b2a8d7859b7a5c84616dcf1409176168f55d

[winterheart]

After some refactoring cf_FindRealFileNameCaseInsensitive() now works much faster and more reliable. Here some benchmarks:

Release:

2024-07-02T20:08:56+03:00
Running ./cfile_bench
Run on (12 X 4300 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x6)
  L1 Instruction 32 KiB (x6)
  L2 Unified 256 KiB (x6)
  L3 Unified 12288 KiB (x1)
Load Average: 2.82, 1.81, 1.42
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-------------------------------------------------------------------
Benchmark                         Time             CPU   Iterations
-------------------------------------------------------------------
cfile_no_exist_old            11384 ns        11065 ns        63154
cfile_no_exist_new             9812 ns         9613 ns        74990
cfile_exist_relative_old       2682 ns         2452 ns       258570
cfile_exist_relative_new        926 ns          925 ns       784549
cfile_exist_absolute_old       3422 ns         3132 ns       222331
cfile_exist_absolute_new       1222 ns         1220 ns       585813

Debug:

2024-07-02T20:09:47+03:00
Running ./cfile_bench
Run on (12 X 4300 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x6)
  L1 Instruction 32 KiB (x6)
  L2 Unified 256 KiB (x6)
  L3 Unified 12288 KiB (x1)
Load Average: 2.21, 1.81, 1.45
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-------------------------------------------------------------------
Benchmark                         Time             CPU   Iterations
-------------------------------------------------------------------
cfile_no_exist_old            11768 ns        11426 ns        61555
cfile_no_exist_new            13682 ns        13423 ns        53839
cfile_exist_relative_old       3147 ns         2907 ns       224424
cfile_exist_relative_new       1591 ns         1588 ns       427936
cfile_exist_absolute_old       4179 ns         3893 ns       183101
cfile_exist_absolute_new       2132 ns         2129 ns       319614

[Jayman2000]

How do you know that UTF-8 is the correct encoding to use for fopen()’s filename parameter?

[winterheart]

This way how std::filesystem::path represent native encoding on systems. POSIX filesystem is native char representation, but Windows' native representation is wchar. If you want use paths with C functions, you need convert them with c_str() using u8string().

[Jayman2000]

This way how std::filesystem::path represent native encoding on systems. POSIX filesystem is native char representation, but Windows' native representation is wchar. If you want use paths with C functions, you need convert them with c_str() using u8string().

I’m a little bit confused. According to cppreference.com, std::filesystem::path::u8string() returns a std::u8string. std::u8string is the same as std::basic_string<char8_t>. std::basic_string::<whatever>::c_str() returns a whatever * which means that my_u8string.c_str() should return a char8_t *, right? It seems to me like using_filename.u8string().c_str() is always going to evaluate to a char8_t *. I just don’t see how it could evaluate to a wchar_t *.

(The std::u8string type was added in C++20. In C++17, a similar argument can be made, but you have to replace std::u8string with std::string and char8_t with char.)

Passing a UTF-8 encoded string to fopen() is potentially a problem. According to Microsoft’s documentation,

The fopen function opens the file specified by filename. By default, a narrow filename string is interpreted using the ANSI codepage (CP_ACP). In Windows Desktop applications, it can be changed to the OEM codepage (CP_OEMCP) by using the SetFileApisToOEM function. You can use the AreFileApisANSI function to determine whether filename is interpreted using the ANSI or the system default OEM codepage.

Is there anything that’s going to guarantee that we’re using a UTF-8 codepage? Windows won’t use a UTF-8 codepage by default unless the user has “Beta: Use Unicode UTF-8 for worldwide language support” enabled, or the application explicitly specifies a UTF-8 codepage.

Additionally, if using_filename.u8string().c_str() sometimes does evaluate to a wchar_t *, then I think that we have a problem. According to Microsoft’s documentation, you’re supposed to pass a char * to fopen(), not a wchar_t *. If you want to use a wchar_t * for the filename parameter, then you’re supposed to call _wfopen().

[winterheart]

We don't even interested here what encoding is actually in filesystem. We effectively receive C-string (and don't bother in what encoding is it, it's just sequence of chars) and we ensuring that C-string is actually valid path in native filesystem. We do not using UTF-8 here at all. It's just way to convert native encoding to C-string.

[Jayman2000]

We don't even interested here what encoding is actually in filesystem.

If we’re going to call fopen() on Windows, then I think that we need to be interested in the filename parameter’s encoding because Windows is interested in the filename parameter’s encoding.

---

We effectively receive C-string (and don't bother in what encoding is it, it's just sequence of chars) and we ensuring that C-string is actually valid path in native filesystem. We do not using UTF-8 here at all.

That last sentence is very surprising to me. I thought that std::filesystem::path::u8string was supposed to return UTF-8, that std::filesystem::path::u16string was supposed to return UTF-16 and that std::filesystem::path::u32string was supposed to return UTF-32. cppreference.com says the following about those functions:

Returns the internal pathname in native pathname format, converted to specific string type. Conversion, if any, is performed as follows:

[…]

3) The result encoding in the case of u8string() is always UTF-8.

Additionally, Microsoft’s documentation for std::filesystem::path::u8string says:

Converts the sequence stored in mypath to UTF-8 and returns it stored in an object of type u8string.

What makes you say “We do not using UTF-8 here at all”?

---

It's just way to convert native encoding to C-string.

I agree. u8string().c_str() does indeed convert a path (which uses its native encoding internally) into a C-string. My concern is about the encoding of the resulting C-string.

[Lgt2x]

Really good job, tested successfully on Linux. Will also test Windows before merging. If someone can do OSX don't hesitate

[Lgt2x]

No problems in game in either Linux or Windows, we're good to merge. I hope we did not break anything for MacOS, but I don't have the hardware to test it. If UTF-8 proves to be a problem in some situations we can fix it later.