Unicode file names aren't recognized

[vToMy]

Running ctags on Unicode file names fails to open them.

Example
For a file called:
こんにちは世界.txt
Running:
ctags --options=NONE *
Will produce:

ctags: Notice: No options will be read from files or environment
ctags: Warning: cannot open input file "???????.txt" : No such file or directory

[hadrielk]

Works for me:

$ ls
こんにちは世界.adoc

$ ../ctags --options=NONE -f - *
ctags: Notice: No options will be read from files or environment
Chapter 1 (Level 0)	こんにちは世界.adoc	/^= Chapter 1 (Level 0)$/;"	c
Level 3 Section 1.1.1.1 Title	こんにちは世界.adoc	/^==== Level 3 Section 1.1.1.1 Title$/;"	t	subsection:Chapter 1 (Level 0).Section 1.1.Subsection 1.1.1
Level 4 Section 1.1.1.1.1 Title	こんにちは世界.adoc	/^===== Level 4 Section 1.1.1.1.1 Title$/;"	T	subsubsection:Chapter 1 (Level 0).Section 1.1.Subsection 1.1.1.Level 3 Section 1.1.1.1 Title
Section 1.1	こんにちは世界.adoc	/^== Section 1.1$/;"	s	chapter:Chapter 1 (Level 0)
Subsection 1.1.1	こんにちは世界.adoc	/^=== Subsection 1.1.1$/;"	S	section:Chapter 1 (Level 0).Section 1.1

What operating system are you running on, and what version ctags?

[vToMy]

OS: Windows 10 (Version 10.0.17134 Build 17134)
ctags version:

>ctags --version
Universal Ctags 0.0.0(2258b24b), Copyright (C) 2015 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Aug 18 2018, 00:09:59
  URL: https://ctags.io/
  Optional compiled features: +win32, +wildcards, +regex, +internal-sort, +iconv, +option-directory, +xpath, +json, +interactive, +yaml, +case-insensitive-filenames

[k-takata]

This is a long-standing issue on Windows.
We currently use ANSI APIs, but we need to use Unicode APIs to handle Unicode file names.
This is a hard work, though.

[Lennon925]

do you want to support this or not ?

[masatake]

I want to support it but I don't know how to do it.

[k-takata]

Basically, we need to modify everywhere we handle filenames.
For example, we need to use _wmain() instead of main() to get UTF-16 command line, and need to use _wfopen() instead of fopen() to open a file with UTF-16 filename.
It might be better to create a wrapper layer for converting UTF-16 between UTF-8 and always use UTF-8 in the core part of u-ctags.

[Lennon925]

How long will you take to fix this issue？

[masatake]

@k-takata, thank you. Now I understand the meaning of "a hard work".

[masatake]

@Lennon925, I'm sorry but I have no plan to fix this.
We have to find a volunteer for fixing this issue.

[masatake]

As the first step, we have to add a test cast to Tmain.
Unlink, Units, there is no way to record a test case for a known bug.
Extending tmain test driver must be done first.

[masatake]

@k-takata, I tried a file having Japanese character as input for ctags on msys-2.
Unexpectedly, it works well. I think I'm doing something wrong. Could you give me more hints?

[k-takata]

On Japanese Windows, we can use Japanese characters, however, characters that cannot be represented by Shift_JIS (e.g. alphabets with diacritical mark, simplified Chinese characters, ...) cannot be used on Japanese Windows. Similarly, Japanese characters cannot be used on English Windows.

A workaround is using Cygwin (or MSYS2) version of u-ctags instead of Win32 version. It handles the filenames in UTF-8.

[Lennon925]

On Japanese Windows, we can use Japanese characters, however, characters that cannot be represented by Shift_JIS (e.g. alphabets with diacritical mark, simplified Chinese characters, ...) cannot be used on Japanese Windows. Similarly, Japanese characters cannot be used on English Windows.

A workaround is using Cygwin (or MSYS2) version of u-ctags instead of Win32 version. It handles the filenames in UTF-8.

Hi k-takata,
this issue is fixed? if not, when will it be finished?

Regards，
Lennon