Convert locale files from ini to json format

[lunny]

• Migrate from the current INI format to JSON for translations. JSON is widely supported, including by platforms such as Crowdin and Weblate.
• Added a check for the json format and duplicate keys

TODO:

• After this PR merged, all JSON translations files should be uploaded to crowdin and switch to json files in crowdin platform.
• As a next step, we can consider converting to the i18next JSON format, which allows using placeholders like {name} instead of the current %[1]s syntax.

---

All original ini files have been converted with go run ./tools/convert_locale_ini_json.go. The file removed from this PR because it's an one-time program.

[silverwind]

Very good choice.

[wxiaoguang]

TBH I won't say it is a good choice. Full screen of \"

[wxiaoguang]

By the way, you also need to fix

• https://github.com/go-gitea/gitea/blob/main/.github/workflows/cron-translations.yml
• ./build/update-locales.sh

[silverwind]

Full screen of "

We should eventually remove all HTML from translations which will solve this. There are always ways to interpolate translation into templates without having HTML in the translation itself.

[wxiaoguang]

Full screen of "

We should eventually remove all HTML from translations which will solve this. There are always ways to interpolate translation into templates without having HTML in the translation itself.

The "eventually has been years, many "eventually" things have been years.

[hiifong]

I believe switching to TOML would be preferable, as existing translations can be easily converted to the TOML format, Nor will you see a screen full of \"

[silverwind]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

[wxiaoguang]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

Yes, any well-defined format is a big step.

Still think it should clean up the legacy HTML quotes before migrating, to make the code base clean and avoid low-level mistakes.

[hiifong]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

One drawback of JSON format is that it does not support adding comments.

[wxiaoguang]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

One drawback of JSON format is that it does not support adding comments.

Agree, that's also a problem.

[silverwind]

YAML has comments. Does crowdin support it? Thought I see no pressing need to comment translations.

[hiifong]

YAML has comments. Does crowdin support it? Thought I see no pressing need to comment translations.

Support, and also support TOML format.
ref: https://store.crowdin.com/categories/file-formats

[hiifong]

ref: #32937
I believe we should add comments in this situation.

[lunny]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

One drawback of JSON format is that it does not support adding comments.

Why do we need comments for translation keys? Currently, almost none of the existing translations include comments. Although JSON has it's drawback, but the format is parsed well by all kinds of languages and platforms like crowdin, weblate and others.

I have a plan to replace all %s to {user} in the translation strings, I think it's also a suitable time to replace all the HTML tags at the same PR. Otherwise, we have to replace to %s at this PR and replace it again next time.

[hiifong]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

One drawback of JSON format is that it does not support adding comments.

Why do we need comments for translation keys? Currently, almost none of the existing translations include comments. Although JSON has it's drawback, but the format is parsed well by all kinds of languages and platforms like crowdin, weblate and others.

I have a plan to replace all %s to {user} in the translation strings, I think it's also a suitable time to replace all the HTML tags at the same PR. Otherwise, we have to replace to %s at this PR and replace it again next time.

So how do you plan to replace the HTML tags?

[lunny]

I believe switching to TOML would be preferable, as existing translations can be easily converted to the TOML format, Nor will you see a screen full of \"

TOML is also a possible choice but it seems weblate haven't support it.
Replace the html tags one by one manually. There are many examples in the template files.

A drawback of the YAML format is that incorrect indentation can easily lead to invalid or unintended keys. i.e.
error: a
vs

form:
     error: a

[lunny]

@lunny CI is failing, and there is a merge conflict @wxiaoguang any further comments?

By the way, I guess you'd like to help to merge if there is no conflict, but I think you shouldn't help to do that for this one even if there is no merge conflict, because I believe there are other steps should be done immediately after the merge, for example: maintain Crowdin contents.

After this PR merged, all the JSON files should be uploaded to crowdin immediately.

[wxiaoguang]

After this PR merged, all the JSON files should be uploaded to crowdin immediately.

Do you have a TODO list? For example: should the GiteaBot be updated for its "I noticed you've updated the locales for non-English languages. " task? https://github.com/GiteaBot/gitea-backporter/blob/main/src/comments.ts#L9

[silverwind]

Yes we will need to update giteabot to support both .ini and .json files, best done before this PR is merged. cc: @yardenshoham.

[yardenshoham]

Sure thing.

• Support JSON-formatted locale files GiteaBot/gitea-backporter#144

[lunny]

After this PR merged, all the JSON files should be uploaded to crowdin immediately.

Do you have a TODO list? For example: should the GiteaBot be updated for its "I noticed you've updated the locales for non-English languages. " task? GiteaBot/gitea-backporter@main/src/comments.ts#L9

Updated the issue content.

[silverwind]

I find it odd that this file is not Record<string, string> but Record<string, Record<string, string>>. It would have been better if it were a flat list because that would make editing easier and would allow copying the key directly into templates. Was this a requirement related to crowdin or our own decision?

[lunny]

I find it odd that this file is not Record<string, string> but Record<string, Record<string, string>>. It would have been better if it were a flat list because that would make editing easier and would allow copying the key directly into templates. Was this a requirement related to crowdin or our own decision?

Both formats are supported by Crowdin; the choice to group the keys is an intentional design decision on our side. Given that we have over 3,600 keys, could you explain why you believe a flat structure would be preferable to the current approach?

[silverwind]

• Can copy the full key directly, without tedious scrolling up to find the parent key
• No ambiguity in queries foo.bar matches {"foo.bar": ""} only and not also {"foo": {"bar": ""}}. Ideally I would even forbid . in translation keys.
• I don't how key count would influence the design, 3.6k is not much and you can order however you like.

[yp05327]

By the way, the locale files are all broken after the first time sync.

[hiifong]

Honestly, I still believe converting from INI to TOML is the most suitable approach. INI currently lacks a clear specification, while TOML doesn't impose strict indentation requirements like YAML does. Moreover, existing INI-to-TOML formats are remarkably similar—incidentally, TOML is Rust's configuration file format.

[lunny]

By the way, the locale files are all broken after the first time sync.

reverted the auto merged crowdin update. I will check what happened.

[silverwind]

Whatever format it is, it needs a strict linter using a custom script. The current JSON linter catches general issues like duplicate keys, but would for example not catch if I start putting numbers in translation. If it is going to be TOML, I would validate it using https://github.com/squirrelchat/smol-toml in a small node script.

[lunny]

• Can copy the full key directly, without tedious scrolling up to find the parent key
• No ambiguity in queries foo.bar matches {"foo.bar": ""} only and not also {"foo": {"bar": ""}}. Ideally I would even forbid . in translation keys.
• I don't how key count would influence the design, 3.6k is not much and you can order however you like.

Sounds reasonable.

[wxiaoguang]

I really don't want to be a baby sitter to keep telling you to do this and do that.

But the locale files on main branch (https://github.com/go-gitea/gitea/blob/main/options/locale/locale_de-DE.json) are still broken, after 2 days.

Do you really know what you should do for this PR? Do you have a TODO list or a feasible plan?

[lunny]

I really don't want to be a baby sitter to keep telling you to do this and do that.

But the locale files on main branch (main/options/locale/locale_de-DE.json) are still broken, after 2 days.

Do you really know what you should do for this PR? Do you have a TODO list or a feasible plan?

The problem has been fixed. Only crowdin command tool can upload files with auto approved option. So that I updated it late.

Another problem is nested json empty translation lines will not be removed which is an official decision from crowdin, so that we have to remove them ourselves. #36225