[1.15.2][Solved] Problem with diacritic

[fcelon]

Hi,

when language file in my mod contains characters with diacritic like š č ř or ž, those characters render as ? in game. However, the same characters render just fine in vanilla texts and in chat.

cs_cz.json:

{
	"entity.wt_improvements.decoy":"Dvojník",
	"item.wt_improvements.totem_of_decoy":"Totem zdvojení",
	"block.wt_improvements.trade_station":"Obchodní stanice",
	"item.wt_improvements.trade_station":"Obchodní stanice",
	"item.minecraft.potion.effect.mighty_night_vision": "Silný lektvar vidění ve tmě",
	"item.minecraft.potion.effect.mighty_invisibility": "Silný lektvar neviditelnosti",
	"item.minecraft.potion.effect.mighty_leaping": "Silný lektvar skákání",
	"item.minecraft.potion.effect.mighty_fire_resistance": "Silný lektvar ohnivzdornosti",
	"item.minecraft.potion.effect.mighty_swiftness": "Silný lektvar svižnosti",
	"item.minecraft.potion.effect.mighty_turtle_master": "Silný lektvar želvího mistra",
	"item.minecraft.potion.effect.mighty_water_breathing": "Silný lektvar dýchání pod vodou",
	"item.minecraft.potion.effect.mighty_healing": "Silný lektvar léčení",
	"item.minecraft.potion.effect.mighty_regeneration": "Silný lektvar regenerace",
	"item.minecraft.potion.effect.mighty_strength": "Silný lektvar síly",
	"item.minecraft.potion.effect.mighty_slow_falling": "Silný lektvar pomalého padání",
	"item.minecraft.potion.effect.mighty_fortitude": "Silný lektvar odolnosti",
	"item.minecraft.potion.effect.mighty_danger_sight": "Silný lektvar zviditelnění nebezpečí",
	"item.minecraft.potion.effect.mighty_stability": "Silný lektvar stability",
	"item.minecraft.potion.effect.mighty_relief": "Silný lektvar úlevy",
	"item.minecraft.potion.effect.mighty_restfulness": "Silný lektvar odpočinku",
	"item.minecraft.potion.effect.mighty_repellence": "Silný lektvar odpudivosti",
	"item.minecraft.potion.effect.mighty_vibing": "Silný lektvar přitažlivosti",
	"item.minecraft.potion.effect.mighty_luck": "Silný lektvar štěstí",
	"item.minecraft.splash_potion.effect.mighty_night_vision": "Silný vrhací lektvar vidění ve tmě",
	"item.minecraft.splash_potion.effect.mighty_invisibility": "Silný vrhací lektvar neviditelnosti",
	"item.minecraft.splash_potion.effect.mighty_leaping": "Silný vrhací lektvar skákání",
	"item.minecraft.splash_potion.effect.mighty_fire_resistance": "Silný vrhací lektvar ohnivzdornosti",
	"item.minecraft.splash_potion.effect.mighty_swiftness": "Silný vrhací lektvar svižnosti",
	"item.minecraft.splash_potion.effect.mighty_turtle_master": "Silný vrhací lektvar želvího mistra",
	"item.minecraft.splash_potion.effect.mighty_water_breathing": "Silný vrhací lektvar dýchání pod vodou",
	"item.minecraft.splash_potion.effect.mighty_healing": "Silný vrhací lektvar léčení",
	"item.minecraft.splash_potion.effect.mighty_regeneration": "Silný vrhací lektvar regenerace",
	"item.minecraft.splash_potion.effect.mighty_strength": "Silný vrhací lektvar síly",
	"item.minecraft.splash_potion.effect.mighty_slow_falling": "Silný vrhací lektvar pomalého padání",
	"item.minecraft.splash_potion.effect.mighty_fortitude": "Silný vrhací lektvar odolnosti",
	"item.minecraft.splash_potion.effect.mighty_danger_sight": "Silný vrhací lektvar zviditelnění nebezpečí",
	"item.minecraft.splash_potion.effect.mighty_stability": "Silný vrhací lektvar stability",
	"item.minecraft.splash_potion.effect.mighty_relief": "Silný vrhací lektvar úlevy",
	"item.minecraft.splash_potion.effect.mighty_restfulness": "Silný vrhací lektvar odpočinku",
	"item.minecraft.splash_potion.effect.mighty_repellence": "Silný vrhací lektvar odpudivosti",
	"item.minecraft.splash_potion.effect.mighty_vibing": "Silný vrhací lektvar přitažlivosti",
	"item.minecraft.splash_potion.effect.mighty_luck": "Silný vrhací lektvar štěstí",
	"item.minecraft.lingering_potion.effect.mighty_night_vision": "Silný doznívající lektvar vidění ve tmě",
	"item.minecraft.lingering_potion.effect.mighty_invisibility": "Silný doznívající lektvar neviditelnosti",
	"item.minecraft.lingering_potion.effect.mighty_leaping": "Silný doznívající lektvar skákání",
	"item.minecraft.lingering_potion.effect.mighty_fire_resistance": "Silný doznívající lektvar ohnivzdornosti",
	"item.minecraft.lingering_potion.effect.mighty_swiftness": "Silný doznívající lektvar svižnosti",
	"item.minecraft.lingering_potion.effect.mighty_turtle_master": "Silný doznívající lektvar želvího mistra",
	"item.minecraft.lingering_potion.effect.mighty_water_breathing": "Silný doznívající lektvar dýchání pod vodou",
	"item.minecraft.lingering_potion.effect.mighty_healing": "Silný doznívající lektvar léčení",
	"item.minecraft.lingering_potion.effect.mighty_regeneration": "Silný doznívající lektvar regenerace",
	"item.minecraft.lingering_potion.effect.mighty_strength": "Silný doznívající lektvar síly",
	"item.minecraft.lingering_potion.effect.mighty_slow_falling": "Silný doznívající lektvar pomalého padání",
	"item.minecraft.lingering_potion.effect.mighty_fortitude": "Silný doznívající lektvar odolnosti",
	"item.minecraft.lingering_potion.effect.mighty_danger_sight": "Silný doznívající lektvar zviditelnění nebezpečí",
	"item.minecraft.lingering_potion.effect.mighty_stability": "Silný doznívající lektvar stability",
	"item.minecraft.lingering_potion.effect.mighty_relief": "Silný doznívající lektvar úlevy",
	"item.minecraft.lingering_potion.effect.mighty_restfulness": "Silný doznívající lektvar odpočinku",
	"item.minecraft.lingering_potion.effect.mighty_repellence": "Silný doznívající lektvar odpudivosti",
	"item.minecraft.lingering_potion.effect.mighty_vibing": "Silný doznívající lektvar přitažlivosti",
	"item.minecraft.lingering_potion.effect.mighty_luck": "Silný doznívající lektvar štěstí",
	"item.minecraft.tipped_arrow.effect.mighty_night_vision": "Silný šíp vidění ve tmě",
	"item.minecraft.tipped_arrow.effect.mighty_invisibility": "Silný šíp neviditelnosti",
	"item.minecraft.tipped_arrow.effect.mighty_leaping": "Silný šíp skákání",
	"item.minecraft.tipped_arrow.effect.mighty_fire_resistance": "Silný šíp ohnivzdornosti",
	"item.minecraft.tipped_arrow.effect.mighty_swiftness": "Silný šíp svižnosti",
	"item.minecraft.tipped_arrow.effect.mighty_turtle_master": "Silný šíp želvího mistra",
	"item.minecraft.tipped_arrow.effect.mighty_water_breathing": "Silný šíp dýchání pod vodou",
	"item.minecraft.tipped_arrow.effect.mighty_healing": "Silný šíp léčení",
	"item.minecraft.tipped_arrow.effect.mighty_regeneration": "Silný šíp regenerace",
	"item.minecraft.tipped_arrow.effect.mighty_strength": "Silný šíp síly",
	"item.minecraft.tipped_arrow.effect.mighty_slow_falling": "Silný šíp pomalého padání",
	"item.minecraft.tipped_arrow.effect.mighty_fortitude": "Silný šíp odolnosti",
	"item.minecraft.tipped_arrow.effect.mighty_danger_sight": "Silný šíp zviditelnění nebezpečí",
	"item.minecraft.tipped_arrow.effect.mighty_stability": "Silný šíp stability",
	"item.minecraft.tipped_arrow.effect.mighty_relief": "Silný šíp úlevy",
	"item.minecraft.tipped_arrow.effect.mighty_restfulness": "Silný šíp odpočinku",
	"item.minecraft.tipped_arrow.effect.mighty_repellence": "Silný šíp odpudivosti",
	"item.minecraft.tipped_arrow.effect.mighty_vibing": "Silný šíp přitažlivosti",
	"item.minecraft.tipped_arrow.effect.mighty_luck": "Silný šíp štěstí",
	"subtitles.wt_improvements.block.trade_station.ding": "Zvonek stanice cinká"
}

result:

Does anyone know can I solve this problem?

Edited September 16, 2020 by fcelon

[Beethoven92]

mmm i don't have knowledge on the matter, but i took a look inside the vanilla cs_cz.json and i see they are using a lot of formatting codes which look like that: \u00e1..i believe those are doing the job to get the special characters you need. Take a look into the vanilla file

[fcelon]

Thakns you. I have been unable to locate vanilla cs_cz.json in my computer, but replacing problematic characters with codes seems to fix the issue.

[Beethoven92]

Deleted link to minecraft assets

Edited September 16, 2020 by Beethoven92

[Curle]

Note the Minecraft End-User License Agreement:

Quote

so you are not allowed to distribute the Game (or any part of it) to anyone else. [...]
Also, don't just rip art resources and pass them around, that's no fun.

 

That link breaks the Minecraft EULA/TOS, and is most likely very illegal. Do not recommend this in the future, or there will be worse repercussions.

The actual fix for this problem would likely be to change the encoding of the language json file. UTF-8 BLE does not encode these characters well, but the game is equipped to handle more sophisticated encodings. UTF-16 should work.

[Beethoven92]

2 minutes ago, Curle said:

Note the Minecraft End-User License Agreement:

 

That link breaks the Minecraft EULA/TOS, and is most likely very illegal. Do not recommend this in the future, or there will be worse repercussions.

The actual fix for this problem would likely be to change the encoding of the language json file. UTF-8 BLE does not encode these characters well, but the game is equipped to handle more sophisticated encodings. UTF-16 should work.

Oh well, i was not aware of that...i mean, those files are literally on a public github page that anyone can consult...its not like i ripped those myself and then distributed them..i just linked an existing resource on the web. But i understand what the problem could be, so i apologize if i broke the rules. I will remember next time. Thank you for letting me know that.

[Curle]

That repository breaks several copyright and EULA laws and has been reported numerous times.

Github's inaction does not make it okay.

[Beethoven92]

3 minutes ago, Curle said:

That repository breaks several copyright and EULA laws and has been reported numerous times.

Github's inaction does not make it okay.

I guess i have to thanks again because i was not aware even of that..

[diesieben07]

9 hours ago, Curle said:

UTF-8 BLE does not encode these characters well

I don't know what "UTF-8 BLE" is. The only thing Google can come up with for that is UTF-8 issues when using Bluetooth Low Energy, which is certainly not applicable here.

In other things, UTF-8 is perfectly capable of encoding all of unicode, that is the point of it. This is why it is called UTF. The U stands for UCS, which is Unicode Transformation Format: A way to represent unicode characters as bytes.

 

There is almost never a reason to use any character encoding other than UTF-8 unless you are dealing with a legacy system, imho.

[diesieben07]

As for the issue at hand here, the best way to solve it would be to use data generators, as they "shield" you from any intricacies the format might have.

However, language JSON files are loaded as UTF-8. Make sure your files are in UTF-8 and they will work fine. Unicode characters are allowed in JSON strings.

 

Also, when using characters outside of ASCII in your code (Java files) note that javac does not use UTF-8 by default. It uses whatever the system default is, which is a) usually a terrible encoding and b) not portable at all. I highly suggest you configure it to use UTF-8.

[Curle]

Quote

I don't know what "UTF-8 BLE" is.

Apologies, i meant UTF-8 BE, the default encoding of most modern text editors.

There are well known problems in the standard Java libraries with reading / decoding these from a file on disk.

 

You're right about all the rest though

[diesieben07]

Still no idea what UTF-8 BE is.

[Alpvax]

I'm assuming big-endian. But TBH, I thought the byte order was part of the UTF-8 spec. I've never seen it called UTF-8 BE.

[diesieben07]

Yes, the smallest unit in UTF-8 is the byte (not for example a 2-byte word like in UTF-16). Therefor the byte order has no meaning. You can put a BOM in a UTF-8 file but it is discouraged because it has no meaning.