National Replacement Character Sets

[j4james]

I noticed you've been adding a bunch of info on character sets, and I think it's worth mentioning how the National Replacement Character Sets (NRCs) work, because that's something that I think is not immediately obvious if you aren't familiar with the history of these terminals.

My understanding (which may be wrong) is that most if not all NRC sets will not work by default. You first have to enable National Replacement Character Set Mode (DECNRCM), and that itself only works if your keyboard dialect is set to something other than North American. Then even with DECNRCM enabled, you'll still only have access to one of the NRC sets - the one that matches your keyboard dialect.

The point of these sets was for use on 7-bit systems, so once DECNRCM is enabled, everything is 7-bit. If you've got a British keyboard, for example, then Shift+3 is labelled as producing a pound symbol (£), but the keyboard can't generate an 0xA3 codepoint on a 7-bit system. So instead it generates 0x23, and relies on the fact that the British NRC set has 0x23 mapped to £. That way it'll at least look like a £ when you type it, even though the backend system is still receiving an ASCII # character.

So unless you're tied to a 7-bit system with a specific non-ASCII locale, and you're also using a matching keyboard, there's not much use for these character sets. And I'm assuming that's why they only allow for one at a time (if my understanding is correct).

[hackerb9]

So unless you're tied to a 7-bit system with a specific non-ASCII locale, and you're also using a matching keyboard, there's not much use for these character sets.

Thank you. NRC was on my to-do list for figuring out. I am documenting things as I learn them and hadn't gotten around to NRC because (a) it was confusing, (b) it seemed unnecessary given features like SS3 (Single Shift to G3), and (c) it is disabled by default on my VT340 with no obvious way to enable it in Set-Up.

May I use your summary of NRC (properly attributed, of course)?

Your explanation about being tied to the keyboard matches my experience and helps me understand better how, even by the time the VT340 was released, NRC must've been too limiting.

It looks like a lot of terminals only have G0 and G1. Do you happen to know if the VT340 was the first to introduce G2 and G3?

The next things on my to-do list are:

• Find out capabilities and limitations of using G2/G3. (Is it that much better than setting G1? How many languages can I have on screen simultaneously?)
• Figure out 94- vs 96-character character sets. (Why can't G0 be assigned a 96 character set?)
• Is it correct to think of GL as the terminal's current interpretation of "low ASCII" (7-bit) chars and GR as so called "high ASCII"?
• What's up with the grotty looking large summation symbol on the VT340? Can I use the much nicer looking one from the VT420? Also, why are the components of it missing from Unicode?
• What is the ideal setup for modern usage given the VT340's lack of Unicode support? So far, Latin-1 has worked well for me, but I should be able to use charmap to display more Unicode symbols automatically.

Anything else you think I should cover in the character sets section?

[j4james]

May I use your summary of NRC (properly attributed, of course)?

Yeah, you're more than welcome to use anything I've posted here, and feel free to adapt it to better fit with the rest of your content. There's really no need for attribution.

even by the time the VT340 was released, NRC must've been too limiting.

I thought maybe it was for backwards compatibility with the VT100 terminals, but as far as I could find there were only a handful of a international VT100 variants (I think just Dutch, German, French, and French Canadian). So that doesn't explain why the VT240 and later terminals included so many additional NRC sets. Perhaps it was to support legacy servers that were still limited to 7-bits?

Do you happen to know if the VT340 was the first to introduce G2 and G3?

I know the VT220 already had G2 and G3 sets. I believe it is actually a conformance requirement for level 2 terminals and above.

Find out capabilities and limitations of using G2/G3. (Is it that much better than setting G1?

I think the main advantage of G2 and G3 is you can more easily map in a single character at a time with the SS2 and SS3 controls. That makes them useful for something like the special graphics set, where you're more likely to just want a one-off character from the set.

How many languages can I have on screen simultaneously?

I think the VT340 should probably be able to do five. I've got a little test script I was going to share with you which I'll follow up with later.

Why can't G0 be assigned a 96 character set?

Technically there is actually an escape sequence for that (I think from one of the ANSI standards), but it's just not supported by DEC. No idea why.

Is it correct to think of GL as the terminal's current interpretation of "low ASCII" (7-bit) chars and GR as so called "high ASCII"?

Yeah, that's the way I would look at it. And I think you'll find in some of the manuals that they'll often document a pair of character sets together, with one mapped into the "low ASCII" range, and the other in the "high ASCII" range. For example, you might have ASCII with Latin-1, or JIS-Roman with JIS-Katana.

What's up with the grotty looking large summation symbol on the VT340? Can I use the much nicer looking one from the VT420?

I'm not sure what you mean. I assumed the DEC Technical set was the same for all the terminals.

Also, why are the components of it missing from Unicode?

I looked into that a while back, and from what I could make out, there was a request to add them to Unicode, but they ended up merging many of them into the Mathematics range in a way that made them useless for a terminal. And even ignoring those flaws, there was at least one key character missing (I think the middle of the summation pattern).

At one point I was tempted to submit a proposal to Unicode myself to try and fix those issues, but I never could work up the enthusiasm.

What is the ideal setup for modern usage given the VT340's lack of Unicode support? So far, Latin-1 has worked well for me.

Yeah. I think ASCII plus Latin-1 is what most Western/European PC users would have worked with by default prior to Unicode. In fact, that's still kind of the default in Windows to some extent (technically Windows-1252 but that's mostly the same as Latin-1).

Anything else you think I should cover in the character sets section?

I think you may have mentioned this somewhere already, but the DECAUPSS sequence is worth discussing. I have some questions about that which I wanted raise.

[j4james]

Here's my little test script for the VT340 character sets: charsets.py. It tries to display a sample of characters from each of the supported sets, but it's expected that most of them won't work at the same time. On the left hand side are the National Replacement Character Sets, and everything else is on the right.

The way the script works, it first applies the DEC Special Graphic set (which is always expected to work), and then applies the character set that we're actually trying to test. So if that second character set designation fails, the sample characters will be from the Special Graphic set, and it will be obvious that it hasn't worked.

The script also starts by setting the DECNRCM mode, which should activate the NRC sets. However, your keyboard dialect is probably set to North American, which prevents DECNRCM from being enabled, so by default I'd expect none of the NRC sets to work, but some subset of those in the right hand column should work.

In this state, DEC Special, ASCII, DEC Technical, DEC Supplemental, and Latin-1 should always work, but I think only one of the two UPSS designations should work. If you've set your User Preference Char Set (in the settings) to DEC-MCS, then I'd expect UPSS 94 to work, and to match the DEC Supplemental set on the line above. If you've set the User Preference to Latin-1, then I'd expect UPSS 96 to work, and to match the ISO Latin-1 set on the line below. I'm not positive about this though - it's one of the things I was hoping you could confirm.

If you now go into the settings and change your keyboard dialect to one of the other languages, the DECNRCM mode should then be allowed, and the NRC sets should be active. So if you run the test again, I'd expect at least one of the sets on the left hand side to work (possible more when there are aliases, like the two Finnish sets, or the three Norwegian sets). And on the right hand side, I think only ASCII, DEC Special and maybe DEC Technical are likely to work (not sure about the last one).

[j4james]

Note that you can also change the User Preference Char Set with an escape sequence.

To select DEC MCS (aka DEC Supplemental):

echo -e '\033P0!u%5\033\\'

And to select ISO Latin-1:

echo -e '\033P1!uA\033\\'

The interesting thing is that the former is a 94-character set, and the latter is a 96-character set, so they're usually designated with different SCS sequences. For example, you use \033)%5 to designate DEC MCS as G1, but \033-A to designate Latin-1 as G1.

But I thought the whole point of the User Preference set was that an application could select it with the < ID without knowing what it was. But that doesn't work if they have to use \033)< when it's a 94-character set, and \033-< when it's a 96-character set.

So that's why I'm curious what you see in the UPSS 94 and UPSS 96 rows in my test script above. Those two cases are both attempting to designate the UPSS (<) set, but the one is using a 94 SCS sequence, and the other is using a 96 SCS sequence.

[hackerb9]

In this state, DEC Special, ASCII, DEC Technical, DEC Supplemental, and Latin-1 should always work, but I think only one of the two UPSS designations should work. If you've set your User Preference Char Set (in the settings) to DEC-MCS, then I'd expect UPSS 94 to work, and to match the DEC Supplemental set on the line above. If you've set the User Preference to Latin-1, then I'd expect UPSS 96 to work, and to match the ISO Latin-1 set on the line below. I'm not positive about this though - it's one of the things I was hoping you could confirm.

For the first half of the test, with keyboard set to "North American" and user preference set to "Latin-1", both UPSS 94 and 96 show the same as ISO Latin-1. After changing the user preference to "MCS", they both match DEC Supplemental.

[I'm waiting for the sixel mediacopy to save to disk. When that's finished, I'll change my keyboard and try again.]

[hackerb9]

With keyboard set to Flemish and user pref set to Latin-1, the result is that only "French (R)" has special characters on the left. On the right, everything except ASCII is identical with DEC Special.

Once the keyboard is not North American, the Character Set Mode option can now be changed from "multinational" to "national", but it is irrelevant since charset.py always switches it back to multinational.

Changing User Preference Char Set from ISO Latin-1 to DEC-MCS also has no effect.

Trying the same test with keyboard set to "Norwegian" shows all three Norwegian sets on the left as identical. Everything else (except ASCII) looks like the Special Graphics.

[j4james]

For the first half of the test, with keyboard set to "North American" and user preference set to "Latin-1", both UPSS 94 and 96 show the same as ISO Latin-1. After changing the user preference to "MCS", they both match DEC Supplemental.

OK. That's very interesting. That does make the UPSS set more useful than I thought it would be. But now I'm wondering whether there might still be a difference in behavior between the 94 and 96 designations.

Would you mind updating the sample text to include the code point \x7F, and then run the Latin-1 version of the test again. So change this line:

write('_#@[\\]^`{|}\x7F~')

to something like this:

write('_#@[\\]^\x7F`{|}\x7F~')

In almost all cases, that \x7F should not display anything (similar to a NUL control), but in Latin-1 it should map to ÿ. What I'm interested in seeing is whether both UPSS variants display that glyph, or only one of them.

With keyboard set to Flemish and user pref set to Latin-1, the result is that only "French (R)" has special characters on the left. On the right, everything except ASCII is identical with DEC Special.

I guess that's reasonable if you think of it as something like a VT100-compatible mode, since all you would have had access to there is ASCII, DEC Special, and whatever the national set was.

[hackerb9]

In almost all cases, that \x7F should not display anything (similar to a NUL control), but in Latin-1 it should map to ÿ. What I'm interested in seeing is whether both UPSS variants display that glyph, or only one of them.

Both. (And, ISO Latin-1, of course.)

When keyboard is set to "British", the script does not produce ÿ. I am still able to type and display one by using Compose Character " y. After additionally changing Set-Up → General Set-Up → Character Set Mode from "multinational" to "national", the Compose key no longer works to type ÿ. While the previous ÿs still show on the screen, newly printed ones are not displayed.

[j4james]

Both. (And, ISO Latin-1, of course.)

OK, thanks. That's good to know. You've uncovered a couple of issues I need to fix in Windows Terminal.

When keyboard is set to "British", the script does not produce ÿ. I am still able to type and display one by using Compose Character " y.

That's understandable. When the keyboard is set to "British", that allows DECNRCM to be enabled, which is something the script does automatically on startup. At that point you're in 7-bit mode, and you're assumedly only getting ASCII, British, and DEC Special. But the script also turn DECNRCM off again when it's finished, so once you're back at the prompt you'll have access to ÿ again.

While the previous ÿs still show on the screen, newly printed ones are not displayed.

Again that makes sense to me. My understanding is that there are a certain range of code points that have glyphs permanently assigned to them. I'm assuming this includes the Latin-1 characters, the various graphics characters, and part of ASCII. The ÿ from Latin-1 would thus be one of these permanent assignments.

Part of the ASCII range is replaceable, though, so for example you can have code point 0x23 stored in the screen buffer, but sometimes rendered as #, and sometimes rendered as £. The interesting thing is that £ is also part of Latin-1, so you can have two different versions of the £ glyph on the screen, only one of which will update when you switch the keyboard dialect.

For example, set your keyboard to "British", make sure your mode is set to "Multinational", and type some £ characters (I'm assuming Shift+3). Then switch the mode to "National", and type some more £ characters. Finally, switch the keyboard to something like "Danish", which has 0x23 mapped to #. I'm expecting the second set of £ glyphs to redraw as #, but the first set should remain unchanged.

[hackerb9]

For example, set your keyboard to "British", make sure your mode is set to "Multinational", and type some £ characters (I'm assuming Shift+3). Then switch the mode to "National", and type some more £ characters. Finally, switch the keyboard to something like "Danish", which has 0x23 mapped to #. I'm expecting the second set of £ glyphs to redraw as #, but the first set should remain unchanged.

Surprisingly, they always remain British pound signs, whether typed in using Shift3 or via ComposeL-. The only difference was that in Danish National mode, I could no longer use Compose to enter new British pound signs. (It would just beep).

Perhaps the VT340 is storing "national" character sets in Latin-1 internally?

[j4james]

Surprisingly, they always remain British pound signs

That is interesting, because they definitely update on the VT240. I wonder if perhaps it's just that the VT340 doesn't redraw the screen when changing modes? On the VT240 you can see clearly see it repainting line by line.

If that is the case, one way you could possibly force a redraw would be by toggling the line attributes with DECDWL/DECSWL, or inserting a space at the start of the line with ICH. Although it's probably a bit of a pain trying to do that when you're working with a foreign keyboard.

Anyway, it's not a big deal - I'm not planning to emulate that behaviour. I just thought it was an interesting effect. Although for higher level terminals, it might be worth investigating further. Because if those characters are different, I suspect the checksum report (DECRQCRA) would need to return different values for them.

[hackerb9]

Would you expect there to be a difference for Media Copy (CSIi)?

[j4james]

That's a good question, but I've just checked on my MAME VT240, and it doesn't make a difference there, so I suspect you'll get the same results.

There's a printer option that lets you choose between "multinational" and "national and line drawing", and that determines whether the £ symbol is output as an 8-bit value, or using escape sequences to shift in an alternate character set. Both forms of the £ are output the same way - so either both will be 8-bit, or both will use escape sequences.

[j4james]

That reminds me, you were asking earlier about the first terminals to introduce G2 and G3, and I said it was probably the VT220. However, I recently came across some documentation that pointed out that the VT100 technically had G2 and G3 tables as well, and I've since confirmed that in MAME.

There's not much you can do with them, because you can't designate an alternate character set into G2 and G3 (they're hardcoded as ASCII). Also the locking shifts don't work, so they can only be mapped in with a single shift. But here's an example usage:

echo -e "\e(A" # Sets the default charset to British
echo -e "#"    # Outputs a pound symbol
echo -e "\eN#" # Outputs a hash symbol

And I'm thinking now that might actually provide some value if you've got a VT100 with a national character set other than British. Because you could make your national character set the default in G0, put the DEC Special Graphics in G1, and rely on ASCII being in G2 for any characters that aren't available in the other two sets.

But that isn't much help with a British character set, since the only ASCII character missing from British is #, and you can just as easily get that from the DEC Special Graphics set.

[j4james]

@hackerb9 I noticed a commit recently where you were discussing the ISO-IR standards, and noted that most of the VT340 character sets were in the "private use" category. However, it's worth mentioning that at least the NRC sets are mostly standard, although DEC also has private aliases for some of them. These are the ones I'm aware of on the VT340:

• British
• Finnish
• French
• German
• Italian
• Norwegian/Danish (there's also a second version of Norwegian, but I think DEC just treats it as an alias - although it might be worth testing this actually)
• Spanish
• Swedish

In theory French Canadian should also be in this list, but I wasn't able to find it in the ISO-IR registry.

And from the VT382-J we also have these two:

• JIS-Katakana
• JIS-Roman