This page is part of the web mail archives of SRFI 52 from before July 7th, 2015. The new archives for SRFI 52 contain all messages, not just those from before July 7th, 2015.
>The other thing I didn't care for (in UTR 15) was the recommendation to >use NFC for case-sensitive languages and NFKC for case-insensitive >languages. NFC is designed for round-trip conversions, and it often uses >different encodings for visually indistinguishable symbols. For example, >the letters "ffi" and the "ffi" ligature are distinct under NFC (IIRC). >That's a very bad property for programming language identifiers. >Unfortunately, NFKC isn't perfect either. One thing I especially dislike >is that it flattens the differences between the mathematical alphabets. >Here you have a case where graphemes *are* visually distinguishable, and >for good reason, but the normalization form treats them as identical. If >you're working on a sublanguage for symbolic mathematics, you might be >tempted to write "double-struck small letter j" for a unit vector and >"italic small letter j" for the imaginary unit. But NFKC folds them >together. You'll need to modify NFKC for mathematics, or track the >semantic data separately (which amounts to the same thing). I'm pretty sure that the hashcode of an identifier (its 'identity') needs to be calculated from a normalized form of that identifier. I'm also pretty sure that a system which didn't also literally convert identical-hashing identifiers to identical representation and screen appearance would be worse instead of better, because the fundamental property of identifiers is that we the programmers need to be able to distinguish between them correctly. Note: In the following, I'm not going to talk about NFD or NFKD; they are equivalent in expressive power to NFC and NFKC, respectively, and we are unconcerned with issues of strictly internal representation. The points below apply to them as well. As you correctly point out there are problems with both NFC and with NFKC. However, the existence of NFKC makes the use of NFC identifiers risky in several ways. Although I like NFC better technically, I'd have to recommend NFKC as a standard because of the danger that text created to be distinguishable under NFKC is "safe" from bogus format changes to NFC, but the reverse is not true. Three scenarios follow. If an implementation goes with NFC, and an editor (even an editor that understands that these are different characters and doesn't confuse the save) fails to make separate, distinct glyphs for all those mathematical variants, then code that uses them to form indistinguishable-looking identifiers that are actually distinct is going to appear damned mysterious to users of that editor. Nothing breaks except the user's mind as he attempts to understand the code. However, if code is written for an NFC system and then read on a system that folds all of those distinctions by converting identifiers into NFKC form, then we will have the same problem, except that the compiler is now confused rather than the user. Now imagine an NFC system containing a source file with a function using identifiers which would be indistinguishable under NFKC -- possibly a math library. A user loads the source into an editor that treats it as "text" meaning NFKC, and without necessarily even seeing the function that uses the now-identical identifiers, he edits some unrelated function and saves the file. Breakage follows even though he has not migrated between compilers. We can 'standardize' NFC, which is capable of finer distinctions, and users will occasionally, until their personal learning curves are climbed, be burned by editors, tools, and nonconforming systems that treat program text as NFKC. Meanwhile code written for those nonconforming systems will not be burned by tools and editors and compilers that conform. or we can 'standardize' NFKC, which is capable of less distinction, thereboy limiting the range of identifiers available to programmers, and develop a generation of editors and compilers that will make the later adoption of NFC effectively impossible because program text for an NFC system cannot be reasonably kept out of 'standard' editors and tools. Or we can let the standard not-speak on this issue, and let a thousand flowers bloom... with chaotic results and lots of source code that won't go between systems and ad-hoc tools for converting from one form to another and etc etc etc with general chaos and nonportable code ensuing for decades. The disadvantages of this are obvious, but the advantage is that we don't lose implementors by demanding that they 'go with' one or the other before they've reached a real consensus about which is better. Remember, it's not a standard unless people follow it, and we're documenting which direction people have gone as much as we are telling them which direction to go. We don't know which direction they're going yet, and this bunch is as predictable as a herd of cats. If we try to standardize it, we may be guessing wrong. >It's especially bad if you're considering a language for typesetting >mathematics! (Not that anybody would ever want to implement a TeX-like >language in Scheme, right?) The Unicode character set is well-suited to >that task, but the normalization forms aren't, IMO. ah-heh... I assume there's a smiley in there somewhere... Bear