[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Surrogates and character representation
> All other undefined codepoints are potentially definable: they correspond
> to Unicode scalar values. Surrogate codepoints are not definable and
> don't correspond to any Unicode scalar value. The difference is
FFFE is never (by architectural design) going to be defined
Surrogate codepoints have a character property. They should be usable
in a string, and individually can be considered a character. Most
implementations won't see them: only library code that is
reading/writing UTF-16 needs to worry about them in any significant
way. Application code should not see them. They will see U+20069 as
having the value 0x20069, not 0xD840DC69.
In other words, I guess I'm saying that surrogates don't need to be
special cased, because the existing Unicode property model accounts
for them, and the generation/interpretation of them should be handled
at a lower level. Special casing them just complicates everything for
> > One question I've had: how are 8-bit (i.e., byte) strings handled
> > here? Is there no distinction between operations on raw bytes and
> > operations on characters?
> Those things are not strings: they are vectors of unsigned 8-bit integers.
Of course. My Python hat is still on where 8-bit strings and Unicode
strings are different beasts, and 8-bit strings are used for
Tom Emerson Basis Technology Corp.
Software Architect http://www.basistech.com
"Beware the lollipop of mediocrity: lick it once and you suck forever"