[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Surrogates and character representation
Alan Watson writes:
> Using UTF-8 internally for a Scheme on a Plan 9 system is not obviously
> a bad idea. Sure, you don't have direct indexing, but you avoid
> conversion when you talk to the C library and OS.
> Using UTF-16 internally doesn't give you direct indexing because of
> characters outside the BMP, but it might make sense on Windows boxes for
> precisely the same reason.
This is a valid point. Python took the view that by default UTF-16 is
used internally then direct indexing into a string could yield part of
a surrogate pair. The feeling (as I remember, I may be wrong) was that
astral plane characters are rare-enough that the common-case (i.e.,
BMP) should not be penalized.
> Using UCS-32 internally in these cases would involve translation to talk
> to the library and OS and would further make my emacs use about four
> times as much memory as it does now (which brings us back the the
> representation for infinity).
Yes, though the glibc folks decided that the wchar_t type be a 4-byte
Unicode value. Python gives you the option of building with a 4-byte
or 2-byte "Unicode" character. (In Python Unicode and "narrow" strings
are separate types.)
> In general, any single representation is a bad idea in some circumstances.
Tom Emerson Basis Technology Corp.
Software Architect http://www.basistech.com
"Beware the lollipop of mediocrity: lick it once and you suck forever"