[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Surrogates and character representation
Tom Emerson wrote:
Let's look at how I handle these in Python right now: the UTF-8 data
is read and transcoded to the internal Unicode string format.
Ah, so you're not doing random-access on "multimegabyte text files" as
as we assumed from your initial message.
If you have the luxury of reading your entire file into memory (and in
the process expanding its size by a good bit) you can of course do all
kinds of processing and index-building.
It appears (from http://www.jorendorff.com/articles/unicode/python.html)
that Python unicode strings are UTF-16 strings, so character offsets
will break as soon as you go beyond the Basic Multilingual Plane.
Scheme implementations can of course fix this, though it means using
4 bytes per character. Hence the discussion.