This page is part of the web mail archives of SRFI 75 from before July 7th, 2015. The new archives for SRFI 75 contain all messages, not just those from before July 7th, 2015.
Tom Emerson wrote:
Let's look at how I handle these in Python right now: the UTF-8 data is read and transcoded to the internal Unicode string format.
Ah, so you're not doing random-access on "multimegabyte text files" as as we assumed from your initial message. If you have the luxury of reading your entire file into memory (and in the process expanding its size by a good bit) you can of course do all kinds of processing and index-building. It appears (from http://www.jorendorff.com/articles/unicode/python.html) that Python unicode strings are UTF-16 strings, so character offsets will break as soon as you go beyond the Basic Multilingual Plane. Scheme implementations can of course fix this, though it means using 4 bytes per character. Hence the discussion. -- --Per Bothner per@xxxxxxxxxxx http://per.bothner.com/