[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Should SRFI-115 character sets match extended grapheme clusters?

This page is part of the web mail archives of SRFI 115 from before July 7th, 2015. The new archives for SRFI 115 contain all messages, not just those from before July 7th, 2015.

On Sun, May 11, 2014 at 10:49 PM, Mark H Weaver <mhw@xxxxxxxxxx> wrote:
Hello all,

It occurs to me that users of languages that make heavy use of combining
marks will likely find the behavior of "character sets" to be quite
unintuitive if they operate on code points.  For example, they might
reasonably expect ("éè") to match either of two graphemes, and never to
match a bare 'e' or a bare combining mark.  They might also expect
(~ ("aeiou")) to match "é", even when represented as multiple code

An excellent point.  I'm inclined to agree that we should handle
grapheme clusters naturally, though it's not completely clear that
this is always the right thing.  Users may want to create character
sets with composing characters and have them matched individually,
not composed with any preceding character (though the workaround
here would be to include them as characters, not inside string literals).

Regardless, I think for practical reasons we're stuck with codepoints
for now.  There exist (to my knowledge) no implementations which
handle grapheme cluster aware char-sets currently, and the overhead
is potentially substantial.  Even simple mappings of large Unicode
char-sets can be expensive to compute (until the ref impl optimized
known case-insensitive char-sets I believe (w/nocase letter) took
over a minute to iterate over all 10k+ Letter code-points, look up all
their case variants an insert them into a new set).  So I'd consider
this an open area for improvement.

We could make room for this, in the same way that we allow but
don't require w/nocase to handle non-1-to-1 mappings.  The concern
again is how do we distinguish ("éè") with non-composed characters
from the character set where they want to match e and the two composing
characters separately, and how do we translate the distinction to pcre?