Re: Bug: grep behaves incorrectly under the locale C.UTF-8, if a file contains Umlaut characters

[Eric Blake <eblake at redhat dot com>]

On 05/24/2017 02:52 AM, Ronald Fischer wrote:
> I have a file X which contains ASCII text, but also in some lines German
> umlaut characters. The file is classified as:
> 
>      $ file X
>      X: ISO-8859 text, with CRLF line terminators

In ISO-8859, a German umlaut occupies one byte with the high-bit set.

> 
> However, if LANG is set to C.UTF-8, two things happen:
> 
> - grep classifies the file as binary file and produces the error message
> "Binary file X matches" 

In UTF-8, any one-byte sequence with the high bit set in isolation is an
encoding error (all high-bit bytes in UTF-8 occur in 2-or-more byte
sequences).  According to POSIX, grep is only required to operate on
text files, and the definition of a text file includes a requirement
that ALL bytes in the file form valid encodings of characters in the
current locale.  Yes, this means that there are files that are valid
text files in some locales and invalid in others (such as your file
here).  Once you violate the POSIX constraint of passing a non-text file
to grep, all bets are off, and grep can do whatever it wants, including
telling you that a binary file matches.

> 
> - Both the grepped lines (i.e. in our example the non-empty lines) AND
> the error message end up in the standard output (i.e. in file Y).

Yes, that's the current intended behavior in upstream grep. It's not
unique to Cygwin, so complaining here won't change it.

> 
> IMO, there are several problems with this:
> 
> 1. It's hard to see, why an umlaut character makes the file X binary
> under encoding C.UTF-8, 

Because it's not a valid UTF-8 encoding. Use iconv to convert your file
from ISO-8859 to UTF-8 if you want to grep it under C.UTF-8.

> but not under encoding UTF-8 or C.en_EN

Those aren't valid locale names.

But if you mean that it does what you want under LC_ALL=C, that's
because in the straight C locale, there are no multi-byte characters,
and therefore no encoding errors are possible, and therefore you can't
get a binary file in that locale due merely to an encoding error.

> 
> 2. If grep classifies a file as binary, I think the desired behaviour
> would be to NOT produce any output, unless the -a flag has been
> supplied.

Once behavior is in the realm of the undefined, it's hard to say what
the desired behavior should be. But again, if you want the current
behavior changed, it's an upstream issue to complain about on bug-grep,
and not something that I'm going to change for Cygwin in isolation.

> 
> 3. If grep writes a message "Binary file ... matches", this message
> should go to stderr, not stdout. The stdout is supposed to contain only
> a subset of the input lines.

The message "Binary file ... matches" has always gone to stdout, even
before upstream was tightened to flag more encoding errors as binary
files.  Whether the behavior of mixing it with actual output is
desirable is a question for upstream.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Attachment: signature.asc
Description: OpenPGP digital signature