The default iso-8859-1 encoding unsafe, can be misdecoded by zbar as Big5 (was: Unicode-related test failures on musl libc) about segno HOT 14 CLOSED

I'm afraid our OpenCV build does not support decoding QR Codes right now. I've requested changing that, and I'll try to remember to test it again when that's done.

from segno.

Curious enough, the generated QRCode (at least according to the file produced by .save() with .txt format) is the same. However, zbar decodes it differently.

On glibc:

[Decoded(data=b'M\xc3\xa4rchenb\xc3\xbccher', type='QRCODE', rect=Rect(left=12, top=12, width=63, height=63), polygon=[Point(x=12, y=12), Point(x=12, y=75), Point(x=75, y=75), Point(x=75, y=12)], quality=1, orientation='UP')]

On musl:

[Decoded(data=b'M\xe9\x85\xbachenb\xf0\xa1\xa1\xb7her', type='QRCODE', rect=Rect(left=12, top=12, width=63, height=63), polygon=[Point(x=12, y=12), Point(x=12, y=75), Point(x=75, y=75), Point(x=75, y=12)], quality=1, orientation='UP')]

However, the test suite of pyzbar itself passes all tests…

from segno.

Ok, using zbarimg I've confirmed that the problem apparently lies in zbar and not here. Sorry for the noise.

from segno.

Actually, this may be a more significant bug. This only works on glibc because iso-8859-1 üc falls into "user-defined" Big5 characters. If I shorten the string to Märchen, zbar is confused by the iso-8859-1 encoding and decodes it as Big5. Perhaps it would be better to use UTF-8 after all?

from segno.

Thanks for your report. Acc. to ISO/IEC 18004 (3rd edition) the default encoding for QR codes should be ISO 8859-1.
The lib tries to use ISO 8859-1 and if the content does not fit, it falls back to UTF-8. However, you may enforce UTF-8 by using the "encoding" parameter:

import segno

qr = segno.make("Märchen", encoding="utf-8")

Acc. to the following test case it does not help, though. All tests fail (pyzbar-0.1.9, segno 1.6.1-dev, libzbar0 0.23.93-1, Python 3.11.7, PyPy 7.3.15, Debian trixie)

import io
import pytest
import segno
from pyzbar.pyzbar import decode as zbardecode


def qr_to_bytes(qrcode, scale):
    buff = io.BytesIO()
    for row in qrcode.matrix_iter(scale=scale):
        buff.write(bytearray(0x0 if b else 0xff for b in row))
    return buff.getvalue()


def decode(qrcode):
    scale = 3
    width, height = qrcode.symbol_size(scale=scale)
    qr_bytes = qr_to_bytes(qrcode, scale)
    decoded = zbardecode((qr_bytes, width, height))
    assert 1 == len(decoded)
    assert 'QRCODE' == decoded[0].type
    return decoded[0].data.decode('utf-8')


@pytest.mark.parametrize('encoding', [None, 'latin1', 'ISO-8859-1', 'utf-8'])
def test_issue134(encoding):
    # See <https://github.com/heuer/segno/issues/134>
    content = 'Märchen'
    qr = segno.make(content, encoding=encoding, micro=False)
    assert 'byte' == qr.mode
    assert content == decode(qr)


if __name__ == '__main__':
    pytest.main([__file__])

I don't know yet whether this is a Segno, pyzbar or zbar problem.

from segno.

It's a zbar problem. It's trying hard to decode QR codes as Big5 and Shift-JIS, before attempting ISO-8859-1. The bugs about that date back to 2012 (and many were filed since), so I don't think there's a much chance of zbar being ever fixed (and I have to admit that the code is a horror).

I don't know how significant zbar is, so I don't know if it's really worth caring for. But if you want things to work on zbar, I'm afraid you can't use ISO-8859-1 for non-ASCII characters.

If you don't care about zbar working, then I suppose it's either a matter of avoiding strings that are misdecoded by zbar (including "Märchenbücher"), or replacing zbar with something else (I don't know any library like that, though).

from segno.

Thanks for the information. zbar is not that important for me, I only use it for some test cases as it is easy to use and has few dependencies.

If you don't mind, I would close the bug. You are welcome to reopen it if I can do anything on the library side.

from segno.

Well, my point is that tests — as they are now — fail on musl systems.

from segno.

Okay, got it.

I can replace zbar with OpenCV. Would that help? Although it creates more dependencies for the test cases, I wouldn't mind

from segno.

I can replace zbar with OpenCV. Would that help? Although it creates more dependencies for the test cases, I wouldn't mind

Well, switching to OpenCV isn't a solution since all Kanji decoding test cases would fail.

from segno.

I suppose you could try zxing-cpp but I haven't tried it; I just recall ZXing app was quite good.

That said, for my purposes it would be sufficient to change the test strings to avoid the problems. However, I'm not sure how hard it would be to find some that work everywhere. If you want to proceed this route, I could try finding some.

from segno.

I think I have found a solution that is satisfactory. Please run the current test suite under musl and report any problems.

from segno.

I took out OpenCV again. That caused more problems than it helped.
Three tests under musl are now skipped. I think this is acceptable.
I tried the test suite on Void Linux/musl and, apart from the three tests skipped, it runs.
I think it will work with Alpine Linux as well.

from segno.

Ok, thanks a lot for your effort!

from segno.