UTF8 filename come back mangles from scandir/walk about scandir HOT 2 CLOSED

This is (annoying) but expected behaviour, due to the way byte and unicode filenames are handled in Python 2.x. To get around it, just pass a unicode string instead of a byte string to scandir, like you're doing with open(), for example:

scandir.scandir(u"C:\Temp\xx鳭僣yy.txt")

Let me know if this works. Note that this is (or it should be!) the same behaviour as os.listdir() on Python 2.x.

from scandir.

Sorry, I said "filename" and copied the unicode filename rather than the unicode directory.

But no, my module isn't "broken". :-) It's operating by design, as per os.listdir(). The behaviour of bytes paths on Windows is kinda weird -- if you pass in a byte string, you get out byte strings with non-ASCII chars replaced by ? characters on Windows. This is different from on Linux, where you get UTF-8. So bytes paths are kind of half broken on Windows Python.

What you need to do is simply path a unicode path to scandir. Like so:

>>> os.mkdir('temp')
>>> f = open(u'temp\\xx\u9ced\u50e3yy.txt', 'w')  # create a unicode filename
>>> f.close()
>>> [e.name for e in scandir.scandir('temp')]  # this is what you are doing
['xx??yy.txt']
>>> [e.name for e in scandir.scandir(u'temp')]  # this is what you need to do
[u'xx\u9ced\u50e3yy.txt']
>>> [e.name.encode('utf-8') for e in scandir.scandir(u'temp')]  # or as UTF-8
['xx\xe9\xb3\xad\xe5\x83\xa3yy.txt']

Note that, by design, this exactly matches the behaviour of os.listdir() on Python 2.x:

>>> os.listdir('temp')
['xx??yy.txt']
>>> os.listdir(u'temp')
[u'xx\u9ced\u50e3yy.txt']

from scandir.