UnicodeDecodeError when connecting to Obsidian Vault about obsidiantools HOT 5 CLOSED

That was spot on. It is now reading my vault in Windows perfectly from the top level down. This is a really nice package. Great to have a bridge from Obsidian to NetworkX! Also a useful reminder for myself on explicit UTF-8 encoding. Many thanks.

from obsidiantools.

I looked into this further and tried running in VSCode to see if I could spot an issue. Validating the path with os.path.exists(vault_dir) returns True. I checked over my Obsidian note.md names for any slashes but the only non-alpha numeric characters I have are: '_', '-' and '&'.

from obsidiantools.

@virtualarchitectures thanks for raising this - this doesn't look like a filename issue from the error trace, it looks like this is happening when content in a file is being parsed.

Are you using non-Latin text in notes, any special characters, etc.? This is more info on the 0x9d character online. A character that you may not be able to see clearly in one of your notes, so it may be a pain to find which file has it.

It's not ideal that the vault 'connection' fails because of (possibly) one note not being able to be read. Would be great to see how we can make this more robust. 🙂

The python frontmatter parsing uses utf-8 as default and that's kept in the implementation of obsidiantools. It would be difficult to get an approach that can infer the right encoding of a file. One of your notes probably doesn't align with utf-8.

Initial ideas on what could be done as a 'fix', to let the vault setup complete:

• Robust but tricky: one way around this may be to introduce chardet as a dependency to handle files where utf-8 doesn't work
• Quick way: if file is not utf-8, print an error and let other files in the vault continue processing so that the vault object can still be set.

from obsidiantools.

Thanks for looking into this. I've isolated it to a folder containing of notes containing iFrames for YouTube videos. I can connect to the individual files .md files in the folder but not to the to the folder itself. Doing so raises the original Unicode Error. The notes in that folder don't contain any inlinks but they do have YAML frontmatter with tags. The tags seem pretty standard though.

I tried writing a loop to read each file with UTF-8 encoding using os.open() but was getting false 'file not found' errors. I think that was because I'm not so familiar with pathlib. If there's a loop you want me to try and run to isolate the error and get some more info I'm happy to give it a try.

from obsidiantools.

Try re-installing the package to bring in changes I've done in dev branch, and see if the error persists?:
pip install git+https://github.com/mfarragher/obsidiantools.git@dev

The main change I've done is to specify utf-8 encoding when a markdown file is opened. What might be happening is that your computer uses something different, so setting it as utf-8 explicitly might help: https://www.python.org/dev/peps/pep-0597/

from obsidiantools.