OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already intialized about argos-translate HOT 10 CLOSED

Just pushed a patch to PyPI fixing this 1.1.3.

from argos-translate.

libiomp5.dylib is the Intel OpenMP runtime that is used for multithreading. It is required by CTranslate2 and is integrated in the Python package. But Argos Translate also uses Stanza that depends on PyTorch which includes another copy of libiomp5.dylib. This is probably why it complains about multiple copies of the library.

Both PyTorch and CTranslate2 are correct to bundle this library into their Python package, so I'm not sure what should be done here. The workaround suggested in the error message should work and I don't think it will cause issues.

from argos-translate.

I'm happy to try and branch and make a pull request for this one - never contributed to a project that's not my own before!

from argos-translate.

Thanks for the amazing project.

Thanks for using it!

The web app launches

Argos Translate has a PyQt native app, LibreTranslate is a web app/api build on top of Argos Translate which are you using?

I've never seen this before but looking around it looks like this is probably some weird interaction between CTranslate and Intel CPUs on Mac. It looks like export KMP_DUPLICATE_LIB_OK=True may fix the issue but I don't really understand why. It seems like it's a bit of a hack and may have negative side effects. @guillaumekln any idea?

Related issues:

• dmlc/xgboost#1715
• https://stackoverflow.com/questions/53014306/error-15-initializing-libiomp5-dylib-but-found-libiomp5-dylib-already-initial

from argos-translate.

Interesting, that's a tricky one. If this is a reoccurring issue maybe Argos Translate should set the KMP_DUPLICATE_LIB_OK environment variable automatically.

from argos-translate.

I have a potential fix for this pushed up. @nikohunt can you try installing from source to see if this fixes the issue? I didn't want to always set the environment variable so I try to detect if Argos Translate is running on an Intel processor with platform.processor(). This isn't foolproof so if this error reoccurs then we can change to always setting the environment variable.

from argos-translate.

Thanks a lot @PJ-Finlay! I pulled down argos just now and built; still the same issue. However, upon checking the fix, I ran something similar on my machine and noticed the following:

>>>import platform  
>>>print(platform.processor().lower())  
i386

from argos-translate.

That was what I was worried about. It doesn't seem like there is a good way to consistently determine if someone is using an Intel CPU with the Python standard library and I don't want to add a dependency for this. platform.processor().lower() doesn't include the CPU type on my Ubuntu 20.04 setup and it looks like it doesn't on yours either (I'm guessing you're using Mac?). I think the best fallback is going to be to just set the environment variable for everyone, it seems specific enough that it won't cause problems.

I'm happy to try and branch and make a pull request for this one - never contributed to a project that's not my own before!

That would be great! I would look into the Python platform docs and see if there is something that works on Mac/Intel otherwise just remove the if statment.

from argos-translate.

Thanks. Yes, I'm on a macbook. I can use the following to get the "intel" piece out reliably in the terminal:
'sysctl -n machdep.cpu.brand_string'
I would need a way to run this reliably, though, as obviously this would error out in other hardware scenarios - is try/except considered bad form? I could check with platform.system()first to see if darwin...
Thanks again.

from argos-translate.

I think that would work but if it's this involved I think it's better to just remove the if statement and set the environment variable for everyone. I have an Intel CPU and don't have this issue on Linux but this could also happen on Windows so best to be as broad as possible.

Thanks for handling this.

from argos-translate.