cannot import name 'AutoTokenizer' from 'transformers' about langchain-chatchat HOT 2 CLOSED

To address the issue with importing AutoTokenizer from the transformers library during your build process, consider the following steps:

1. Check Environment Consistency: Ensure the build process is running in the same Python environment where transformers is installed. Environment discrepancies can lead to such import issues.

2. Verify Dependency Versions: Make sure the versions of transformers and tokenizers are compatible with each other and with your Python version (3.8.10). Version incompatibility is a common cause of import errors.

3. Reinstall Packages: Try reinstalling transformers and tokenizers to resolve any hidden issues. Use the command:

   pip install --upgrade --force-reinstall transformers tokenizers
   

   Ensure this is done within the correct environment.

4. PYTHONPATH Check: Confirm that the PYTHONPATH includes the directories where the packages are installed. Incorrect PYTHONPATH settings can prevent Python from finding installed packages.

5. Review Code for Conditional Imports: Look through the code for any conditional imports that might affect the availability of AutoTokenizer during the build process.

6. Debugging: Add logging before the import statement of AutoTokenizer to trace the execution path and use debugging to pinpoint where the import fails.

If after following these steps the issue persists, consider isolating the problem with a minimal script that only attempts to import AutoTokenizer during the build process. This can help determine if the issue is specific to the build environment or the script itself.

To continue the conversation, mention @dosu.

from langchain-chatchat.

已解决。
是因为embedding model: bge-large-zh下载的有问题，build时ADD的目录也不对。

from langchain-chatchat.