Comments (11)
That's great! Yes, khoj
supports multiple org files.
- Just click the "Add" button multiple times on the Configure GUI to add specific org files. Example shown below:
- The GUI (currently) doesn't provide a way to add all org files under a directory. But you can do that by manually editing the
input-filter
field in the~/.khoj/khoj.yml
file. Example shown below:
from khoj.
Hey @yibie, try run the commands below and see if that fixes your issue?
pip3 install setuptools-rust
pip3 install --upgrade pip
pip3 install --upgrade khoj-assistant
It seems like setuptools-rust
package with an upgraded pip
is required based on this github comment and this stackoverflow answer
from khoj.
thx to repsond.
but there are still some problem here, even i was deployed setuptools-rust
.
here shows some message:
running build_ext running build_rust error: can't find Rust compiler
and
Applications $ python3 -m pip install --upgrade pip
Requirement already satisfied: pip in /opt/homebrew/lib/python3.10/site-packages (22.2.2)
/Applications $ pip3 install setuptools-rust
Requirement already satisfied: setuptools-rust in /opt/homebrew/lib/python3.10/site-packages (1.5.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/homebrew/lib/python3.10/site-packages (from setuptools-rust) (4.3.0)
Requirement already satisfied: setuptools>=62.4 in /opt/homebrew/lib/python3.10/site-packages (from setuptools-rust) (63.4.3)
Requirement already satisfied: semantic-version<3,>=2.8.2 in /opt/homebrew/lib/python3.10/site-packages (from setuptools-rust) (2.10.0)
from khoj.
Hey @yibie I suspect the new issue you're facing is related to huggingface/transformers#2831.
Fix
The easiest way to fix it would be to install rust
with below commands. Let me know if that doesn't resolve it for you.
brew install rustup
rustup-init
source ~/.cargo/env
pip install khoj-assistant
Details
I've seen the issue before while building the tokenizer
package. It's trying to build the package using rust
as it didn't find a binary wheel for your OS/architecture.
- What OS are you on?
- Can you share more of the logs around the rust compiler missing message, i.e
running build_ext running build_rust error: can't find Rust compiler
? It'll tell which package is trying to userust
from khoj.
problem still. i tried twice, and use your commands.
- macOS 12.5.1, python 3.10.6
- here are the complete logs. hopes helpful.
/Applications $ pip3 install khoj-assistant
Collecting khoj-assistant
Using cached khoj_assistant-0.1.8-py3-none-any.whl (433 kB)
Collecting pyyaml==6.0
Using cached PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl (173 kB)
Collecting fastapi==0.77.1
Using cached fastapi-0.77.1-py3-none-any.whl (54 kB)
Collecting sentence-transformers==2.1.0
Using cached sentence_transformers-2.1.0-py3-none-any.whl
Collecting uvicorn==0.17.6
Using cached uvicorn-0.17.6-py3-none-any.whl (53 kB)
Collecting pydantic==1.9.1
Using cached pydantic-1.9.1-cp310-cp310-macosx_11_0_arm64.whl (2.4 MB)
Collecting aiofiles==0.8.0
Using cached aiofiles-0.8.0-py3-none-any.whl (13 kB)
Collecting torch==1.12.1
Using cached torch-1.12.1-cp310-none-macosx_11_0_arm64.whl (49.1 MB)
Collecting pytest==7.1.2
Using cached pytest-7.1.2-py3-none-any.whl (297 kB)
Collecting pillow==9.2.0
Using cached Pillow-9.2.0-cp310-cp310-macosx_11_0_arm64.whl (2.8 MB)
Collecting transformers==4.21.0
Using cached transformers-4.21.0-py3-none-any.whl (4.7 MB)
Collecting huggingface-hub==0.8.1
Using cached huggingface_hub-0.8.1-py3-none-any.whl (101 kB)
Collecting torchvision==0.13.1
Using cached torchvision-0.13.1-cp310-cp310-macosx_11_0_arm64.whl (1.2 MB)
Collecting numpy==1.22.4
Using cached numpy-1.22.4-cp310-cp310-macosx_11_0_arm64.whl (12.8 MB)
Collecting jinja2==3.1.2
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting openai==0.20.0
Using cached openai-0.20.0-py3-none-any.whl
Requirement already satisfied: pyqt6==6.3.1 in /Users/chenyibin/Library/Python/3.10/lib/python/site-packages (from khoj-assistant) (6.3.1)
Collecting dateparser==1.1.1
Using cached dateparser-1.1.1-py2.py3-none-any.whl (288 kB)
Collecting pytz
Using cached pytz-2022.2.1-py2.py3-none-any.whl (500 kB)
Collecting regex!=2019.02.19,!=2021.8.27,<2022.3.15
Using cached regex-2022.3.2-cp310-cp310-macosx_11_0_arm64.whl (281 kB)
Collecting tzlocal
Using cached tzlocal-4.2-py3-none-any.whl (19 kB)
Collecting python-dateutil
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting starlette==0.19.1
Using cached starlette-0.19.1-py3-none-any.whl (63 kB)
Collecting tqdm
Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting filelock
Using cached filelock-3.8.0-py3-none-any.whl (10 kB)
Collecting requests
Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting packaging>=20.9
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/homebrew/lib/python3.10/site-packages (from huggingface-hub==0.8.1->khoj-assistant) (4.3.0)
Collecting MarkupSafe>=2.0
Using cached MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_universal2.whl (17 kB)
Collecting pandas-stubs>=1.1.0.11
Using cached pandas_stubs-1.4.4.220906-py3-none-any.whl (135 kB)
Collecting pandas>=1.2.3
Using cached pandas-1.4.4-cp310-cp310-macosx_11_0_arm64.whl (10.4 MB)
Collecting openpyxl>=3.0.7
Using cached openpyxl-3.0.10-py2.py3-none-any.whl (242 kB)
Requirement already satisfied: PyQt6-Qt6>=6.3.0 in /Users/chenyibin/Library/Python/3.10/lib/python/site-packages (from pyqt6==6.3.1->khoj-assistant) (6.3.1)
Requirement already satisfied: PyQt6-sip<14,>=13.4 in /Users/chenyibin/Library/Python/3.10/lib/python/site-packages (from pyqt6==6.3.1->khoj-assistant) (13.4.0)
Collecting py>=1.8.2
Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting pluggy<2.0,>=0.12
Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting attrs>=19.2.0
Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting tomli>=1.0.0
Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting iniconfig
Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting sentencepiece
Using cached sentencepiece-0.1.97-cp310-cp310-macosx_11_0_arm64.whl (1.1 MB)
Collecting nltk
Using cached nltk-3.7-py3-none-any.whl (1.5 MB)
Collecting scipy
Using cached scipy-1.9.1-cp310-cp310-macosx_12_0_arm64.whl (29.9 MB)
Collecting scikit-learn
Using cached scikit_learn-1.1.2-cp310-cp310-macosx_12_0_arm64.whl (7.7 MB)
Collecting tokenizers>=0.10.3
Using cached tokenizers-0.12.1.tar.gz (220 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting h11>=0.8
Using cached h11-0.13.0-py3-none-any.whl (58 kB)
Collecting asgiref>=3.4.0
Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
Collecting click>=7.0
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting anyio<5,>=3.4.0
Using cached anyio-3.6.1-py3-none-any.whl (80 kB)
Collecting et-xmlfile
Using cached et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting types-pytz>=2022.1.1
Using cached types_pytz-2022.2.1.0-py3-none-any.whl (4.5 kB)
Collecting six>=1.5
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Requirement already satisfied: charset-normalizer<3,>=2 in /Users/chenyibin/Library/Python/3.10/lib/python/site-packages (from requests->huggingface-hub==0.8.1->khoj-assistant) (2.1.1)
Collecting certifi>=2017.4.17
Using cached certifi-2022.6.15.1-py3-none-any.whl (160 kB)
Collecting joblib
Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting pytz-deprecation-shim
Using cached pytz_deprecation_shim-0.1.0.post0-py2.py3-none-any.whl (15 kB)
Collecting sniffio>=1.1
Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting tzdata
Using cached tzdata-2022.2-py2.py3-none-any.whl (336 kB)
Building wheels for collected packages: tokenizers
Building wheel for tokenizers (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for tokenizers (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [53 lines of output]
Error in sitecustomize; set PYTHONVERBOSE for traceback:
AssertionError:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-12-arm64-cpython-310
creating build/lib.macosx-12-arm64-cpython-310/tokenizers
copying py_src/tokenizers/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers
creating build/lib.macosx-12-arm64-cpython-310/tokenizers/models
copying py_src/tokenizers/models/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/models
creating build/lib.macosx-12-arm64-cpython-310/tokenizers/decoders
copying py_src/tokenizers/decoders/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/decoders
creating build/lib.macosx-12-arm64-cpython-310/tokenizers/normalizers
copying py_src/tokenizers/normalizers/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/normalizers
creating build/lib.macosx-12-arm64-cpython-310/tokenizers/pre_tokenizers
copying py_src/tokenizers/pre_tokenizers/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/pre_tokenizers
creating build/lib.macosx-12-arm64-cpython-310/tokenizers/processors
copying py_src/tokenizers/processors/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/processors
creating build/lib.macosx-12-arm64-cpython-310/tokenizers/trainers
copying py_src/tokenizers/trainers/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/trainers
creating build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
copying py_src/tokenizers/implementations/byte_level_bpe.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
copying py_src/tokenizers/implementations/sentencepiece_unigram.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
copying py_src/tokenizers/implementations/sentencepiece_bpe.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
copying py_src/tokenizers/implementations/base_tokenizer.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
copying py_src/tokenizers/implementations/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
copying py_src/tokenizers/implementations/char_level_bpe.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
copying py_src/tokenizers/implementations/bert_wordpiece.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
creating build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
copying py_src/tokenizers/tools/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
copying py_src/tokenizers/tools/visualizer.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
copying py_src/tokenizers/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers
copying py_src/tokenizers/models/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/models
copying py_src/tokenizers/decoders/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/decoders
copying py_src/tokenizers/normalizers/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/normalizers
copying py_src/tokenizers/pre_tokenizers/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/pre_tokenizers
copying py_src/tokenizers/processors/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/processors
copying py_src/tokenizers/trainers/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/trainers
copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
running build_ext
running build_rust
error: can't find Rust compiler
If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
To update pip, run:
pip install --upgrade pip
and then retry package installation.
If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
from khoj.
Yup, that helps! As suspected, the tokenizer
package doesn't seem to have a wheel binary for Mac ARM64 architecture it can directly download. To build the tokenizer
package locally, it needs rust
. And you don't have rust
so it fails.
Setting up rust
(as suggested in my previous comment) should resolve this issue. Let me know if that doesn't fix it.
@yibie Are you sure rust is installed? What does rustc --version
return?
from khoj.
sure, i installed rust.
/Applications % rustc --version
rustc 1.63.0 (4b91a6ea7 2022-08-08)
from khoj.
I see. Interesting that the tokenizer
package build is still failing with same error:
copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
running build_ext
running build_rust
error: can't find Rust compiler
Let me try figure out why it's still failing for you 🤔.
In the meantime, it'd be great if you could try install khoj in a new terminal with:
pip install --no-cache-dir --upgrade khoj-assistant
Just to make sure it's not some stale cache issue
from khoj.
i'm not sure, but it looks succeed installed khoj on my mac.
here are the messages, judge by you. thx again.
Building wheels for collected packages: openai, sentence-transformers, tokenizers
Building wheel for openai (pyproject.toml) ... done
Created wheel for openai: filename=openai-0.20.0-py3-none-any.whl size=54094 sha256=ca0db0a2efe7bf5d750cf0ccdc6fd5af72101c82fa3628b17fd61d40b4a7ee1b
Stored in directory: /private/var/folders/ns/sfzfbcd16d19rky4mtm4gpbm0000gn/T/pip-ephem-wheel-cache-mf1lu864/wheels/42/bd/1d/f1d7e85562515ca1b9504a9450c89623329c0e24af7c749dc6
Building wheel for sentence-transformers (setup.py) ... done
Created wheel for sentence-transformers: filename=sentence_transformers-2.1.0-py3-none-any.whl size=120986 sha256=e19c50682f8de4f742b48a7297db55fad8ca1a3eea13fdbf817699b19946afdf
Stored in directory: /private/var/folders/ns/sfzfbcd16d19rky4mtm4gpbm0000gn/T/pip-ephem-wheel-cache-mf1lu864/wheels/7b/ed/fd/16b8222e673f5eaa48ed71a0a2a6b66767b5e31bea0e5d3895
Building wheel for tokenizers (pyproject.toml) ... done
Created wheel for tokenizers: filename=tokenizers-0.12.1-cp310-cp310-macosx_12_0_arm64.whl size=3554034 sha256=4745a415cd94ea827038a4fb216f5b4fdfd459c09a19b21231f5cc534090ebcc
Stored in directory: /private/var/folders/ns/sfzfbcd16d19rky4mtm4gpbm0000gn/T/pip-ephem-wheel-cache-mf1lu864/wheels/bd/22/bc/fa8337ce1ccf384c8fc4c1dbfa9cb1687934c0f24719082d49
Successfully built openai sentence-transformers tokenizers
Installing collected packages: types-pytz, tokenizers, sentencepiece, pytz, iniconfig, urllib3, tzdata, tqdm, torch, tomli, threadpoolctl, sniffio, six, regex, pyyaml, pyparsing, pydantic, py, pluggy, pillow, pandas-stubs, numpy, MarkupSafe, joblib, idna, h11, filelock, et-xmlfile, click, certifi, attrs, asgiref, aiofiles, uvicorn, scipy, requests, pytz-deprecation-shim, python-dateutil, packaging, openpyxl, nltk, jinja2, anyio, tzlocal, torchvision, starlette, scikit-learn, pytest, pandas, huggingface-hub, transformers, openai, fastapi, dateparser, sentence-transformers, khoj-assistant
Successfully installed MarkupSafe-2.1.1 aiofiles-0.8.0 anyio-3.6.1 asgiref-3.5.2 attrs-22.1.0 certifi-2022.6.15.1 click-8.1.3 dateparser-1.1.1 et-xmlfile-1.1.0 fastapi-0.77.1 filelock-3.8.0 h11-0.13.0 huggingface-hub-0.8.1 idna-3.3 iniconfig-1.1.1 jinja2-3.1.2 joblib-1.1.0 khoj-assistant-0.1.8 nltk-3.7 numpy-1.22.4 openai-0.20.0 openpyxl-3.0.10 packaging-21.3 pandas-1.4.4 pandas-stubs-1.4.4.220906 pillow-9.2.0 pluggy-1.0.0 py-1.11.0 pydantic-1.9.1 pyparsing-3.0.9 pytest-7.1.2 python-dateutil-2.8.2 pytz-2022.2.1 pytz-deprecation-shim-0.1.0.post0 pyyaml-6.0 regex-2022.3.2 requests-2.28.1 scikit-learn-1.1.2 scipy-1.9.1 sentence-transformers-2.1.0 sentencepiece-0.1.97 six-1.16.0 sniffio-1.3.0 starlette-0.19.1 threadpoolctl-3.1.0 tokenizers-0.12.1 tomli-2.0.1 torch-1.12.1 torchvision-0.13.1 tqdm-4.64.1 transformers-4.21.0 types-pytz-2022.2.1.0 tzdata-2022.2 tzlocal-4.2 urllib3-1.26.12 uvicorn-0.17.6
WARNING: There was an error checking the latest version of pip.
from khoj.
Yup, looks like the tokenizer
package installed successfully now! It must have been some stale cache issue that didn't see that rust
was installed. If you run khoj
you should see the configuration GUI pop up
from khoj.
i run khoj
, and see the configuration GUI pop up. done. thx a lot.
and a advice, could khoj search multiple org files?
from khoj.
Related Issues (20)
- [Regression] Obsidian Plugin connectivity check no longer works with local server
- ModuleNotFoundError: No module named 'app'
- Static files not included when installying via the pypi package
- Document recommended host URL for locally running Khoj HOT 1
- Emacs: continuously polling for the server HOT 1
- Add a health check to Khoj HOT 7
- 1000-file limit? HOT 4
- Fix Post Login Redirects for Khoj Pages behind Auth
- Clear up UX confusion related to the desktop setting page HOT 1
- [Bug] Cannot Use Khoj in Obsidian HOT 3
- gpt4all missing on docker image? HOT 4
- File content from obsidian can alter web interface HOT 2
- Using Khoj with the Postgres DB on Remote Server HOT 1
- [Request] Allow for using OpenAI or other 3P to generate embeddings HOT 2
- Index PDF, Markdown and Plaintext files using the Emacs client
- cant start server HOT 2
- docer dont run on my system HOT 1
- django.db.utils.OperationalError HOT 2
- khoj.el: Failed to force update all content index. Status: http 400. Response: {"detail":"There was an error parsing the body"} HOT 5
- Obsidian Khoj Chat does not respond HOT 15
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from khoj.