Comments (2)
This is due to an imported package pinning the processor affinity, which Ray then inherits when spawning processes.
One solution would be to use psutil to clear the processor affinity on the main thread after import but before spawning Ray threads.
Another would be to assign each worker its own processor affinity after worker creation.
The former would leave you hopping cores, while the latter is not easily solved in the general case of platforms with unknown physical-vs-virtual core counts. The solution I eventually used was to assign each worker a job that assigned it specific cores, based on deployment details.
from aphrodite-engine.
Fixed with #187
We now properly set the affinities at launch.
from aphrodite-engine.
Related Issues (20)
- [Bug]: gguf loading failed. config.json? HOT 4
- [Feature]: Support hqq quantize method.
- [Bug]: Mixtral-8x22b-instruct not running with AWQ HOT 10
- [Feature]: Provide configuration via env vars or a configuration file
- [Usage]: odd use of GPUS number and tensor parallelism HOT 2
- [Installation]: Cannot install the library
- [Bug]: Unable to use OpenAI API with an auth key via a web browser due to OPTIONS preflight request returning 401. HOT 1
- [Bug]: HOT 1
- [Usage]: Please provide the environment variable that closes the KoboldAI Lite page.
- [Performance]: Memory Usage Fix for gguf. HOT 3
- [Installation]: ValueError: 17 is not a valid GGMLQuantizationType HOT 21
- [Installation]: Upload Aphrodite v0.5.2 On Pypi.org HOT 3
- [Usage]: What to set to get acceptable performance on Pascal GPUs? (Non-P100) HOT 2
- [Installation]: Installing from source does not work. undefined symbol: _ZN3c104cuda14ExchangeDeviceEa HOT 8
- [Bug]: PermissionError: [Errno 13] Permission denied: '/app/aphrodite-engine/.triton' HOT 3
- [Bug]: LoRA broken when TP>1
- [Bug]: LoRA fails to load HOT 1
- [Feature]: Exllamav2 Q4 cache HOT 2
- [Usage]: Lora Adapter Parameter while inferencing HOT 1
- [Bug]: Flash attention cannot be used on v0.5.3 HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aphrodite-engine.