Comments (3)
chebbyChefNEQ
commented on June 14, 2024
While I don't have a solution for the linearly increasing timing right now. I have added async loading in #1687. Maybe that is a sufficient work around for now?
Note that you need to call this in your script
multiprocessing.set_start_method("spawn")
from lance.
chebbyChefNEQ
commented on June 14, 2024
See example of using AsyncDataset here https://github.com/lancedb/lance/pull/1687/files#diff-99b8dda3b022577fd8a0d1bbef0425ac0f21c7ea65fe73adfb86d3330922b331L168-R180
from lance.
stabilize-ai
commented on June 14, 2024
Thanks - I also notice that this only happens in the 1st loop of KMeans, the subsequent loops are fast
from lance.
Related Issues (20)
- Support PutIfNotExist commit handler for R2 and Minio
- Lift the max object size for non-R2 stores HOT 1
- implement existing V2 indices as `VectorIndexExtension`
- flaky test: `python/tests/test_optimize.py::test_compact_with_write`
- Add "blobs" to the table format
- chore: setup maven repo for java jni core SDK
- Initial support for writing blob files
- Support for blob compaction (and the needed remapping)
- Move Lance to use preview releases
- Support for blob file cleanup
- Add when_matched_delete to merge_insert
- Handle GCS rate limit on manifest write contention
- Enable S3 Anonymous Mode HOT 3
- Lance / Burn sub-dependency conflict on `half` version HOT 1
- Propagate `storage_options` and other read parameters when pickling `LanceFragment` HOT 1
- Make an internal API warning macro HOT 1
- bug: round trip `FixedSizeList` lance <-> arrow. nullability is not preserved HOT 2
- Epic: Stable Row Ids
- Create the row id index data structure
- Support more encodings in format v2 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lance.