Comments (5)
There are two parts, "ingesting" and "inserting". The ingesting is basically obsolete with the updated optimizations and can be removed. The inserting still takes some time but there's no notion of a progress bar we could acquire. That being said, we should remove the progress bar all together
from fiftyone.
For the types of datasets that we currently work with, parsing the samples (ie., creating fo.Sample
instances) is pretty quick (thanks to Tyler's recent work). BUT, if the user provides a complicated SampleParser
, it may not be so quick to parse the samples. So, a ProgressBar may still be warranted.
The saving part is currently done in a single batch, regardless of dataset size, so we don't have a hook into the process to display a progress bar. However, it may be a better practice to save the samples in batches, both to avoid DB transactions that take a second or longer, and to enable a progress bar to track the progress on the saving.
In fact, it would be preferable to interleave parsing and saving (calls to add_samples()
) to prevent the need for storing ALL of the fo.Sample
s in memory. We're designing the system in such a way that the user can work with any number of samples. So, this would be a required change anyway.
from fiftyone.
Until we have a chance to address the underlying implementation, I removed the progress bar as requested. It's the best option for the current implementation.
from fiftyone.
Updated the issue title to reflect the underlying spirit of this issue now that the existing progress bar has been removed
from fiftyone.
This is completely resolved by #239. There is now a single progress bar that tracks parsing + saving to the DB together as a single operation
from fiftyone.
Related Issues (20)
- [BUG] ConnectionResetError while processing big dataset HOT 1
- [FR] Support setting tags in custom importer class HOT 1
- [BUG] Fiftyone fails to create the metadata for large images HOT 1
- [BUG] upload dataset with 'keypoints' field to labelstudio correctly
- [BUG] CVAT fouc.import_annotations not importing video annotations
- No label colors not showing in Embeddings panel[BUG] HOT 1
- [BUG] Mongoengine: using pop() to update dict fields is dangerous and should be avoided HOT 1
- [BUG] GraphQL API Error name '_HAS_DEFAULT_FACTORY' is not defined HOT 2
- Wrong DateField value loaded from MongoDB HOT 3
- [BUG] operator set_progress error HOT 1
- fo.core.video.make_frames_dataset is sneakily considered a frame view HOT 9
- [FR] support ultralytics yolov8 models pretrained on open images v7 dataset in the model zoo
- [BUG] lancedb backend doesnt work in exfat disk HOT 1
- [BUG] Memory leak when using `add_coco_labels` for instance segmentation with coco_id_field set HOT 3
- [BUG]The bbox pixel coordinate is less than 0, causing the mask to not be displayed.
- [BUG] Errors while uploading large amounts of files to a label-studio
- How to rename the dataset to groundtruth using the frontend [BUG] HOT 1
- [INSTALL] fiftyone.core.config.FiftyOneConfigError: MongoDB could not be installed on your system. Please define a `database_uri` in your `fiftyone.core.config.FiftyOneConfig` to connect to yourown MongoDB instance or cluster HOT 1
- [BUG] SuperGradients YOLO-NAS inference not working HOT 3
- [FR] Using fiftyone for multitask classifier
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fiftyone.