giddyyupp / coco-minitrain Goto Github PK
View Code? Open in Web Editor NEWa subset of coco dataset for faster experimentation
a subset of coco dataset for faster experimentation
Hi,
how are we setting the path for this database?
I mean where is the instances_train2017.json with all the 25k images. My model requires instance_train2017.json file path and train2017 image path to start the training. I downloaded this dataset and can see only 25k images not the json file. How do we get that?
Thanks
It should be 5000.
Hey there,
I noticed that the category list doesn't cover all the numbers from 1 to 80 when I recently checked using pycocotools.coco
. Here's the list of categories that I retrieved:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 67, 70, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 84, 85, 86, 87, 88, 89, 90]
Is it normal?
Hi @giddyyupp ,
what should I modify to obtain a 10% dataset of COCO? ๐
Thanks a lot!
Thank you for your great work, after clicking the link I get "8026 This link has been deleted by the owner" error.
Thanks
Hi,
Thanks for the contribution. When running the csv_to_coco_json.py, the 'coco_class_labels.csv' is missing.
FileNotFoundError: [Errno 2] No such file or directory: 'coco_class_labels.csv''
Could pls add this file? Thanks!
One clarification question: is the link of
instances_minitrain2017.json you shared is the 25k training samples that has been sampled by sample_coco.py
for 10M
times? Then I don't need to run it for 25M
times #17.
So by using this instances_minitrain2017.json
, we can essentially download these 25k
training images as the same as yours by
python3 coco_download.py --annotation <path_to_instances_minitrain2017.json> --output train2017_mini_25k
?
Thanks in advance @giddyyupp
Hi,
Thank you for the paper and dataset, it is really useful for running ablations faster.
I wanted to ask if the segmentations provided in the JSON files are changed in any way from the originaltrain2017
JSON.
Also, do you have any metrics for segmentation?
Thanks a lot.
Hi...
How to use this coco-minitrain?
I want to get APsmall, APmedium, APlarge?
I confuse no example code.
Thank's alot
i use sample_coco.py to sample my own mini-coco dataset for my experiment, it helps me save lots of time. Thanks for your work!
Starred and wanted to Starred more than 1k stars
Is there any reference data on the average precision (AP) achieved by various networks in instance segmentation tasks?
In annotation json files,
There is image 000000449108.jpg
but no where in image 000000449108.jpg
Could you give the download link for minitrain images ?
Thank you for your excellent project!
I'm training the keypoint RCNN network implemented in torchvision by scaling the image to the size denoted in object detection result tables.
I know the size of input image between simplebaseline2D and networks of RCNN family(faster, mask, etc.) is different.
I'd like to know how to scale the input image size for minicoco dataset in the keypoint detection experiment using a simplebaseline2D network.
Despite using different networks, is the same size between the object detection task and the keypoint detection task?
Thanks in advance!
Example
img_id : 000000007278
have label as [0 ,0.391727 ,0.221691 ,0.190391 ,0.196867]
as far as i can understand 0
stands for class 2nd and 3rd element represent x_min
and y_min
respectively and 4th and 5th are width and height
hello there! I am currently attempting to train COCO minitrain on the CenterNet-better algorithm. However, as shown in the image below, I am required to have the person_keypoints_val_2017.json AND person_keypoints_train_2017.json file. I am only able to find the person_keypoints_train2017.json file but not the val one. May I ask where would it be? Thank you!
Hello, are the models mentioned on the github page trained with the exact same methods, params, epochs,... to their original dataset? The only difference is the training set itself right? (from "train2017" to "minitrain")
I'm asking this because I'm currently training Deformable DETR on "minitrain" for my university final project, 26 epochs in (maximum is 50 epochs) and all the APs are still at 0. The model seems to peform quite well on the original dataset (I even checked with "val2017"). I kept all the config the same to the original model (I should mention that I divided "minitrain" into 5 folders, 5000 images each since google colab encounter error with drive if there are too many files in 1 folder, I think this shouldn't affect the result).
So did you guys encounter this issue as well in your training? Is this normal and I should wait until 50 epochs, or is there something that I'm supposed to change that I'm not aware of? ("minitrain" and annotations are from the shared file and val set is just "val2017")
Can you release the code which can generate the coco-minitrain set? I want to test some models on different proportions of COCO, but coco-minitrain only contain 20% images of COCO
็ฑไบ็ฝ็ป้ๅถ๏ผไธ่ฝไปcocoๅฎ็ฝๆฝๆ ทไธ่ฝฝ๏ผๆไปฅ
ๆๆดๆนไบไธไปฃ็ ๏ผๅฏไปฅ็ดๆฅไป ๅฎๆด็trainๆฐๆฎไธญ๏ผ ๆ นๆฎไฝ่
ๆๅฆ็instances_minitrain2017.json ๅฐๅพ็ๆฝๅๅบๆฅ
import os
from pycocotools.coco import COCO
import wget
import concurrent.futures
import argparse
import pathlib
from shutil import copy
parser = argparse.ArgumentParser(description="Download COCO images")
parser.add_argument(
"--annotation",
type=str,
default="",
help="Json file containing annotations",
)
parser.add_argument(
"--output_dir", type=str, default="", help="Output file to save images to"
)
parser.add_argument(
"--start_dir", type=str, default="", help="all train data(20G) dir"
)
args = parser.parse_args()
annotation = args.annotation
start_dir= args.start_dir
root = pathlib.Path().absolute()
ann_file = root / annotation
assert pathlib.Path(args.output_dir).is_dir(), "not valid dir"
if not os.fsdecode(ann_file).endswith(".json"):
assert "Only support COCO style JSON file"
try:
coco = COCO(os.fsdecode(ann_file))
img_ids = list(coco.imgs.keys())
except FileNotFoundError:
raise
def download_images(id):
try:
filename = "{0:0>12d}".format(id)
filename = filename + ".jpg"
source = f"{start_dir}/{filename}"
# wget.download(full_url, out=args.output_dir)
copy(source,args.output_dir)
except Exception as e:
print(f"The download exception is {e}", flush=True)
with concurrent.futures.ThreadPoolExecutor() as executor:
executor.map(download_images, img_ids)
Hi
Thanks for curating this. I can see the download link is not working. Can you please update the link if possible?
Thanks
Traceback (most recent call last):
File "sample_coco.py", line 156, in <module>
main()
File "sample_coco.py", line 62, in main
for i in keys[:parser.sample_image_count]:
TypeError: slice indices must be integers or None or have an __index__ method
when running sample_coco.py
script.
The default value of run_count
is 10M
and it takes a long time for me to sample. Does it only affect to what extent the stats/distribution of sampled train dataset matches the original train2017
? If I set run_count
to be 10
, should I assume that the stats/distribution would not be affected too much? Thanks!
In Detection & Segmentation,
for example, provided instances_minitrain2017.json "000000527649.jpg" is exist
but not for train2017/ which i downloaded at huggingface
coco_minitrain_25k.zip [Huggingface] (maintained by @bryanbocao)
Hi, thanks for your contribution.
When i run sample_coco.py, the following lines appear:
loading annotations into memory...
Done (t=14.37s)
creating index...
index created!
Then it's stuck here all the time without generating the final json file.
Could you help me? @giddyyupp @nerminsamet
There is a "/" missing at the end of line 37 in the scripts coco_download.py. I could not initiate download without it.
while running the sample_coco.py there's an error "cannot import name 'read_data_sets' from 'dataloader' "
Hi folks,
Thank you for curating the useful subset. It seems that the google drive link on your github readme is broken, could you please look into it?
Thanks
When starting sample_run.py
as python sample_coco.py --coco_path "coco/" --save_format "json" --sample_image_count 1000 --run_count 100
, every run is a failed run.
Did I get something wrong or?
If I add print('failed_run)
inside failed run check, this is the output:
loading annotations into memory...
Done (t=12.92s)
creating index...
index created!
100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 100/100 [00:06<00:00, 15.12it/s]
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
{"info": {"description": "COCO 2017 Dataset", "url": "http://cocodataset.org", "version": "1.0", "year": 2017, "contributor": "COCO Consortium", "date_created": "2017/09/01"}, "licenses": [{"url": "http://creativecommons.org/licenses/by-nc-sa/2.0/", "id": 1, "name": "Attribution-NonCommercial-ShareAlike License"}, {"url": "http://creativecommons.org/licenses/by-nc/2.0/", "id": 2, "name": "Attribution-NonCommercial License"}, {"url": "http://creativecommons.org/licenses/by-nc-nd/2.0/", "id": 3, "name": "Attribution-NonCommercial-NoDerivs License"}, {"url": "http://creativecommons.org/licenses/by/2.0/", "id": 4, "name": "Attribution License"}, {"url": "http://creativecommons.org/licenses/by-sa/2.0/", "id": 5, "name": "Attribution-ShareAlike License"}, {"url": "http://creativecommons.org/licenses/by-nd/2.0/", "id": 6, "name": "Attribution-NoDerivs License"}, {"url": "http://flickr.com/commons/usage/", "id": 7, "name": "No known copyright restrictions"}, {"url": "http://www.usa.gov/copyright.shtml", "id": 8, "name": "United States Government Work"}], "categories": [{"supercategory": "person", "id": 1, "name": "person"}, {"supercategory": "vehicle", "id": 2, "name": "bicycle"}, {"supercategory": "vehicle", "id": 3, "name": "car"}, {"supercategory": "vehicle", "id": 4, "name": "motorcycle"}, {"supercategory": "vehicle", "id": 5, "name": "airplane"}, {"supercategory": "vehicle", "id": 6, "name": "bus"}, {"supercategory": "vehicle", "id": 7, "name": "train"}, {"supercategory": "vehicle", "id": 8, "name": "truck"}, {"supercategory": "vehicle", "id": 9, "name": "boat"}, {"supercategory": "outdoor", "id": 10, "name": "traffic light"}, {"supercategory": "outdoor", "id": 11, "name": "fire hydrant"}, {"supercategory": "outdoor", "id": 13, "name": "stop sign"}, {"supercategory": "outdoor", "id": 14, "name": "parking meter"}, {"supercategory": "outdoor", "id": 15, "name": "bench"}, {"supercategory": "animal", "id": 16, "name": "bird"}, {"supercategory": "animal", "id": 17, "name": "cat"}, {"supercategory": "animal", "id": 18, "name": "dog"}, {"supercategory": "animal", "id": 19, "name": "horse"}, {"supercategory": "animal", "id": 20, "name": "sheep"}, {"supercategory": "animal", "id": 21, "name": "cow"}, {"supercategory": "animal", "id": 22, "name": "elephant"}, {"supercategory": "animal", "id": 23, "name": "bear"}, {"supercategory": "animal", "id": 24, "name": "zebra"}, {"supercategory": "animal", "id": 25, "name": "giraffe"}, {"supercategory": "accessory", "id": 27, "name": "backpack"}, {"supercategory": "accessory", "id": 28, "name": "umbrella"}, {"supercategory": "accessory", "id": 31, "name": "handbag"}, {"supercategory": "accessory", "id": 32, "name": "tie"}, {"supercategory": "accessory", "id": 33, "name": "suitcase"}, {"supercategory": "sports", "id": 34, "name": "frisbee"}, {"supercategory": "sports", "id": 35, "name": "skis"}, {"supercategory": "sports", "id": 36, "name": "snowboard"}, {"supercategory": "sports", "id": 37, "name": "sports ball"}, {"supercategory": "sports", "id": 38, "name": "kite"}, {"supercategory": "sports", "id": 39, "name": "baseball bat"}, {"supercategory": "sports", "id": 40, "name": "baseball glove"}, {"supercategory": "sports", "id": 41, "name": "skateboard"}, {"supercategory": "sports", "id": 42, "name": "surfboard"}, {"supercategory": "sports", "id": 43, "name": "tennis racket"}, {"supercategory": "kitchen", "id": 44, "name": "bottle"}, {"supercategory": "kitchen", "id": 46, "name": "wine glass"}, {"supercategory": "kitchen", "id": 47, "name": "cup"}, {"supercategory": "kitchen", "id": 48, "name": "fork"}, {"supercategory": "kitchen", "id": 49, "name": "knife"}, {"supercategory": "kitchen", "id": 50, "name": "spoon"}, {"supercategory": "kitchen", "id": 51, "name": "bowl"}, {"supercategory": "food", "id": 52, "name": "banana"}, {"supercategory": "food", "id": 53, "name": "apple"}, {"supercategory": "food", "id": 54, "name": "sandwich"}, {"supercategory": "food", "id": 55, "name": "orange"}, {"supercategory": "food", "id": 56, "name": "broccoli"}, {"supercategory": "food", "id": 57, "name": "carrot"}, {"supercategory": "food", "id": 58, "name": "hot dog"}, {"supercategory": "food", "id": 59, "name": "pizza"}, {"supercategory": "food", "id": 60, "name": "donut"}, {"supercategory": "food", "id": 61, "name": "cake"}, {"supercategory": "furniture", "id": 62, "name": "chair"}, {"supercategory": "furniture", "id": 63, "name": "couch"}, {"supercategory": "furniture", "id": 64, "name": "potted plant"}, {"supercategory": "furniture", "id": 65, "name": "bed"}, {"supercategory": "furniture", "id": 67, "name": "dining table"}, {"supercategory": "furniture", "id": 70, "name": "toilet"}, {"supercategory": "electronic", "id": 72, "name": "tv"}, {"supercategory": "electronic", "id": 73, "name": "laptop"}, {"supercategory": "electronic", "id": 74, "name": "mouse"}, {"supercategory": "electronic", "id": 75, "name": "remote"}, {"supercategory": "electronic", "id": 76, "name": "keyboard"}, {"supercategory": "electronic", "id": 77, "name": "cell phone"}, {"supercategory": "appliance", "id": 78, "name": "microwave"}, {"supercategory": "appliance", "id": 79, "name": "oven"}, {"supercategory": "appliance", "id": 80, "name": "toaster"}, {"supercategory": "appliance", "id": 81, "name": "sink"}, {"supercategory": "appliance", "id": 82, "name": "refrigerator"}, {"supercategory": "indoor", "id": 84, "name": "book"}, {"supercategory": "indoor", "id": 85, "name": "clock"}, {"supercategory": "indoor", "id": 86, "name": "vase"}, {"supercategory": "indoor", "id": 87, "name": "scissors"}, {"supercategory": "indoor", "id": 88, "name": "teddy bear"}, {"supercategory": "indoor", "id": 89, "name": "hair drier"}, {"supercategory": "indoor", "id": 90, "name": "toothbrush"}], "images": [], "annotations": []}
Hey!
Amazing dataset!
I was hoping to get a small data set for my pose detection model but I couldnt find keypoints in the annotations file. Are the keypoints not included in this dataset?
Thanks!
Hi. Thank you for the great project!
I'm curious about the training settings for each model epoch, learning rate, etc.
Can I get information about each model? (Faster RCNN, RetinaNet, etc)
I want to benchmark my model via coco-minitrain. As you know, training whole coco data is too slow to develop models.
So I decided to use coco-minitrain.
And you submitted your paper to ECCV with this coco-minitrain as a minor contribution.
But I downloaded your json file and downloaded via coco_download.py.
But this does NOT match at all.
Is there something I did wrong???????
I think not.
Simply to say, Could you share your json and image files when you used for making the benchmark table?
Error when running
python3 coco_download.py --annotation train2017_mini_25k.json --output train2017_mini_25k
Traceback (most recent call last):
File "coco_download.py", line 23, in <module>
assert pathlib.Path(args.output_dir).is_dir(), "not valid dir"
AssertionError: not valid dir
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.