lhoestq Goto Github PK

beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

dask

Parallel computing with task scheduling

datasets

🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools

datasets-tagging

A Streamlit app to add structured tags to the datasets

datasets_readme_generator

Automatically create READMEs for datasets

dedloc

Official code for "Distributed Deep Learning in Open Collaborations"

img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

libsndfile-binaries

Pre-compiled shared libraries for libsndfile

lm-evaluation-harness

A framework for few-shot evaluation of language models.

notebooks

Some notebooks for NLP

pytorch-image-models

PyTorch image models, scripts, pretrained weights -- ResNet, ResNeXT, EfficientNet, EfficientNetV2, NFNet, Vision Transformer, MixNet, MobileNet-V3/V2, RegNet, DPN, CSPNet, and more

pytorch-lightning

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.

super-dataloading-benchmarks

transformers

🤗Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.

webdataset

A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.