bigcode-project / bigcode-dataset Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
We need to detect and filter languages on GitHub issues. For example the language detector from fasttext
could be used for this
The first version of The Stack included those weak copyleft licenses and we should exclude them.
We should come up with good filters to get rid of low quality issues. This could be done by going through a sample (e.g. 100) issues and working out patterns of low quality data.
Build a benchmark (for short term) for PII detection of Emails, IP addresses and SSH & API keys:
The ratio of code and docstrings in a document can be used as a proxy for code quality. Code with no comments or comments without any code at all could not be very useful for training and we want to test that hypothesis with some experiments.
In first experiments near-deduplication seemed very effective for bumping up the model's performance. We want to investigate if further, more aggressive deduplication would help even more.
Hi -
I'm from the huggingface community. Can you please approve the users for the dataset on HF?
The cmake subdirectory contains files with the .cmake
extension.
Most of the CMake logic to build projects is contained in files named CMakeLists.txt
, which get placed in the text
directory in this dataset.
In the dedup dataset, there are 186,517 files in cmake
, while there are 469,291 files named CMakeLists.txt
in the text
directory.
The Stack includes Jupyter Notebooks, we want to convert them to text-code pairs after appropriate filtering for each programming language.
An example is the conversion of this Jupter Notebook data to this parsed version in Python.
While we are preparing a large code dataset from GitHub we also want to include other datasets that might be useful for pretraining or fine-tuning.
Note: For evaluation there is a separate list here.
There are already a few datasets gathered here:
https://docs.google.com/spreadsheets/d/1otSgs2-iDp1KTbasEDlFhTHiaKsYbIRvWrgH_TiubFQ/edit#gid=0
We would like to gather an extensive lists of resources that could be useful for training language models for code. You can contribute by commenting on this issue with a missing dataset with the template below. A resource could be a specific code dataset (e.g. CodeXGlue), a related dataset (e.g. CS part of Wikipedia), or a website (e.g. Stackoverflow), or something similar.
The following is the template for reporting an interesting data source:
Name | Link | License | Number of files/samples | Size in GB | Languages | Available on the HF Hub |
---|---|---|---|---|---|---|
Transformers repo | https://github.com/huggingface/transformers | Apache 2.0 | 100 | 0.1 | Python | No |
Here's the Markdown snippet that you can copy/paste:
|Name|Link|License|Number of files/samples| Size in GB| Languages |Available on the HF Hub|
|:-|:-|:-|:-|:-|:-|:-|
| | | | | | | |
Git commits pose an interesting data source as they could be used as a natural source for prompts for code models with a format like:
CODE BEFORE
COMMIT MESSAGE
CODE AFTER
The dataset requires some cleaning and filtering since:
Wrap PII pipeline (detection + redaction) in a library, to easily detect PII in text or HF datasets in an efficient way.
In order to make sure that the evaluation results reflect the true performance of the model, it is important to make sure that the evaluation benchmarks are not part of the training data. For that purpose we want to use #15 to search and remove evaluation benchmarks from the code dataset.
StackOverflow contains a lot of interesting data that can be used for classification, language modeling or retrieval. This is an issue to gather all ideas and sub-issues to add datasets to the Hub.
It would be great if we could do some exploratory data analysis on the full dataset:
To filter out low-quality code one can use the char/token ratio of a tokenizer to e.g. identify files with hashes or the train a small LM (e.g. KenLM models) on a clean dataset and use its perplexity to remove noisy data.
I recently downloaded The Stack (the-stack-dedup) from Huggingface via GIT LFS. I have two questions that I need help with:
The size on disk of the dedup datset is only around 900GB (much smaller than the 1.5TB indicated on the data card - https://huggingface.co/datasets/bigcode/admin/resolve/main/the-stack-infographic-v11.png)
Is there somewhere were the file counts are listed in full for each dataset by language (dedup and full)?
Essentially I am looking to make sure that I have accessed the entirety of the dataset, so I either need to understand the dataset size difference, or know how many files there should be for each language so I can validate my download. Ideally both.
Thanks in advance!
The next iteration of PII, we want yo train an NER model to detect PII on the benchmark we will annotate. Here are some tasks:
1 - Train an encoder (Bert) model on languages of The Stack (same languages as the target model) => @loubnabnl
2 - Prepare pipeline for fine-tuning the encoder model on NER PII task => @mrm8488
Help wanted:
3 - Estimate cost of using a Bert sized model to detect PII on The Stack (e.g. estimate inference with CodeBert-NER on the-stack-smol (300k samples) on 1 GPU)
4- Implement PII redaction using special tokens in the tokenizer
5- Analyze PII in a dataset of Github issues, and investigate using our regexes + standard english NER models to remove it
For any intermediate code/analysis please open PRs in bigcode-analysis repo.
Benchmarks like HumanEval
could by now be part of GitHub repositories and thus part of the the pretraining. We could search and delete those before training.
We should come up with good filters to get rid of low quality commits. This could be done by going through a sample (e.g. 100) issues and working out patterns of low quality data.
For data filtering or model conditioning we would like to have GH metadata such as stars, forks, contributors etc. as part of the dataset.
洗干净的数据 下载链接?
Building a dataset search index is necessary for two reasons:
The current idea is to use pyserini
to build the index of the near-deduplicated dataset.
Hi,
I now need to do PII detection and hope to use the already trained NER model.
In infer.slurm, I found the project attempt to load model from bigcode/bigcode-encoder-pii-ner-v2.
However, when I want to do it by myself, it shows
OSError: bigcode/bigcode-encoder-pii-ner-v2 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with huggingface-cli login
or by passing token=<your_token>
I cannot find bigcode/bigcode-encoder-pii-ner-v2 in hugging face.
So, may I ask how can I get a trained pii detection model?
Try converting Jupyter Notebooks in The Stack to scripts for different programming languages with some filtering (as data could be included in for the pre-training)
Based on a recent tweet, it sounds like issues and commits will be added to the dataset soon (which is great news!).
Any chance on adding code reviews, as well? Or is this limited by the capabilities for the GitHub API?
Test and update regexes for detecting the following entities in The Stack
Before training the first model we can filter out any opt-out repository from the dataset.
Hi,
I'm wondering how to get the issues from the gh archive.
It seems the issues are from the gh archive.
Could you provide scripts that do the transformation?
Thank you!
While we are preparing a large code dataset from GitHub we also want to include other datasets that might be useful for pretraining or fine-tuning.
Note: For evaluation there is a separate list here.
There are already a few datasets gathered here:
https://docs.google.com/spreadsheets/d/1otSgs2-iDp1KTbasEDlFhTHiaKsYbIRvWrgH_TiubFQ/edit#gid=0
We would like to gather an extensive lists of resources that could be useful for training language models for code. You can contribute by commenting on this issue with a missing dataset with the template below. A resource could be a specific code dataset (e.g. CodeXGlue), a related dataset (e.g. CS part of Wikipedia), or a website (e.g. Stackoverflow), or something similar.
The following is the template for reporting an interesting data source:
Name | Link | License | Number of files/samples | Size in GB | Languages | Available on the HF Hub |
---|---|---|---|---|---|---|
Transformers repo | https://github.com/huggingface/transformers | Apache 2.0 | 100 | 0.1 | Python | No |
Here's the Markdown snippet that you can copy/paste:
|Name|Link|License|Number of files/samples| Size in GB| Languages |Available on the HF Hub|
|:-|:-|:-|:-|:-|:-|:-|
| | | | | | | |
Which programming languages should be included in the pretraining dataset?
In contrast to natural language does multilinguality not increase the vocab size significantly so I think it could be interesting to be on the liberal side here to open the doors for interesting downstream use-cases.
I noticed ngrams
function returns empty list when number of tokens is smaller than ngram_size value.
This has the side effect of clustering all sentences with number of tokens smaller than the ngram_size value together & removing them as dupes. Not sure if this is intentional?
so for the default of ngram_size=5 "a sentence like this" gets removed by default.
Could maybe filter & rerun the deduping with a smaller n_gram value for those sentences.
Most code models following Codex used some content based filtering (e.g. max/mean line length of file, fraction of alphanumeric characters). These work well for Python but can fail for other langugages as HTML. We need to gather good filters for each language, especially the high volume languages such as HTML, Java, JavaScript, Python, C, C++ etc.
If I need to remove parts of the file other than the code, is there any other way besides regular matching? For example with the codebert model?
programming-languages-to-file-extensions.json correctly has the most common rkt file extension of 'rkt' for Racket, but the data subset (for Racket) at https://huggingface.co/datasets/bigcode/the-stack/tree/main/data/racket has zero instances of files with this extension, and rkt is mentioned specifically as being an excluded extension in the paper at https://arxiv.org/abs/2305.06161 This would likely exclude the majority of actual racket files found on github.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.