seonwkim / dataverse Goto Github PK

Dataverse is a freely-accessible open-source project that supports your ETL(Extract, Transform and Load) pipeline with Python. We offer a simple, standardized and user-friendly solution for data processing and management, catering to the needs of data scientists, analysts, and developers in LLM era. Even though you don't know much about Spark, you can use it easily via dataverse.

• Integrated library: Dataverse streamline your workflow by integrating multiple preprocessing libraries into one, eliminating the hassle of settings and searching for the right tools. You can even use HuggingFace datasets from hub directly into the pipeline.
• Simplified Spark usage: You don't have to be a pro with Spark. With just a few setting configurations, you can easily take Spark's high performance effortlessly.
• Facilitated collaboration: Offer uniform preprocessing codes to ensure consistent results whether who runs the code. Dataverse also enable collaboration among users with varying levels of Spark proficiency.

• Block-Based: Dataverse lets you build Spark code like putting together puzzle pieces. You can easily add, take away, or rearrange pieces to get the results you want.
• Configure-Based: Dataverse has a user-friendly setup where you don't need to know all the code. Just set up the options, and you're good to go.
• Extensible: It's designed to meet your specific demands, allowing for custom features that fit perfectly with your project.

To use this library, the following conditions are needed:

• Python (version between 3.10 and 3.11)
• JDK (version 11)
• PySpark Detail installation guide for prerequisites can be found on here.

pip install dataverse

Various and more detailed tutorials are here.

• add_new_etl_process.ipynb : If you want to use your custom function, you have to register the function on Dataverse. This will guide you from register to apply it on pipeline.
• test_etl_process.ipynb : When you want to get test(sample) data to quickly test your ETL process, or need data from a certain point to test your ETL process.
• scaleout_with_EMR.ipynb : For people who want to run their pipeline on EMR cluster.

# 1. Set your ETL process as config.

from omegaconf import OmegaConf

ETL_config = OmegaConf.create({
    'spark': {
        'appname': 'ETL',
        'driver': {'memory': '4g'},
    },
    'etl': [
        { 
            'name': 'data_ingestion___huggingface___hf2raw', # Extract; You can use HuggingFace datset from hub directly!
            'args': {'name_or_path': ['ai2_arc', 'ARC-Challenge']}
        },
        {
            'name': 'utils___sampling___random',
            'args': {'sample_n_or_frac': 0.5}
        },
        {
            'name': 'deduplication___minhash___lsh_jaccard', # Transform
            'args': {'threshold': 0.1,
                    'ngram_size': 5,
                    'subset': 'question'}
        },
        {
          'name': 'data_load___parquet___ufl2parquet', # Load
          'args': {'save_path': './guideline/etl/sample/quickstart.parquet'}
        }
      ]
  })

# 2. Run ETLpipeline.

from dataverse.etl import ETLPipeline

etl_pipeline = ETLPipeline()
spark, dataset = etl_pipeline.run(config=ETL_config, verbose=True)

# 3. Result file is saved on the save_path

Dataverse is an open-source project orchestrated by the Data-Centric LLM Team at Upstage, designed as an data ecosystem for LLM(Large Language Model). Launched in March 2024, this initiative stands at the forefront of advancing data handling in the realm of LLM.

Dataverse is completely freely-accessible open-source and licensed under the Apache-2.0 license.

If you want to cite our 🌌 Dataverse project, feel free to use the following bibtex

@misc{dataverse,
  title = {Dataverse},
  author = {Hyunbyung Park, Sukyung Lee, Gyoungjin Gim, Yungi Kim, Seonghoon Yang, Jihoo Kim, Changbae Ahn, Chanjun Park},
  year = {2024},
  publisher = {GitHub, Upstage AI},
  howpublished = {\url{https://github.com/UpstageAI/dataverse}},
}