🚀 Synthetic Data Generator

The Synthetic Data Generator (SDG) is a specialized framework designed to generate high-quality structured tabular data. It incorporates a wide range of single-table, multi-table data synthesis algorithms and LLM-based synthetic data generation models.

Synthetic data, generated by machines using real data, metadata, and algorithms, does not contain any sensitive information, yet it retains the essential characteristics of the original data. There is no direct correlation between synthetic data and real data, making it exempt from privacy regulations such as GDPR and ADPPA. This eliminates the risk of privacy breaches in practical applications.

High-quality synthetic data can be safely utilized across various domains including data sharing, model training and debugging, system development and testing, etc. Read the latest API docs for more details!

🔧 Features

Technological advancements
- Supports a wide range of statistical data synthesis algorithms, LLM-based synthetic data generation model is also integrated;
- Optimised for big data scenarios, effectively reducing memory consumption;
- Continuously tracking the latest advances in academia and industry, and introducing support for excellent algorithms and models in a timely manner.
Privacy enhancements
- SDG supports differential privacy, anonymization and other methods to enhance the security of synthetic data.
Easy to extend
- Supports expansion of models, data processing, data connectors, etc. in the form of plug-in packages

🎉 LLM-integrated synthetic data generation

For a long time, LLM has been used to understand and generate various types of data. In fact, LLM also has certain capabilities in tabular data generation. Also, it has some abilities that cannot be achieved by traditional (based on GAN methods or statistical methods) .

Our sdgx.models.LLM.single_table.gpt.SingleTableGPTModel implements two new features:

Synthetic data generation without Data

No training data is required, synthetic data can be generated based on metadata data.

Off-Table feature inference

Infer new column data based on the existing data in the table and the knowledge mastered by LLM.

🔛 Quick Start

Pre-build image

You can use pre-built images to quickly experience the latest features.

docker pull idsteam/sdgx:latest

Install from PyPi

pip install sdgx

Local Install (Recommended)

Use SDG by installing it through the source code.

git clone [email protected]:hitsz-ids/synthetic-data-generator.git
pip install .
# Or install from git
pip install git+https://github.com/hitsz-ids/synthetic-data-generator.git

Quick Demo of Single Table Data Generation and Metric

Demo code

from sdgx.data_connectors.csv_connector import CsvConnector
from sdgx.models.ml.single_table.ctgan import CTGANSynthesizerModel
from sdgx.synthesizer import Synthesizer
from sdgx.utils import download_demo_data

# This will download demo data to ./dataset
dataset_csv = download_demo_data()

# Create data connector for csv file
data_connector = CsvConnector(path=dataset_csv)

# Initialize synthesizer, use CTGAN model
synthesizer = Synthesizer(
    model=CTGANSynthesizerModel(epochs=1),  # For quick demo
    data_connector=data_connector,
)

# Fit the model
synthesizer.fit()

# Sample
sampled_data = synthesizer.sample(1000)
print(sampled_data)

Comparison

Real data are as follows：

>>> data_connector.read()
       age         workclass  fnlwgt  education  ...  capitalloss hoursperweek native-country  class
0        2         State-gov   77516  Bachelors  ...            0            2  United-States  <=50K
1        3  Self-emp-not-inc   83311  Bachelors  ...            0            0  United-States  <=50K
2        2           Private  215646    HS-grad  ...            0            2  United-States  <=50K
3        3           Private  234721       11th  ...            0            2  United-States  <=50K
4        1           Private  338409  Bachelors  ...            0            2           Cuba  <=50K
...    ...               ...     ...        ...  ...          ...          ...            ...    ...
48837    2           Private  215419  Bachelors  ...            0            2  United-States  <=50K
48838    4               NaN  321403    HS-grad  ...            0            2  United-States  <=50K
48839    2           Private  374983  Bachelors  ...            0            3  United-States  <=50K
48840    2           Private   83891  Bachelors  ...            0            2  United-States  <=50K
48841    1      Self-emp-inc  182148  Bachelors  ...            0            3  United-States   >50K

[48842 rows x 15 columns]

Synthetic data are as follows：

>>> sampled_data
     age workclass  fnlwgt     education  ...  capitalloss hoursperweek native-country  class
0      1       NaN   28219  Some-college  ...            0            2    Puerto-Rico  <=50K
1      2   Private  250166       HS-grad  ...            0            2  United-States   >50K
2      2   Private   50304       HS-grad  ...            0            2  United-States  <=50K
3      4   Private   89318     Bachelors  ...            0            2    Puerto-Rico   >50K
4      1   Private  172149     Bachelors  ...            0            3  United-States  <=50K
..   ...       ...     ...           ...  ...          ...          ...            ...    ...
995    2       NaN  208938     Bachelors  ...            0            1  United-States  <=50K
996    2   Private  166416     Bachelors  ...            2            2  United-States  <=50K
997    2       NaN  336022       HS-grad  ...            0            1  United-States  <=50K
998    3   Private  198051       Masters  ...            0            2  United-States   >50K
999    1       NaN   41973       HS-grad  ...            0            2  United-States  <=50K

[1000 rows x 15 columns]

🤝 Join Community

The SDG project was initiated by Institute of Data Security, Harbin Institute of Technology. If you are interested in out project, welcome to join our community. We welcome organizations, teams, and individuals who share our commitment to data protection and security through open source:

Read CONTRIBUTING before draft a pull request.
Submit an issue by viewing View First Good Issue or submit a Pull Request.

👩‍🎓 Related Work

CTGAN：Modeling Tabular Data using Conditional GAN
C3-TGAN: C3-TGAN- Controllable Tabular Data Synthesis with Explicit Correlations and Property Constraints
TVAE：Modeling Tabular Data using Conditional GAN
table-GAN：Data Synthesis based on Generative Adversarial Networks
CTAB-GAN:CTAB-GAN: Effective Table Data Synthesizing
OCT-GAN: OCT-GAN: Neural ODE-based Conditional Tabular GANs

📄 License

The SDG open source project uses Apache-2.0 license, please refer to the LICENSE.

titalk / synthetic-data-generator Goto Github PK