Comparative Metrics Framework.pdf provides a comprehensive and intuitive framework to evaluate the performance of any synthetic dataset vis-à-vis its original dataset and other synthetic datasets. It captures information on the tradeoff between general utility, specific utility and disclosure risk.

An abridged summary can be found here.

Whilst researching on the best data synthesis method, I struggled with finding suitable benchmarks to evaluate the quality of synthetic datasets. Taub et al.'s work on this is highly excellent, but was not easily reproducible.

I coded previously publicly unavailable functions on ROC and DCAP in the package cmf.

I made modifications to Taub's work when necessary, to ensure an intuitive, comprehensive and systematic evaluation of synthetic datasets.

For code on the ROC and CAP functions as well as their usage examples, refer to the package cmf.

Here is a high-level overview of the comparative metrics framework.

To install cmf

devtools::install_github("MUNFAI15/cmf")

Here, we synthesise the mtcars dataset with the package synthpop. Many of the metrics in this comparative metrics framework are built in synthpop functions; however, they can be applied to any synthetic dataset generated by other methods.

library(cmf)
library(synthpop)
df <- mtcars
syn1 <- syn(df, seed = 1234)
synthpop_df <- syn1$syn

General utility is the degree of similarity in statistical distributions between the original and synthetic dataset.

### ROC - a higher score denotes better overlap. 
ROC_list(df, synthpop_df)
ROC_indiv(df, synthpop_df, "disp")
ROC_score(df, synthpop_df)
ROC_numeric(df, synthpop_df, "disp", y=2)

### Propensity Scores - a lower score denotes better overlap. 
key_var <-("cyl") 
synthpop::utility.tab(syn1, df, key_var)
> Voas Williamson (VW): 2.04 # This is the most relevant result. 

Specific utility evaluates the extent to which we can use our synthetic dataset to gain the same statistical inference as if we were using the original dataset.

### CIO - a higher score denotes better performance. 
lm1 <- lm(formula = cyl ~ gear + wt + carb , data = df)
synfit1 <- lm.synds(data = syn1, cyl ~ gear + wt + carb)
compare(synfit1, df)
summary(synfit1)

Disclosure risk evaluates the probability of disclosing confidential information about individuals in the dataset.

key_var <- c("cyl", "gear")
target_var <- c("wt", "carb")

### CAP - a lower score denotes better disclosure risk. 
CAP_original(df, key_var, target_var)
CAP_baseline(df, target_var)
CAP_synthetic(df, synthpop_df, key_var, target_var)

Author : Chan Mun Fai