We perform extensive real data analyses to systematically evalute several key factors in supervised celltyping tasks, including: feature selection, prediction method, data preprocessing and the choice of the reference dataset. In our paper, we benchmark 9 classifiers (MLP, SVM with Radial Basis Function kernel , SVM with linear kernel, Random Forest, GEDFN, scmap, CHETAH, ItClust and MARS) along with 6 feature selection strategies. We also investigate the impact of data preprocessing (e.g., batch effect correction between reference and target; data imputation on both reference and target), size of reference, number of cell types in reference and different annotation methods in target. Furthermore, we focus on showing how discrepancies between reference and target data would affect the prediction performance. In the end, we explore the strategies of pooling and purifying reference data.

Our paper is currently accepted by Genome Biology and is online here.

We provide all scripts here for running our prediction and analysis results which can be used to reproduce the results in the paper.

In our analysis, we in total set up three conda virtual environments in case some underlying packages are incompatible.

We have stored the package requirements under each virtual environment to envs/and the corresponding environment can be installed by conda create -n NAME --file xx_env.txt. Be aware that R packages installed under the virtual environment will not be listed in the requirement file. We install all R packages (shown below) under the celltyping environment:

• FEAST: Accurate feature selection improves single-cell RNA-seq cell clustering.
• scmap: scmap: projection of single-cell RNA-seq data across data sets.
• CHETAH: Fast and accurate scRNA-seq cell type identification.
• Harmony: Fast, sensitive and accurate integration of single-cell data with Harmony.
• fastMNN:Batch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors.
• SAVER: SAVER: gene expression recovery for single-cell RNA sequencing.

• xx.slurm: These files are used to run experiments on the cluster. If you are also using a computation cluster, you may change some #SBATCH parameters and directly run your own clusters.
• pipelines/: This folder contains all written pipelines along with some auxiliary and analytical scripts. The xx_pipeline.py are all written in Python with argparse to parse argument such as classifier name, feature selection method and other parameters. The xx_analysis.py scripts are used to analyze the result and plot figures. The xx_utils.py are used as auxiliary functions to train the classifiers or load the data. Rcode folder contain FEAST, scmap and CHETAH which are also embedded in Python script to execute.
• preprocessing/: This folder contain all data preprocessing including loading the data into an AnnData object and use scanpy to preprocess the data.
• test_xx/: These folders contain the model implementations.
  • test_GEDFN: MLP and GEDFN are implemented here. We have made some minor modifications to GEDFN to adapt to our pipeline.
  • test_ItClust: ItClust is included here. We also made some minor modifications to adapt to our pipeline.
  • test_batcheffect: Harmony and fastMNN are included here.
  • test_imputation: SAVER, MAGIC and svVI are included here.
  • SVM and Random Forest are directly implemented by scikit-learn package.

• Mouse brain

• Human peripheral blood mononuclear cells (PBMC)

• Human pancreas

References

1. Saunders A, Macosko EZ, Wysoker A, Goldman M, Krienen FM, Rivera H de, et al. Molecular Diversity and Specializations among the Cells of the Adult Mouse Brain. Cell. 2018;174:1015-1030.e16.
2. Bhattacherjee A, Djekidel MN, Chen R, Chen W, Tuesta LM, Zhang Y. Cell type-specific transcriptional programs in mouse prefrontal cortex during adolescence and addiction. Nature communications. Nature Publishing Group; 2019;10:1–18.
3. Ding J, Adiconis X, Simmons SK, Kowalczyk MS, Hession CC, Marjanovic ND, et al. Systematic comparison of single-cell and single-nucleus RNA-sequencing methods. Nature biotechnology. Nature Publishing Group; 2020;38:737–46.
4. Yao Z, van Velthoven CTJ, Nguyen TN, Goldy J, Sedeno-Cortes AE, Baftizadeh F, et al. A taxonomy of transcriptomic cell types across the isocortex and hippocampal formation. Cell. 2021;184:3222-3241.e26.
5. Kang HM, Subramaniam M, Targ S, Nguyen M, Maliskova L, McCarthy E, et al. Multiplexed droplet single-cell RNA-sequencing using natural genetic variation. Nature biotechnology. Nature Publishing Group; 2018;36:89.
6. Zheng GX, Terry JM, Belgrader P, Ryvkin P, Bent ZW, Wilson R, et al. Massively parallel digital transcriptional profiling of single cells. Nature communications. Nature Publishing Group; 2017;8:1–12.
7. Muraro MJ, Dharmadhikari G, Grün D, Groen N, Dielen T, Jansen E, et al. A single-cell transcriptome atlas of the human pancreas. Cell systems. Elsevier; 2016;3:385-394. e3.
8. Segerstolpe Å, Palasantza A, Eliasson P, Andersson E-M, Andréasson A-C, Sun X, et al. Single-cell transcriptome profiling of human pancreatic islets in health and type 2 diabetes. Cell metabolism. Elsevier; 2016;24:593–607.
9. Xin Y, Kim J, Okamoto H, Ni M, Wei Y, Adler C, et al. RNA sequencing of single human islet cells reveals type 2 diabetes genes. Cell metabolism. Elsevier; 2016;24:608–15.

Our code is released under MIT license.