Benchmarking R and C++ for Machine Learning Metrics.

Code is provided here to copy & paste very quickly if needed to use immediately in R. Requires Rtools if using Windows.

Hardware / Software used:

• Intel i7-4600U
• Compilation flags for C/C++: -O2 -mtune=core2 (R’s defaults)
• Windows 8.1 64-bit
• R 3.3.2 + Intel MKL (unless said otherwise)
• Rtools 34 + gcc 4.9

Note: the metrics are tuned for speed. Algorithm wise, interpretability might be lost.
Which means if you were to explain, you will have issues.

Reported numbers are both for log10 weighted average (up) and peak performance (down):

• The multiplication factor (Rcpp / R) for the Throughput+ (over 1 means Rcpp faster, lower than 1 means R faster)
• The throughput observations per second
• The peak vector size

Reported numbers (from log10 weighted average) are:

• Rcpp is in average 22.376% faster than R.
• Rcpp has an estimated average throughput of 18,113,090 observations per second.
• R has an estimated average throughput of 14,801,221 observations per second.
• Fastest functions only. Compiled with -O2 -mtune=core2 flags (R's defaults).

Reported numbers (from the peaks) are:

• Rcpp function throughput peaks at 1,000 observations per call.
• R function throughput peaks at 10,000 observations per call.
• Rcpp is at peak throughput in average 4.434% faster than R.
• Rcpp has an estimated maximum throughput of 19,380,200 observations per second.
• R has an estimated maximum throughput of 18,557,420 observations per second.

For a 2-class vector of 1,000,000 observations:

• Vector A of length=(1000000)
• Vector B of length=(1000000) with 2 classes

A = [1, 2, 3, 4, ..., 1000000]
B = [0, 1, 1, 0, ...]

Get the following Vector C and D:

C = Clamped A by 1e-15
D = Mean of logloss(C, B)

Lpp_logloss(preds, labels, eps):

• preds = your predictions (between 0 and 1)
• labels = your labels (binary, 0 or 1)
• eps = the clamping on [0, 1]

cppFunction("double Lpp_logloss(NumericVector preds, NumericVector labels, double eps) {
  int label_size = labels.size();
  NumericVector clamped(label_size);
  clamped = clamp(eps, preds, 1 - eps);
  NumericVector loggy(label_size);
  loggy = -log((1 - labels) + ((2 * labels - 1) * clamped));
  double logloss = sum(loggy) / label_size;
  return logloss;
}")

Reported numbers (from log10 weighted average) are:

• Rcpp is in average 23.386% faster than R.
• Rcpp has an estimated average throughput of 14,807,458 observations per second.
• R has an estimated average throughput of 12,000,957 observations per second.
• Fastest functions only. Compiled with -O2 -mtune=core2 flags (R's defaults).

Reported numbers (from the peaks) are:

• Rcpp function throughput peaks at 1,000 observations per call.
• R function throughput peaks at 10,000 observations per call.
• Rcpp is at peak throughput in average 22.665% faster than R.
• Rcpp has an estimated maximum throughput of 16,721,500 observations per second.
• R has an estimated maximum throughput of 13,631,870 observations per second.

For a 10-class vector of 1,000,000 observations:

• Vector A of length=(1000000 * 10)
• Vector B of length=(1000000) with 10 classes

A = [1:1, 1:2, 1:3, 1:4... 1:10, 2:1, 2:2, 2:3..., 1000000:8, 1000000:9, 1000000:10]
B = [3, 5, 9, 1, 4, 8, 6, ...]

Get the following Vector C, D, and E:

C = [1:4, 2:6, 3:10, 4:2, 5:5, 6:9, 7:7, ...]
D = Clamped C by 1e-15
E = Mean of logloss(D, B)

Lpp_mlogloss(preds, labels, eps):

• preds = your predictions (size = length(labels) * number of different labels)
• labels = your labels (starting from 0)
• eps = the clamping on [0, 1]

Rcpp::cppFunction("double Lpp_mlogloss(NumericVector preds, NumericVector labels, double eps) {
  int labels_size = labels.size();
  NumericVector selected(labels_size);
  selected = (preds.size() / labels_size) * seq(0, labels_size - 1);
  selected = selected + labels;
  NumericVector to_return(labels_size);
  to_return = preds[selected];
  NumericVector clamped = clamp(eps, to_return, 1 - eps);
  NumericVector loggy = -(log(1 - clamped));
  double logloss = sum(loggy) / labels_size;
  return logloss;
}")

Reported numbers (from log10 weighted average) are:

• Rcpp is in average 43.178% faster than R.
• Rcpp has an estimated average throughput of 4,706,947 observations per second.
• R has an estimated average throughput of 3,287,479 observations per second.
• Fastest functions only. Compiled with -O2 -mtune=core2 flags (R's defaults).

Reported numbers (from the peaks) are:

• Rcpp function throughput peaks at 100 observations per call.
• R function throughput peaks at 1,000 observations per call.
• Rcpp is at peak throughput in average 26.637% faster than R.
• Rcpp has an estimated maximum throughput of 9,162,998 observations per second.
• R has an estimated maximum throughput of 7,235,654 observations per second.

For a 2-class vector of 500,000 observations:

• Vector A of length=(500000)
• Vector B of length=(500000) with 2 classes

A = [1, 2, 3, 4, ..., 500000]
B = [0, 1, 1, 0, ...]

Get the following Vector C:

C = ROC of A and B

Lpp_ROC(preds, labels):

• preds = your predictions
• labels = your labels (binary, 0 or 1)

cppFunction("double Lpp_ROC(NumericVector preds, NumericVector labels) {
  double LabelSize = labels.size();
  NumericVector ranked(LabelSize);
  NumericVector positives = preds[labels == 1];
  double n1 = positives.size();
  Range positives_seq = seq(0, n1 - 1);
  ranked[seq(0, n1 - 1)] = positives;
  double n2 = LabelSize - n1;
  NumericVector negatives = preds[labels == 0];
  NumericVector x2(n2);
  ranked[seq(n1, n1 + n2)] = negatives;
  ranked = match(ranked, clone(ranked).sort());
  double AUC = (sum(ranked[positives_seq]) - n1 * (n1 + 1)/2)/(n1 * n2);
  return AUC;
}")

Reported numbers (from log10 weighted average) are:

• Rcpp is in average 94.945% faster than R.
• Rcpp has an estimated average throughput of 96,532,971 observations per second.
• R has an estimated average throughput of 49,518,000 observations per second.
• Fastest functions only. Compiled with -O2 -mtune=core2 flags (R's defaults).

Reported numbers (from the peaks) are:

• Rcpp function throughput peaks at 100,000 observations per call.
• R function throughput peaks at 10,000 observations per call.
• Rcpp is at peak throughput in average 25.503% faster than R.
• Rcpp has an estimated maximum throughput of 110,431,900 observations per second.
• R has an estimated maximum throughput of 87,991,400 observations per second.

For a regression vector of 1,000,000 observations:

• Vector A of length=(1000000)
• Vector B of length=(1000000)

A = [1, 2, 3, 4, ..., 1000000]
B = [1, 2, 3, 4, ..., 1000000]

Get the following Vector C:

C = SMAPE of A and B

Lpp_ROC(preds, labels):

• preds = your predictions
• labels = your labels (binary, 0 or 1)

cppFunction("double Lpp_SMAPE(NumericVector preds, NumericVector labels) {
  int labels_size = labels.size();
  NumericVector zeroes(labels_size);
  zeroes = (abs(labels - preds)) / (abs(labels) + abs(preds));
  LogicalVector nan = is_nan(zeroes);
  double loss = 0;
  for (int i = 0; i < labels_size; i++) {
    if (!nan[i])
      loss += zeroes[i];
  }
  return(2 * loss / labels_size);
}")

Reported numbers (from log10 weighted average) are:

• Rcpp is in average 65.543% faster than R.
• Rcpp has an estimated average throughput of 97,085,266 observations per second.
• R has an estimated average throughput of 58,646,657 observations per second.
• Fastest functions only. Compiled with -O2 -mtune=core2 flags (R's defaults).

Reported numbers (from the peaks) are:

• Rcpp function throughput peaks at 100,000 observations per call.
• R function throughput peaks at 10,000 observations per call.
• Rcpp is at peak throughput in average 20.260% faster than R.
• Rcpp has an estimated maximum throughput of 110,976,100 observations per second.
• R has an estimated maximum throughput of 92,279,800 observations per second.

For a regression vector of 1,000,000 observations:

• Vector A of length=(1000000)
• Vector B of length=(1000000)

A = [1, 2, 3, 4, ..., 1000000]
B = [1, 2, 3, 4, ..., 1000000]

Get the following Vector C:

C = SMAPE of A and B

Lpp_ROC(preds, labels):

• preds = your predictions
• labels = your labels (binary, 0 or 1)

cppFunction("double Lpp_SMAPE(NumericVector preds, NumericVector labels) {
  int labels_size = labels.size();
  NumericVector zeroes(labels_size);
  zeroes = (abs(labels - preds)) / (abs(labels) + abs(preds));
  LogicalVector nan = is_nan(zeroes);
  double loss = 0;
  for (int i = 0; i < labels_size; i++) {
    if (!nan[i])
      loss += zeroes[i];
  }
  return(2 * loss / labels_size);
}")

Reported numbers (from log10 weighted average) are:

• Rcpp is in average 32.671% faster than R.
• Rcpp has an estimated average throughput of 56,359,885 observations per second.
• R has an estimated average throughput of 42,481,015 observations per second.
• Fastest functions only. Compiled with -O2 -mtune=core2 flags (R's defaults).

Reported numbers (from the peaks) are:

• Rcpp function throughput peaks at 10,000 observations per call.
• R function throughput peaks at 10,000 observations per call.
• Rcpp is at peak throughput in average 40.420% faster than R.
• Rcpp has an estimated maximum throughput of 87,803,300 observations per second.
• R has an estimated maximum throughput of 62,528,900 observations per second.

For a 10-class vector of 1,000,000 observations:

• Vector A of length=(1000000 * 10)
• Vector B of length=(1000000) with 10 classes

A = [1:1, 1:2, 1:3, 1:4... 1:10, 2:1, 2:2, 2:3..., 1000000:8, 1000000:9, 1000000:10]
B = [3, 5, 9, 1, 4, 8, 6, ...]

Get the following Vector C:

C = [1:4, 2:6, 3:10, 4:2, 5:5, 6:9, 7:7, ...]

Lpp_vect2mat2vect(preds, labels):

• preds = your predictions (size = length(labels) * number of different labels)
• labels = your labels (starting from 0)

Rcpp::cppFunction("NumericVector Lpp_vect2mat2vect(NumericVector preds, NumericVector labels) {
  int labels_size = labels.size();
  NumericVector selected(labels_size);
  selected = (preds.size() / labels_size) * seq(0, labels_size - 1);
  selected = selected + labels;
  NumericVector to_return(labels_size);
  to_return = preds[selected];
  return to_return;
}")

Reported numbers (from log10 weighted average) are:

• Rcpp is in average 8.261% faster than R.
• Rcpp has an estimated average throughput of 23,089,263 observations per second.
• R has an estimated average throughput of 21,327,489 observations per second.
• Fastest functions only. Compiled with -O2 -mtune=core2 flags (R's defaults).

Reported numbers (from the peaks) are:

• Rcpp function throughput peaks at 1,000,000 observations per call.
• R function throughput peaks at 10,000 observations per call.
• Rcpp is at peak throughput in average 5.011% faster than R.
• Rcpp has an estimated maximum throughput of 24,111,600 observations per second.
• R has an estimated maximum throughput of 22,961,000 observations per second.

For a 10-class vector of 1,000,000 observations:

• Vector A of length=(1000000)

A = [1, 2, 3, 4, ..., 1000000]

Get the following Vector B:

B = sin(A)

Lpp_sin(x):

• x = your vector to apply sin on.

cppFunction("NumericVector Lpp_sin(NumericVector x) {
  return sin(x);
}")

Reported numbers (from log10 weighted average) are:

• Rcpp is in average 4.675% faster than R.
• Rcpp has an estimated average throughput of 21,228,563 observations per second.
• R has an estimated average throughput of 20,280,469 observations per second.
• Fastest functions only. Compiled with -O2 -mtune=core2 flags (R's defaults).

Reported numbers (from the peaks) are:

• Rcpp function throughput peaks at 100,000 observations per call.
• R function throughput peaks at 10,000 observations per call.
• Rcpp is at peak throughput in average 2.542% faster than R.
• Rcpp has an estimated maximum throughput of 22,113,300 observations per second.
• R has an estimated maximum throughput of 21,565,100 observations per second.

For a 10-class vector of 1,000,000 observations:

• Vector A of length=(1000000)

A = [1, 2, 3, 4, ..., 1000000]

Get the following Vector B:

B = cos(A)

Lpp_cos(x):

• x = your vector to apply cos on.

cppFunction("NumericVector Lpp_cos(NumericVector x) {
  return cos(x);
}")

Reported numbers (from log10 weighted average) are:

• Rcpp is in average 22.649% faster than R.
• Rcpp has an estimated average throughput of 56,551,314 observations per second.
• R has an estimated average throughput of 46,108,091 observations per second.
• Fastest functions only. Compiled with -O2 -mtune=core2 flags (R's defaults).

Reported numbers (from the peaks) are:

• Rcpp function throughput peaks at 100,000 observations per call.
• R function throughput peaks at 1,000 observations per call.
• Rcpp is at peak throughput in average 13.996% faster than R.
• Rcpp has an estimated maximum throughput of 60,454,400 observations per second.
• R has an estimated maximum throughput of 53,031,900 observations per second.

For a 10-class vector of 1,000,000 observations:

• Vector A of length=(1000000)

A = [1, 2, 3, 4, ..., 1000000]

Get the following Vector B:

B = tan(A)

Lpp_tan(x):

• x = your vector to apply tan on.

cppFunction("NumericVector Lpp_tan(NumericVector x) {
  return tan(x);
}")