different results on Mac, Linux, Windows about abess HOT 2 CLOSED

I'm very sorry for taking such a long time to reply to you. Your question did indeed trouble me for quite some time.

Let me start with the conclusion: there are no bugs in the program, and the differing results are within the normal range.

One way to improve the results is by specifying an appropriate value for the parameter lambda in abess. Additionally, removing highly correlated variables can be also helpful, particularly c500_c694, c640_c694, c220_c694, c750_c694, c92_c694, c235_c694, c811_c694, etc., when only one sample is non-zero for each of them.

Below is the specific reason behind the differing results: When selecting s variables, abess requires that there isn't excessive multicollinearity among any 2s variables. For instance, if we have 1000 samples and need to select 50 out of 10000 variables, even though there will inevitably be multicollinearity issues involving not more than 1000 variables in the data, abess typically works correctly. However, if variable1 and variable2 have a (nearly) perfect correlation (a multicollinearity issue involving only 2 variables), the choice of variable1 over variable2 by abess will depend on the underlying numerical computation. It's worth noting that either choice is correct, and abess does not perform any additional checks in this case.

In fact, I have observed the phenomenon where the line of code "Eigen::VectorXd beta_full = XTX.ldlt().solve(XTy);" yields different results on macOS and Windows. This is due to the fact that the matrix XTX is singular, so the solution is not unique. This aligns with Eigen's documentation: "This method just tries to find as good a solution as possible." Since abess is an iterative algorithm, it's foreseeable that subsequent results may differ.

Finally, thank you for your feedback on the abess project. If you have any further questions or need to continue the discussion, please feel free to reach out. Your input is greatly appreciated.

from abess.

Thanks for looking into this. I appreciate the detailed response! In the pipeline we're developing we'll implement some additional screening to head this off before running ABESS.

from abess.