cov2cor() variant "robust" to non-positive variances about openmx HOT 18 CLOSED

from openmx.

@tbates Yeah, good points. I changed the name of the issue to reflect the worthwhile "cov2cor" enhancement.

from openmx.

I believe

• Function for MxAlgebras that assembles a covariance matrix from an upper left block, a lower left block, and a lower right block.

exists as the algebras vech2full and vechs2full.

from openmx.

@mhunter1 Are you sure about that? The motivation was to condense frequently occurring algebra expressions like

rbind( cbind(A+C+E , A+C),
     cbind(A+C   , A+C+E))

from openmx.

Ahh, you're right vech2full would not work in this case. We could use

ha %x% A + hc %x% C + he %x% E

for

• ha equal to a 2x2 matrix of all ones for MZ or a correlation matrix with .5 correlation for DZ
• hc equal to a 2x2 matrix of all ones
• he equal to a 2x2 identity matrix

I've also brought up the possibility of a new mxExpectationNormal subtype for BG models.

mxExpectationBG(components=c('A', 'C', 'E'), relations=nice defaults, means=nice defaults)

from openmx.

something like ACEbind(A, C, E) that spat out

rbind( cbind(A+C+E , A+C),
     cbind(A+C   , A+C+E))

might be useful, but in my experience going down these routes, the obscurantism that results makes them not worthwhile. Code readability and transparency is a big plus. For learners too.

Likewise mxExpectationBG(components=c('A', 'C', 'E'), relations, means), I think. People working at this level are fine with the raw OpenMx, and mainly need flexible variations on the theme.

One function I've played with adding back in is a simple mxObjectiveNormalML which would obviate the double-tap we now go through in writing:

mxExpectationNormal("top.expCovDZ"),
mxFitFunctionML()

Even this hasn't reached the good-enough-to-maintain-and-get-people-to-learn-a-new-function threshold.

Good examples of functions-worth-having, IMHO, is mxFitFunctionMultigroup(c("MZ", "DZ")). That's a time save that add features while improving code readability and transparency.

from openmx.

As I see it, there are three options for how a "robust" cov2cor() would work:

1. Mask invalid elements as zero. This seemed to be the preferred solution at Boulder, but silently returning a seemingly valid numerical value seems like a bad design to me. Also, further matrix operations involving the resulting correlation matrix could propagate the zero masks via multiplication and lead to elements of other matrices being bogus zeroes.
2. Mask invalid elements as something non-finite. If we do this, then we need to ensure that the presence of non-finite elements doesn't lead to fatal errors later on. Also, further matrix operations involving the resulting correlation matrix could propagate the non-finite values into other algebras.
3. Filter out rows and columns corresponding to invalid elements. This is really sensible if the filtered matrix can inherit the dimnames of the non-filtered rows and columns. In fact, maybe it wouldn't be sensible even then: the resulting correlation matrix might be used in additional algebras in which it would then be non-conformable for the operations involved.

from openmx.

I'm bumping this issue because the matter of non-PD covariance matrices of latent variables has recently been brought to my attention as a source of user confusion and frustration. I suggest a new function, usable in MxAlgebras, called omxCov2cor(), which does not throw errors if the input matrix is non-PD (and in particular, if any of its diagonals is negative).

As for my comment immediately above, the only sane is option 2, that is, if an element of the output matrix is NaN, let it be.

from openmx.

the only sane is option 2, that is, if an element of the output matrix is NaN, let it be.

Hm, okay, but this conflicts with #240. Or I guess you could have both, but you'd need to do extra checking to prevent users from shooting themselves in the foot.

from openmx.

Those unsafe BLAS optimizations would be off by default. A user who does, say,

mxOption(NULL,"Unsafe omxBLAS","Yes")

does so at their own risk. It says it's unsafe right in the name!

from openmx.

In classic Mx, certain illegitimate results would flag a MODE=-1 to tell the optimizer that the function could not be evaluated at this point. The optimizer would back away from such a place by reducing its step size, something that I think we have in OpenMx. In other cases of non-pd matrices I would manually to add a penalty function that would ideally be somewhat a smooth function of the degree of violation. This is more iffy, since the function would have to be a worse value than what was tried previously (and ideally worse than any previously found point).

from openmx.

BTW, what's the use case for omxCov2cor? Is it intended to read out results from a model or is it used to define a model?

from openmx.

In other cases of non-pd matrices I would manually to add a penalty function that would ideally be somewhat a smooth function of the degree of violation.

Doesn't this lead back to a Cholesky parameterization? Or do you want models that are some kind of compromise between Cholesky and non-pd solutions?

from openmx.

A Cholesky parameterization is one way to avoid non-pd A C or E covariance matrices. However, this formulation with implicit boundaries has bad statistical properties (see ). Usually, non-pd estimates arise because the model is deficient in some respect, e.g., not enough factors or a residual or two need to be allowed to correlate. Although (ultra- or plain) Heywood cases* may be considered to be different, they are essentially the same when considering a transformation of the model into common and specific factors instead of a covariance path.

*Verhulst, B., Prom-Wormley, E., Keller, M., Medland, S., Neale, M.C.(2019). Type I Error Rates and Parameter Bias in Multivariate Behavioral Genetic Models. Behav Genet. 49(1):99-111).

**https://v8doc.sas.com/sashtml/stat/chap26/sect21.htm

from openmx.

Actually, never mind about omxCov2cor(). cov2cor() is already usable in MxAlgebras, and does not throw errors at runtime if it's given a matrix that is non-PD, or even has a negative diagonal. It does warn if called from R in the latter situation, though.

I am now trying to remember why this issue was ever opened. I remember that it's a feature Sarah Medland requested.

from openmx.

@mcneale Yeah, I understand why the Cholesky parameterization has trouble. Thanks for the reminder.

I am now trying to remember why this issue was ever opened.

Yeah, I'm wondering that too. If researchers want a silver bullet to ward off negative variances or proportions greater than one, I don't think we have anything to offer them yet.

from openmx.

In the OP of this issue, I wrote:

Variation on cov2cor(), for MxAlgebras, that is "robust" to non-positive variances, either by masking them with zeroes or filtering out the necessary rows and columns.

So I guess that was the original idea? Both of those sound like terrible ideas to me now, especially the second one!

from openmx.

Given that the title of the issue cov2cor() variant "robust" to non-positive variances is rejected, closing.

from openmx.