Reduce mxTryHard console output about openmx HOT 23 CLOSED

Affected code all in OpenMx/R/MxTryHard.R

non-interactive execution

• No change: leave behavior as is.

Interactive execution

• Output only 1 line to the console. Over-write this line with updated information
• Format fit values format(105.0469144067, sci=3)

Initial console line:

• Beginning fit attempt 1 of (maximum) 10

Subsequent runs should over write:

• Run 1 fit= 17452.243 Worse than previous best (7515.561)
• Run 1 fit= 7452.243 New current best! (was 17515.156)

Last run prints:

• Running final fit for Hessian and/or standard errors and confidence intervals

Terminates by (over-) writing a summary line:

• Solution found! Final fit= 6452.243 (started at 17452.243). (6 attempts: 6 valid, 0 errors)
• Solution not found. Best fit= 168452.243 (started at 17452.243). Retry limit reached. (11 attempts: 6 valid, 5 errors)

Other thoughts for future fixes after this output clean up is completed:

1. Consider writing attributes to the model to store interesting data:
2. start = c(start values for final model)
3. parameterVariance = c(variance found at each parameter)
4. Consider changing default of paste = to FALSE.

• Most users don't want start values to repro the solution, they want the model that ran.

3. Change extraTries to 9 (was 10)

• Means default tries = 10, which suits humans. Consensus is no one will care.

4. Consider (after run 3), adding the 2 parameters which are most variable

• Noisiest parameters: var(T[3,4]) = .03, var(T[2,4]) = .024

code for interactive mode

Add argument silent=interactive()
if(silent){
# short...
}else{
# as is
}

code for overwriting

previousLen <<- 0
imxReportProgress(msg, previousLen)
previousLen <<- nchar(msg)

from openmx.

That should be an argument like (..., silent=interactive()) then if (silent) { ... } else { ... }

from openmx.

would clean up the output to round the fit, maybe to 2 or 3 digits?

Scientific notion with 4-5 sig digits should cover all types of units.

2/11 tries: Best fit= 2.123 (Initially 3.321, previous 4.123)

Default to 10 tries. 11 just looks funny. Also, I'll like to see the 1-2 parameter names with the highest variance (after 3 attempts). So maybe:

4/10: fit=2.124 best=2.123, var(T[3,4]) = .03, var(T[2,4]) = .024

from openmx.

Seconded. Using in-place reporting, tryHard could go from 30+ lines to 2 without loss of information + adding a helpful summary. Just a 1-line, in-place progress bar (perhaps with current min appended in brackets).

m3 = mxTryHardOrdinal(m3)
	Fit 9/11 (min so far =11919)

then erase that, replacing with

Solution (not) found (11 attempts: 6 valid, 5 errors): minimum = 11919.406

The current output in this situation is:

Begin fit attempt 1 of at maximum 11 tries

 Lowest minimum so far:  11919.4063881857

Begin fit attempt 2 of at maximum 11 tries

 Fit attempt generated errors

Begin fit attempt 3 of at maximum 11 tries

Begin fit attempt 4 of at maximum 11 tries

 Fit attempt generated errors

Begin fit attempt 5 of at maximum 11 tries

Begin fit attempt 6 of at maximum 11 tries

Begin fit attempt 7 of at maximum 11 tries

Begin fit attempt 8 of at maximum 11 tries

 Fit attempt generated errors

Begin fit attempt 9 of at maximum 11 tries

 Fit attempt generated errors

Begin fit attempt 10 of at maximum 11 tries

 Fit attempt generated errors

Begin fit attempt 11 of at maximum 11 tries

Retry limit reached

Solution found

Running final fit, for Hessian and/or standard errors and/or confidence intervals

from openmx.

from openmx.

yes @mcneale , wrap in

if(interactive()){
    # short...
}else{
    # as is
}

from openmx.

Also, I hate that the default number of tries goes to 11. Can't we make it 10 instead?

from openmx.

from openmx.

I'll look into this sometime after Thursday, possibly not until next week.

from openmx.

bump... "Just" add:

previousLen <<- 0
imxReportProgress(msg, previousLen)
previousLen <<- nchar(msg)

from openmx.

would clean up the output to round the fit, maybe to 2 or 3 digits?
line to overwrite during run sequence:
paste0(n,"/11 tries: Best fit= ", round(bestFit, digits), " (Initially ", startFit, ", previous ", round(lastFit, digits), ")")

2/11 tries: Best fit= 2.123 (Initially 3.321, previous 4.123)

Maybe don't print "Running final fit, for Hessian and/or standard errors and/or confidence intervals"?

Maybe the final line says

"Solution found: Best fit: 1.123 (Initially 3.321) see someGlobal$start for the final start values, someGlobal$var for the variance found at each parameter."

from openmx.

Default to 10 tries. 11 just looks funny.

Would it make you feel better if we started counting attempts at zero? That is, the initial attempt would be try #0, and the first extra try would be try #1, etc.

from openmx.

Yeah, start counting at zero. 👍

from openmx.

Question: what is the purpose of the <<- operator here? I don't see that the message-printing code would ever need to search "up the stack" into parent environments. Right?

from openmx.

If you can do it without <<-, go for it. I'm lazy.

from openmx.

I just pushed some commits that mostly implement @tbates 's spec. A few remarks:

• Some of the ephemeral printing gets overwritten by other ephemeral printing so quickly that it's difficult or impossible to read.
• In 'Solution found! Final fit= 6452.243 (started at 17452.243). (6 attempts: 6 valid, 0 errors)', is the starting fit value meant to be the fit value from the initial attempt, or at the starting values before mxTryHard() has made any fit attempts?
• As explained in in-code comments, I'm letting the persistent printing requested by iterationSummary=TRUE and showInits=TRUE proceed even when silent=TRUE.

from openmx.

Great stuff! I think starting at should be the model as input (i.e., before any tryhard attempts)

from openmx.

See 7318cfb and a1ab521 . Can this issue be closed, @tbates and @jpritikin ?

from openmx.

the example in mxTryHard.Rd prints nothing to the console. I thought the idea was to leave a summary line, along the lines of:

Solution (not)found! Final fit 6452.243 (started at 17452.243). (6 attempts: 6 valid, 0 errors)

note, I'm just using the last travis build to test this

OpenMx version: 2.9.9.265 [GIT v2.9.9-265-gcac5008]

from openmx.

@tbates : Oh, I see. I misunderstood. You want the summary line to persist.

from openmx.

@tbates : See 6455c49 98eb3ac . Better now?

from openmx.

Looks good @RMKirkpatrick !
I wonder if round(3) or some kind of sci precision might help people quickly grok the fit

model <- mxTryHard(model)

Solution found!  Final fit=-3660.59669854831 (started at NA)  (9 attempt(s): 1 valid, 8 errors) 

from openmx.

See 959a07a .

from openmx.