Calculation that returns dateTime data loses time data about aggregate HOT 13 OPEN

To not break existing rounding (which was added to work around timezone confusion issues),
I have added two new JR functions:

date-time(x) 
-- equivalent to date(x), but retains the time.

format-date-time(x, formatstr) 
-- equivalent to format-date(x, formatstr) but retains time of 'x'.

So if you want to refer to a dateTime field, you should always refer to it as date-time(fieldname)?
Otherwise, you lose the time part?

Frustrating. Looked into this further -- my fix is incomplete. date arithmetic that
preserves times is very badly broken.

I'm not sure there is a backward-compatible solution, as the changes would need to
be made within the DateUtils class, and toNumeric() and several other areas where dates
are truncated to date values. These all seem to be kludges for the time zone being
ill-defined.

date-time(x) was intended to convert any non-date value 'x' into a Date, preserving
the time.  This is simply not working because of the rounding within the methods it
relies on.

format-date-time(x,fmt) will format any date value 'x' into a string, with the time
preserved (vs. set to 00:00:00.000 for format-date(...)).

But this has limited utility because any arithmetic on a date will wipe the time.

Ideally, the data type of the field (dateTime vs. date) could be used to determine whether
to retain the time portion of the value.  Then users could get the old rounding behavior
using type="date" and get the behavior retaining the time by using type="dateTime".

But I think that is a bigger fix. It is also where I think the current evaluation logic
is incorrect.

Looking at the raft of recent changes to this code, I fully expected this to work now.
However, there is still a problem with calculate fields being stripped of their times.
(Perhaps it's a minor problem.) Here's an example, where "birthhour" is a datetime
field in an earlier repeat group named "names". The slashes separate the type, name,
and calculation columns in an XLSForm:

calculate / birthhourfromearlier / indexed-repeat(${birthhour}, ${names}, position(..))
calculate / helperlabelhour / format-date-time(${birthhourfromearlier}, '%Y-%b-%e %H:%M:%S')

The second field, in this example, always has 00:00:00 for the time. However, this
works:

calculate / helperlabelhour2 / format-date-time(indexed-repeat(${birthhour}, ${names},
position(..)), '%Y-%b-%e %H:%M:%S') 

In this case, the time is not lost. The only difference is that the first case goes
datetime->calculate->formatting whereas the second goes datetime->formatting.

It's not obvious to me where the time is being lost, because Recalculate.wrapData()
now appears to be returning a datetime object for datetime fields. I am happy to investigate
further, but wanted to check first: is this behavior now by design, and/or was a fix
for this case not intended due to compatibility concerns, etc.? I don't want to spend
time tracing the issue if this is now intentional behavior. Thanks!

By the way, in my example, the birthhourfromearlier field contains just the date (not
even 00:00:00, just "2013-07-29" for, e.g., today). That would seem to imply that,
the datetime field, when converted to a string for display, had no time (and thus went
into the calculate field with no time... so that format-date-time had no time to format
later on). And indeed, if you reference ${birthhour} directly within its repeat group,
you get just the date. So that seems likely the source of the trouble.

From the code, it looks like the display string for datetime fields should be including
the time (via DateUtils.formatDateTime), but clearly something is going wrong somewhere.
Again, I'm happy to trace further if somebody thinks it's worth while (i.e., if this
isn't by design). Thanks.

I hate this code. I hate this code. I hate this code.

Yes, this is definitely does not work-as-intended.

FYI, the calculate expression is held/managed by
 org.javarosa.core.model.condition.Recalculate

I'd love it if you can investigate this.

Okay, we've been delayed here due to our v1.15 release. But I'm slowly investigating.

The date and time display correctly when you edit a form, so the DateTimeData.getDisplayText()
appears to be doing its job. When you refer to the field with ${fieldname}, though,
it drops the time. So it's just a matter of figuring out where that convert-to-string
is happening, and why it doesn't include the time. If it's fixed there, it might also
solve the datetime->calculate issue.

Okay, so much for simplifying assumptions: there are at least two (very similar) problems
here:

1. XPathFuncExpr.toString() doesn't distinguish between date and datetime fields since
they both appear as "instanceof Date". It assumes just a date and formats accordingly
-- so times are lost if you call string(${field}) where field is a datetime field.

2. Recalculate.wrapData() looks like it should work for putting datetime values into
calculate fields, but it still won't work because the target dataType is not datetime
(it's probably string for a calculate field). Thus, again, the code sees "instanceof
Date", sees that the target dataType is NOT DATATYPE_DATE_TIME or DATATYPE_TIME, so
it assumes just a date.

There is almost certainly at least a third case: when ${datetimefield} is included
in a label, code somewhere has to convert datetimefield to a string (to add it to the
label). It may call XPathFuncExpr.toString(), but I sincerely doubt it.

So, pretty much anywhere a Date object is converted to a string, we have problems whereby
times are lost. Two cases are above, there is almost certainly a third case, and there
may be more.

Fun, eh?

Issue 855 has been merged into this issue.

Issue 630 has been merged into this issue.

Issue 808 has been merged into this issue.