see negative number in the Duration about spark HOT 6 CLOSED

Hi @aviadr! thanks for reporting the issue!

Duration metric is coming for the difference in start/end dates (if spark is running end date is the current time in your machine)
The data is taken from the first application that is returned from {spark_url}/api/v1/applications API.

It means that in your environment (cloudera onprem, which I haven't tested DataFlint on yet) for some reason the values are skewed.

I suspect that's the root cause for your issues and other calculations such as the DCU and Activity Rate is also skewed because the start/end date are wrong.

Can you please share with me the console output and network requests that DataFlint make, specifically the api/v1/applications one (feel free to blur anything that seems sensitive)? Also, did you install on spark or on spark history server?

I will also be happy to setup a short zoom call and try to debug the issue together

from spark.

Hi,
I installed it on spark (add the jar in the spark-submit command)
the application is still up (it is Jupyter notebook running on yarn cluster )

[ {
"id" : "application_1695491546005_444571",
"name" : "e56d8618-6148-45c6-8596-8edeb7029857",
"attempts" : [ {
"startTime" : "2024-01-21T12:21:26.358GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"lastUpdated" : "2024-01-21T12:21:26.358GMT",
"duration" : 0,
"sparkUser" : "my user",
"completed" : false,
"appSparkVersion" : "3.2.0.3.2.7170.0-49",
"startTimeEpoch" : 1705839686358,
"endTimeEpoch" : -1,
"lastUpdatedEpoch" : 1705839686358
} ]
} ]

from spark.

So the start time seems fine (beside that the ISO format ends with "GMT" and not "T", but in version 0.1.1 DataFlint uses startTimeEpoch which is fine)

Is there a chance that the clock in the machine you are running chrome in is off by ~8-9 hours?
Because DataFlint calculate the duration by taking Date.now() which is the UTC time of your machine and subtract from startTimeEpoch.

from spark.

Anyway if it's not the clock, I will be happy to have an online session to try to debug the problem with you

You can contact me at the DataFlint slack community, join link: https://join.slack.com/t/dataflint/shared_invite/zt-28sr3r3pf-Td_mLx~0Ss6D1t0EJb8CNA

from spark.

Hi,
you are right, it is a clock issue.
the endpoint was falling behind the yarn workers
now it works fine.
thanks for your quick response

from spark.

Awesome :) I will be happy to help with any other question you have, and once you use it a bit I will be really happy to hear what you think about DataFlint :)
As mentioned, you can contact me at the DataFlint slack community, join link: https://join.slack.com/t/dataflint/shared_invite/zt-28sr3r3pf-Td_mLx~0Ss6D1t0EJb8CNA

[Also, I would I suggest for you NOT upgrading to version 0.1.2. I changed a logic relates to calculating the duration and for some reason in Cloudera the time strings ends with GMT and not Z like in any other spark platform, and javascript does not parse natively strings with GMT.

I already fixed it in the main branch and the fix will be available from the next version]

from spark.