Incorrect mapping of contributions to green square boxes about wikicontrib HOT 21 CLOSED

I spent some more time trying to investigate this issue. Though the data stored on the backend and fetched in the frontend matches (see screenshots ), I'm almost concluding that we are logging the dates incorrectly on the backend.

Frontend:

Backend:

Let me take an example of Gerrit data to justify my conclusion:

• I downloaded my commits data via the changes API https://gerrit.wikimedia.org/r/changes/?q=owner:srishakatux.
• From looking at format_data function in views.py here, I sense that we are trying to convert the dates from both Gerrit (default is UTC) and Phabricator (default is epoch) data in Epoch/Unix format.
• I'm looking at the commit created on 2019-09-11 22:54:43.000000000 in the download.json file. If I convert it to epoch using the same formula followed in views.py, that would be:

date_time = utc.localize(datetime.strptime(('2019-09-11 22:54:43.000000000').split('.')[0].split(' ')[0],"%Y-%m-%d"))
epouch = int(date_time.replace(hour=0, minute=0, second=0).strftime("%s"))
print(epouch)
---
1568188800

• Now, ideally, I should be able to find this timestamp 1568188800 in here https://tools.wmflabs.org/contraband/result/6BLO7wadtPjtWw6iDovl0OEi5L4Ey7t8OxCIA0mehdWCZe5P2VgHalL51hiqnM12/. But, I am not able to find it :-/ So, that is one problem.
• Another problem is that when I try to convert 1568188800 back to readable date via a python equivalent of javscript formula used in the codebase, here is what I find:

from time import gmtime
print('{}-{}-{} {}:{}:{}'.format(*gmtime(1568188800)))
2019-9-11 8:0:0

So, it seems like the conversion back from epoch to date is not right as well. Neither the output shows the right date in UTC or PST (my local timezone) format. Also, for the created_on field the format is models.DateTimeField(default=django.utils.timezone.now), I'm not entirely sure what this is doing but might be adding some confusion too, to the data.

I think, first of all; we should store the data, not in epoch but a readable format anyways on the backend. So, that means that for Phabricator, only we do the necessary conversions on the backend. And, then on the frontend, populate the data as per the local timezone. But, this would mean investigating why the date conversions are not working the way they should anyways.

@rammanoj What do you think?

from wikicontrib.

@srish Thanks for the research!!

So, it seems like the conversion back from epoch to date is not right as well. Neither the output shows the right date in UTC or PST (my local timezone) format.

Initially, the datetime is in the form (2019-09-11 22:54:43.000000000) and by converting it to the epouch time, the value is 1568188800. If this is converted back, you are getting same date right ?? I mean (2019-09-11). But the time is varying !!

created_on field the format is models.DateTimeField(default=django.utils.timezone.now), I'm not entirely sure what this is doing but might be adding some confusion too, to the data.

For created_on field, charField is being used (link). Can I know where is the models.DateTimeField is used ?

I think, first of all; we should store the data, not in epoch but a readable format anyways on the
backend. So, that means that for Phabricator, only we do the necessary conversions on the backend. And, then on the frontend, populate the data as per the local timezone. But, this would mean investigating why the date conversions are not working the way they should anyways.

Yeah might be!! The error in timestamps might be because of the local time zones. But I guess there need to be a change in it. I will try to search for the actual error and get back.

from wikicontrib.

@rammanoj If this helps, I just read here that "Contributions are timestamped according to Coordinated Universal Time (UTC) rather than your local time zone" https://help.github.com/en/github/setting-up-and-managing-your-github-profile/why-are-my-contributions-not-showing-up-on-my-profile

from wikicontrib.

Thanks @srish for reporting the issue. This was already faced in development. I forgot to send a patch to it, I am sorry for that, will fix it soon.

from wikicontrib.

@srish I can see 2 commits from gerrit created on 2019-9-12 (in app).

But in the case of tool, all the commits are being displayed on sep 11, this is causing the incorrect mapping!

This might be because of the local timezone's. The tool need to consider the time even!! The information being displayed is according to the UTC format.

All the date time's provided here ( https://gerrit.wikimedia.org/r/changes/?q=owner:srishakatux.) are in the UTC format. So, all the commits being displayed according to the UTC format. Any comments about it @srish ??

If this is the actual error, The commits are needed to be localised I guess...

from wikicontrib.

All the date time's provided here ( https://gerrit.wikimedia.org/r/changes/?q=owner:srishakatux.) are in the UTC format. So, all the commits being displayed according to the UTC format. Any comments about it @srish ??

I think ideally we ensure that the timestamps we get from both the APIs are stored in UTC format on the backend and then we populate the localized version to the user on the frontend.
I am not sure I fully understand, I am guessing that somehow we are storing the local timestamp for both Gerrit and Phabricator looking at L142 and L122. Is that true?

For created_on field, charField is being used (link). Can I know where is the models.DateTimeField is used ?

Here it is at L142, but maybe this is not relevant in the context of this issue as it is related to the query.

from wikicontrib.

Here it is at L142, but maybe this is not relevant in the context of this issue as it is related to the query.

It is query created time. It does not affect the users. It is used because each query has a expiry time of 30 days. The cron job runs for every day night to check if there are any queries that are older than a month. If exists, it deletes all of them!

I think ideally we ensure that the timestamps we get from both the APIs are stored in UTC format on the backend and then we populate the localized version to the user on the frontend.

Yeah! This is the ideal one. I will try to implement this out.

I am not sure I fully understand, I am guessing that somehow we are storing the local timestamp for both Gerrit and Phabricator looking at L142 and L122. Is that true?

Yeah might be!! I should have looked through the time zones when I've implemented it.

from wikicontrib.

@srish Can you paste the link to the query link that displays the result as shown the in the issue description ?

i.e for this image:

from wikicontrib.

@rammanoj sorry for the delay! I couldn't find the exact query link, but here is another one https://tools.wmflabs.org/wikicontrib/ixZb6W7Lkg4MStOxQ656zUxuBZITT95osVX0wyPT0Zro8Q9XsYAaZGQdw3NIOg2p/.

from wikicontrib.

(@rammanoj assigning this issue to you, as this is way beyond my expertise :-/)

from wikicontrib.

@srish I am an Outreachy applicant. I would like to work on this issue

from wikicontrib.

if this issue is still open I would love to work on it @srish

from wikicontrib.

@isunitha98selvan @NdibeRaymond Thank you for your interest in working on this issue! This is a really complicated one. As you could see from the previous comments, we struggled to resolve this earlier :-/ I would not like this issue to be the first issue that you work on this project. So, maybe what you could instead do is, first set up the project, and create any new easy-to-fix issues as and when you discover them :) And, then when you feel a little bit more familiar with the inner workings of this project, you can come back to this issue and claim it.
But, if you feel like you want to give it a shot, you are welcome to claim it even now.

from wikicontrib.

@srish i still want to give it a shot. will help in understanding the project even more

from wikicontrib.

@srish I would be needing your help to replicate this issue. Maybe the contributor's details that your searched for

from wikicontrib.

@NdibeRaymond As there are already lot of issues in your plate right now, maybe wait to get them merged first? :)
If someone else wants to give it a shot, please go ahead!

from wikicontrib.

Ok that's reasonable @srish

from wikicontrib.

@srish can I work on this task now since most of my PRs have been merged?
I will be needing help reproducing it. Maybe the specific user names used

from wikicontrib.

@srish since this seems like a dodgy bug, I would be needing your help reproducing it so I can work my way back from there. A link or name that leads to an occurrence of this issue would be really helpful

from wikicontrib.

@srish, based on the syntaxes I see in the backend code, I have reason to believe we are running python 3.5 in production. if so, your initial assertion that pull request #135 might have fixed this issue might be worth testing

from wikicontrib.

Sharing a gif that shows that the issue exists still:

from wikicontrib.