Support for datetime variables about pyfixest HOT 4 OPEN

Hi @NickCH-K, thanks so much, this is exactly the type of feedback I had hoped for!

from pyfixest.

Hi @NickCH-K , I have finally gotten around to take a look at this, and I am not sure if R "gets it right". Depending on the respective datetime, lm and feols do all sorts of transformations, and I am not sure if it's particularly safe or user friendly what pops out.

But I very much agree that an error message is quite annoying, so I am thinking about internally transforming date variables to categoricals and to add a clear warning message that if users want a continuous time variable, they need to turn their datetime variable into a numeric "by hand". On the other hand, I this might backfire if the date variable is high dimensional. What do you think?

Below are R example of different transformations of POSIXct and Date variables:

library(fixest)
library(causaldata)

data(restaurant_inspections)

restaurant_inspections$Time = lubridate::ymd_hms(paste0(restaurant_inspections$Year, '-01-01 00:00:00'))
class(restaurant_inspections$Time)
# [1] "POSIXct" "POSIXt" 

fit = feols(Year~Time, data = restaurant_inspections)
fixest:::model.matrix.fixest(fit, type = "rhs") |> head()
# (Intercept)       Time
# 1           1 1483228800
# 2           1 1420070400
# 3           1 1451606400
# 4           1 1041379200
# 5           1 1483228800
# 6           1 1199145600

restaurant_inspections$Time2 <- as.Date(restaurant_inspections$Time)
class(restaurant_inspections$Time2)
# Date

fit = feols(Year~Time2, data = restaurant_inspections)
fixest:::model.matrix.fixest(fit, type = "rhs") |> head()
# (Intercept) Time2
# 1           1 17167
# 2           1 16436
# 3           1 16801
# 4           1 12053
# 5           1 17167
# 6           1 13879

fit = feols(Year~as.factor(Time2), data = restaurant_inspections)
fixest:::model.matrix.fixest(fit, type = "rhs") |> head()

from pyfixest.

The R transformations are such that you can estimate a linear time trend with them though, right? Even if the scale of the variable isn't easily guessable. It even handles stuff like leap years properly, fitting the time trend on the actual amount of time passed. That seems to me a reasonable expectation when you put a date variable in the non-FE part of the model.

as.numeric(as.Date('2020-01-01'))-as.numeric(as.Date('2019-01-01'))
# 365
as.numeric(as.Date('2021-01-01'))-as.numeric(as.Date('2020-01-01'))
# 366 (leap year!)
as.numeric(lubridate::ymd_hms('2020-01-01 00:00:00'))-as.numeric(lubridate::ymd_hms('2019-01-01 00:00:00'))
# 31536000 = 60*60*24*365

from pyfixest.

Ah ok, I get it, this indeed makes sense! Thanks!

So one simple solution is to convert all dates to int / POSIX:

int(pd.Timestamp('2020-01-01').timestamp()) - int(pd.Timestamp('2019-01-01').timestamp())
# 31536000

I'll try to implement that tomorrow =)

from pyfixest.