why is the `in_rate_base` records from the liabilities table all null from 2005-2020? about pudl HOT 8 CLOSED

sigh.... it looks like this problem is coming straight from the dbf sqlite db

@zaneselvans any ideas on checking the source of this?

from pudl.

Ooooh, your plot is so pretty.

from pudl.

You could use dbfread directly to try and pull this table out of say the 2010 archive of the FERC Form 1 inputs and see what you find. The f1_bal_sheet_cr table corresponds to F1_11.DBF and looking at the 2010 data, it seems to contain data. The file is about 2.7MB.

from pudl.

hahah 🎨

import mplcyberpunk
plt.style.use("cyberpunk")

and thanks i'll take a peek

from pudl.

I think I see what's happening:

import dbfread
f1_bal_sheet_cr_dbf = dbfread.DBF(
    "/tmp/UPLOADERS/FORM1/working/F1_11.DBF",
)
f1_bal_sheet_cr_dbf.load()
f1_bal_sheet_cr_df = pd.DataFrame(f1_bal_sheet_cr_dbf.records)
f1_bal_sheet_cr_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 23736 entries, 0 to 23735
Data columns (total 16 columns):
 #   Column      Non-Null Count  Dtype 
---  ------      --------------  ----- 
 0   RESPONDENT  23736 non-null  int64 
 1   REPORT_YEA  23736 non-null  int64 
 2   SPPLMNT_NU  23736 non-null  int64 
 3   ROW_NUMBER  23736 non-null  int64 
 4   ROW_SEQ     23736 non-null  int64 
 5   ROW_PRVLG   23736 non-null  object
 6   BEGIN_YR_B  0 non-null      object
 7   END_YR_BAL  0 non-null      object
 8   BEGIN_YR_2  23736 non-null  int64 
 9   END_YR_BA2  23736 non-null  int64 
 10  REPORT_PRD  23736 non-null  int64 
 11  END_QTR_BA  23736 non-null  int64 
 12  PRI_YR_Q4_  23736 non-null  int64 
 13  END_QTR_B2  23736 non-null  int64 
 14  PRI_YR_Q42  23736 non-null  int64 
 15  _NullFlags  23736 non-null  object
dtypes: int64(12), object(4)
memory usage: 2.9+ MB

The starting and ending year balances are indeed NULL. But there's quarterly data, and there's "end of Q4 last year" data.

f1_bal_sheet_cr_df.sample(20)

from pudl.

yea there is nothing in the ending balance in several years I just checked. There a far amount of END_QTR_BA or end_qtr_bal.

sigh it looks like they reported all of the money as end of quarter balances instead of end of year balances for these years.

from pudl.

I think that means we could integrate the end of quarter data as annual data

from pudl.

Yeah, I think if you select ONLY the report_prd==12 records.

from pudl.