kenbloom / resource-modeling Goto Github PK

I successfully made a histogram in cpu.py, but it was a real pain in the neck! matplotlib doesn't work the way I think of it. @bbockelman, what would DIANA suggest that I use to make a histogram?

13 PB through LS3, then to 5x that

Size is 1 MB in LHC, 5MB in HL-LHC

Notes: Re-reco passes? A legacy pass during shutdown and legacy MC over previous run. Produces new versions of MC and data. Increases tape, but recover eventually.

Some discussion of whether this really changed much on the disk side

and the base programs as well. Can chain many model pieces together that way

Things I know for sure are missing:

• Analysis model: we are saying it will be trivial in the HL-LHC, but it sure isn't right now.
• Reprocessing of data during shutdown years, i.e. legacy reprocessing.
• HL-LHC simulation in the short term, where it is still quite expensive and a growing part of what we do.

Note that at the moment our model for 2017 has 409 kHS06 needed for data processing and reprocessing; the CRSG document has 628 kHS06, although I suspect that reprocessing in CRSG is an overestimate.

Never archived on tape, only stored temporarily. Add a years worth to disk model?

There are a few numbers that I'd suggest moving into a config file rather than in the code.

We should have the model provide "pretty" human-readable output (for example, a markdown document that includes the produced plots).

Alternately, maybe we have a separate document in this repo that includes a few scenarios that is augmented by the output of the model?

My long-term goal would be that each improvement to the model results in a new build (github pages document?) that can be downloaded.

Things that need to be done:

• Again remove any reference to years and data tiers in the actual code
• fix all TODO
• remove mc_factor and software_improvement as they now have fancy evolving factors my MC flavor

I got the numbers for the current-day GEN/SIM/DIGI/RECO out of the spreadsheet that was used for our most recent C-RSG request:

https://cms-docdb.cern.ch/cgi-bin/DocDB/ShowDocument?docid=13271

I used PU=35 and assumed that was a suitable luminosity scenario through both Run 2 and Run 3, given that the instantaneous luminosity isn't supposed to change much from here through Run 3. But I might have totally misinterpreted the numbers, so can someone please check? Maybe I will try assigning the issues to Chris and Liz. I've never assigned an issue before.

Also: there wasn't a separate value for DIGI in the spreadsheet. But since re-DR = 350 HS06 and RECO alone is listed as 250 HS06, I assumed DIGI = 100 HS06.