Some prompt skimming T0AST queries that could be DAS'ed up about t0 HOT 8 CLOSED

gowdy: Doesn't Oli want;

SELECT run_id FROM run WHERE run_status = ( SELECT id FROM run_status WHERE status = 'Complete' );

Or does he need to know more than jsut what runs have been done?

from t0.

evansde: This actually should be a T0DataSvc call to be added, I think Seangchan handles these still....

from t0.

hufnagel: From that query I can't really tell: What is the use case ? What are you trying to do ?
The one short description "list of runs that should or should not have skimmed" can be had with a much shorter query...

Anyways, adding a DAS query is not going to happen quickly. First the query needs to be optimized/reviewed. Then we add the data service call. Then the rpm gets build and deployed on dev and testbed instances. Will likely be a month or more until this is on cmsweb.

from t0.

dmason: Replying to [comment:1 gowdy]:

Doesn't Oli want;

SELECT run_id FROM run WHERE run_status = ( SELECT id FROM run_status WHERE status = 'Complete' );

Or does he need to know more than jsut what runs have been done?

What Oli ultimately wants in this case is (for his operators) to be able to determine that skims are running as expected. We could go the big green checkmark route, though I've always used the above queries (well actually only the first one) to diagnose whats up.

Results of such a query is here:

http://cmsprod.web.cern.ch/cmsprod/notyetskimmed.txt

With this I can see what blocks need to be picked up, what status they're in and what skims they belong to -- what site, etc. In fact immediately looking at this brings the question why is run 171518 still sitting there in CloseOutAlcaSkimMerge?

Use case: Useful monitoring/debugging table.

I make no claims in favor of the quality of my sql, just that I like the results :)

from t0.

valya: Am I right that query is applied against T0 database? If so can T0 experts outline existing API which can serve such request?

from t0.

hufnagel: Looked at the queries. They aren't too bad as far as the sql is concerned, just needed a bit of restructuring for readability.

Looking at what the queries do, I don't like them. They expose a lot of internal T0AST details for no good reason. To first order Ops should let the system do it's work and only react to actual problems. Almost all the information returned by these queries is no problem at all, just a snapshot of the current Tie0state, which in itself is irrelevant (the information displayed is out-of-date as soon as you display it).

If Ops can clearly define what a problem is, we can implement something that detects that problem. Here for example the run stuck in CloseOutAlcaSkimMerge was a problem and I fixed it. But we have a much simpler query/script for that and Tier0Ops already checks for this (not sure why this wasn't flagged earlier).

The second query is worse, because it basically returns the full export/skimming state for everything in T0AST. Running that query is completely unreasonable unless you do it manually when you expect a major bookkeeping inconsistency and want to compare global states from different sources.

As far as the first query goes, I could be convinced to add a data service call for it if we only check for blocks in 'Exported'. That would detect blocks that for some reason are not picked up by the PromptSkimming. Since the Tie0and PromptSkimming are still separate systems, we cannot really track this seamlessly. Everything before is normal Tie0run closeout completion, which is already tracked by the Tie0 Any block that hasn't reached the Exported stage should not be checked for PromptSkimming completion.

from t0.

hufnagel: On second thought, even that is not needed. I can integrate this check into the normal Tie0run closeout advancement checks. If a run is stuck in CloseOutT1Skimming, I can add a printout to the script which tells Tier0Ops exactly what blocks/sites/skims it is waiting for. That should be sufficient to take the necessary action.

Anyways, I'll be at Fermilab next week, can talk to Oli about it then.

from t0.

hufnagel: After consulting with Steve, it looks like the PromptSkimming system will only update the block status to 'Skimmed' in the Tie0after the data from the block has been feed to the PromptSkimming WMAgent.

Therefore we already monitor everything we need to monitor here. Block status advancement checks are part of the run closeout checks in the Tie0 Tier0Ops just has to look at it (they already run a cron job I think) and react to it. Samir could even add an SLS alarm for it (runs should not stay longer in CloseOutT1Skimming than a couple hours or so).

After the run is set to Complete in the Tie0 all the data that is supposed to be skimmed is in the PromptSkimming WMAgent and can be monitored there.

Therefore I am closing this with wontfix.

from t0.