Preprocessor feature request: Add a way to specify Dataset start_month and end_month about esmvalcore HOT 14 CLOSED

Currently, there is no way to specify the starting month and year that I am aware off. I made a proposal on issue #565 to solve this.

from esmvalcore.

I think there as an alternative solution, which is to use the time_slice preprocessor (loaded as extract_time). I just though it would be easier to add it into the dataset description.

from esmvalcore.

Indeed this is simply not implemented yet. I think the most flexible and clear approach is more or less what @ledm proposed above, i.e. to allow specification at model level. I would propose to use

{
  dataset: HadGEM2-CC
  project: CMIP5
  exp: historical
  ensemble: r1i1p1
  start_year: 1989
  start_month: March
  start_day: 3
  end_year: 2004
  end_month: October
  end_day: 15
}

Specifically, writing out the names of the months is more clear than using a number for the month, because you never really know if 01-09-2001 means the September 1, 2001 or January 9, 2001.

from esmvalcore.

Alternatively, the more concise syntax proposed by @jvegasbsc is also fine with me:

{
  dataset: HadGEM2-CC
  project: CMIP5
  exp: historical
  ensemble: r1i1p1
  start: 1989-03-03 10:24:00
  end: 2004-10-15 11:32:00
}

where we allow any valid yaml datetime specification.

from esmvalcore.

@ledm if you are happy with the proposed solution please close this.

from esmvalcore.

I think we would need to choose and then implement the proposed feature before closing this issue.

At the moment, the only way to do this is with the preprocessor function and it will use the same settings for all datasets:

preprocessors:
  profile:
    extract_time:
      start_year: 1989
      start_month: 12
      end_year: 2004
      end_month: 3

from esmvalcore.

Both solutions are fine, but I also have a preference for the more concise syntax proposed by @jvegasbsc.

Just to make sure, the previous settings will remain valid too, right? We don't want to have to rewrite all the recipes to accommodate this solution.

While we're at it, would it possible to leave the dataset time settings blank and load all available years? That would also be very useful!

Cheers,

Lee

from esmvalcore.

If we're going to change the way you specify the start and end date/time in the diagnostic from 'the current start_year, end_year to start and end as proposed here, I would recommend allowing only that way, because if there's two ways to do it, it is confusing. It isn't that much work to change the few recipes we have so far.

@mattiarighi What is your preferred solution? Option 1 or option 2?

from esmvalcore.

I find Option 1 better, also for the sake of backward compatibility.

Going for Option 2 would not only mean to change the recipes, but also a lot of diagnostics and some shared scripts, since many of the NCL scripts use the attributes start_year and end_year from dataset_info in the interface file.

from esmvalcore.

@ledm Just to get a sense of the priority of this issue: does just specifying the start and end month in the preprocessor solve your issue? Or are the required start and end month different for each dataset?

Can you define a preprocessor like this:

preprocessors:
  profile:
    extract_time:
      start_month: 12
      end_month: 3

and let me know if it solves your problem?

from esmvalcore.

This has been my solution for the entire time. It certainly solved the problem, but it's not great to have to define the time range twice.

In addition, it really would be incredibly useful to have a "run over all available model time" option too. Here are some possible use cases:

1. Every observational dataset will have a different time range.
2. For monitoring ongoing model runs, run over all data would be more useful than looking at a strict pre-defined time range.
3. In case there are some problems with CMIP6 data appearing on BADC in stages.

from esmvalcore.

In addition, it really would be incredibly useful to have a "run over all available model time" option too. Here are some possible use cases:

1. Every observational dataset will have a different time range.
2. For monitoring ongoing model runs, run over all data would be more useful than looking at a strict pre-defined time range.
3. In case there are some problems with CMIP6 data appearing on BADC in stages.

Please open a separate issue to discuss this feature.

from esmvalcore.

Did we ever make a decision on this issue?

@bouweandela wrote:

Alternatively, the more concise syntax proposed by @jvegasbsc is also fine with me:

{
  dataset: HadGEM2-CC
  project: CMIP5
  exp: historical
  ensemble: r1i1p1
  start: 1989-03-03 10:24:00
  end: 2004-10-15 11:32:00
}

where we allow any valid yaml datetime specification.

I agree that this would be a nice change, as long of the old method (specified by year only) was also supported.

This would be a nice change to the way that we load datasets by time range, notably for seasonal calculations, instead of just loading a range of years.

from esmvalcore.

This has been implemented some time ago, it can be done using timerange.

from esmvalcore.