Download SPEI data for all ForestGeo sites about climate HOT 28 OPEN

@biancaglez, before you finish this, please be sure to document thoroughly (or enough so that @forestgeoadm knows what you did and can fix it up). Let's organize with folders labeled "scripts", "figures", "data_calculated", and "data_downloaded". The existing SCBI data will go in that folder.

Also, it would be great if you could go ahead and run the script for all ForestGEO sites this file.

from climate.

You're right. There was a gap in the file, now deleted.

from climate.

otherwise pointing people to the R function page should be enough (I think?) since we accepted all of the defaults but the time-scale parameter. https://github.com/sbegueria/SPEI

I think pointing people to the webpage is sufficient.

from climate.

Hi, @biancaglez, got it, that sounds like a good idea. I'll plan to leave the SCBI SPEI readme alone and create a new one that recognizes the methods you followed.

from climate.

found data source: https://spei.csic.es/database.html

• find SPEI source

from climate.

Hi @teixeirak -- I'd appreciate some input on the time scale value to use for the SPEI calculation.
When calculating SPEI per site -- I can input a value for # of months, e.g., 12 would consider the past twelve months in the computation. See the below description for a reference. For now, I am using the default value six and have uploaded these results in in the climate_data > SPEI > ForestGEOsites folder (except for NM which is in the location we talked about above). Find the data here.

An important advantage of the SPEI and the SPI is that they can be computed at different time scales. This way it is possible to incorporate the influence of the past values of the variable in the computation enabling the index to adapt to the memory of the system under study. The magnitude of this memory is controlled by parameter scale. For example, a value of six would imply that data from the current month and of the past five months will be used for computing the SPEI or SPI value for a given month. By default all past data will have the same weight in computing the index, as it was originally proposed in the references below.

Here is an example of using the 12 months to calculate SPEI on the left and on the right is six months.

For additional references when deciding:
http://spei.csic.es/

I would think we should use the 12 value so the data per month considers what the previous year looked like during drought... That or a value like 3 so the previous three months are considered (I would think seasonality would be incorporated in this case)

I will keep reading up on this, but thought I'd put it on your radar for now.

from climate.

some more information from the documentation to decide which time -scale we'd like to choose

At the shortest time scales the drought series show a high frequency of drought, and moist periods of short duration. In contrast, at the longest time scales the drought periods are of longer duration and lower frequency. Thus, short time scales are mainly related to soil water content and river discharge in headwater areas, medium time scales are related to reservoir storages and discharge in the medium course of the rivers, and long time-scales are related to variations in groundwater storage. Therefore, different time scales are useful for monitoring drought conditions in different hydrological sub-systems.

from climate.

From the climate sensitivity analysis paper (I'm trying to figure out how this would fit within the whole method)

So perhaps because we are using the full 15 month time period of growth of the tree ring, we could use this as our time-scale for the SPEI drought index. This will allows us to compare the full annual growth of the tree and the precipitation happening during this time. If we are looking for an annual SPEI value --- this wouldn't overlap with actual years -- 12 monthly value would. ---> this 15 month period would be used as a proxy 'annual' value

Alternatively, using a value of the time window identified by climwin for the month-variable combinations could be another number used in calculating SPEI at sites... Specifically the unique time window for variable with the most influence. This would take into account the previous X months when calculating SPEI -- so for example if WET of June-August of the previous year are the most important months, then using the value three to calculate SPEI would take into account this unique time growth window for that time period (and the drought or lack there of) occuring during that time ... We could then select SPEI annual values from this unqiue time window as a 'annual' value --- ie August 2018 value for SPEI would have incorporated the three previous months...

I like the 15 or 12 month value the most.

What I've found below and in previous comments is the extent of SPEI documentation
https://spei.csic.es/img/fig1.png

Figure 1. Time series of the sc-PDSI and 3-, 6-, 12-, 18- and 24-month SPIs in Indore (India) (1910−2007). (clic for a larger version )

from climate.

and once again here is the current SPEI data for sites (excluding NM which is here) --- six is the default in the SPEI documentation (but up to us to decide)

These data were calculated with the time-scale 6 -- using the previous 6 months to calculate the SPEI for every month..... e.g, you will see the first 5 months in 1901 don't have values because they would need the previous 6 months to be computed...

from climate.

Thanks for working on this, and sorry for the slow reply!

First, a question-- you downloaded these from the SPEI database, as opposed to calculating them yourself, right? (When I've downloaded from that database, I got all the time-scales together in one spreadsheet, but that was just for one site.)

If it's easy, I'd just go ahead and save all the timescales for all the sites, then pick up the best timescale later using climwin or such.

The analysis for the methods paper is giving us the most relevant time scale for each site, and it varies from just 1-2 months up to >8 months. See this figure.

from climate.

Hi @teixeirak -- no I calculated SPEI from our CRU data following the SPEI methodology from their R package.

See the relevent lines from our script here and their documentation explaining how to calculate SPEI for multiple sites here.

and I can save the data for all timescales -- but what is the input value we want to use to calculate this? Maybe we aren't on the same page.--- I guess I can leave the data I already have and later on change the value within the function to generate SPEI easily.

from climate.

I see what you mean saving for all timescales after looking at SCBI data. Sure I can go ahead and do that. So later on we can pick the best time scale using climwin -- so we'd need the file for later analysis... Okay makes sense.

from climate.

Sounds good. I will document and organize thoroughly for Caly.

from climate.

Hi @forestgeoadm

I am dropping what I think would help in creating a README.

The script spei_forestGEO.R is in the Climate > Climate_data > SPEI > Forestgeo_sites location.

This script calculates the Standardised Precipitation-Evapotranspiration Index (SPEI) for all Forestgeo sites and New Mexico (not included in the final CSV and stored in a different location because it is not a forestgeo site -- find NM data here)

The general methodology to calculate SPEI (explained step by step in script if you need additional reference) is as follows:

1. Calculate climatic water balance (BAL) or Precipitation - potential evapotranspiration

note that PET and PRE are computed from our CRU data process -- you could reference that readme.

2. Use SPEI function and calculate using 1-48 months as time scales generating multiple SPEI outputs --
   The value_months_# columns are time scales at which the SPEI is calculated. See below explanation directly from this website: https://spei.csic.es/home.html

3. there was a hack I had to do for calculating BCI SPEI because it had different date ranges from all other sites -- but I don't think you have to mention this...

To cite this function we have to use the following (found on this website-- https://github.com/sbegueria/SPEI)

You can cite these references if you use the SPEI library on your work:

S.M. Vicente-Serrano, S. Beguería, J.I. López-Moreno. 2010. A Multi-scalar drought index sensitive to global warming: The Standardized Precipitation Evapotranspiration Index – SPEI. Journal of Climate 23: 1696, DOI: 10.1175/2009JCLI2909.1.

Beguería S, Vicente-Serrano SM, Reig F, Latorre B. 2014. Standardized precipitation evapotranspiration index (SPEI) revisited: parameter fitting, evapotranspiration models, tools, datasets and drought monitoring. International Journal of Climatology 34(10): 3001-3023.

Value_months_# explanation:
_

Time scales
An important advantage of the SPEI and the SPI is that they can be computed at different time scales. This way it is possible to incorporate the influence of the past values of the variable in the computation enabling the index to adapt to the memory of the system under study. The magnitude of this memory is controlled by parameter scale. For example, a value of six would imply that data from the current month and of the past five months will be used for computing the SPEI or SPI value for a given month. By default all past data will have the same weight in computing the index, as it was originally proposed in the references below. Other kernels, however, are available through parameter kernel. The parameter kernel is a list defining the shape of the kernel and a time shift. These parameters are then passed to the function kern.

_

from climate.

@teixeirak all months calculated

here is the current SPEI data for sites (excluding NM which is here) ---

@forestgeoadm let me know if you have any questions!

from climate.

Hi @biancaglez, thank you for all of this information! I'm working to wrap my head around it and have a few basic questions before I get into it:

1. This README that I'll be putting together based off the above notes that you shared, it should apply to the entire SPEI folder, right? As in, I can take the README currently nestled within the SBCI folder and expand it to the larger network?

2. The link to the NM data is yielding a 404 error.

Take care,
Caly

from climate.

Thanks, @biancaglez ! Looks good on the surface, except that I only see 3 sites in the .csv file.

from climate.

2\. The link to the NM data is yielding a 404 error.

@forestgeoadm , this is because its in a private repo to which you don't have access, so no worries there. It works for me.

from climate.

Hi @teixeirak are you sure you are only getting three sites? I control f'ed and found Scotty Creek, Harvard, Zofin, SCBI, Utah, BCI.... This file correct? https://raw.githubusercontent.com/forestgeo/Climate/master/Climate_Data/SPEI/ForestGEOSites/spei_all_months.csv

from climate.

1. This README that I'll be putting together based off the above notes that you shared, it should apply to the entire SPEI folder, right? As in, I can take the README currently nestled within the SBCI folder and expand it to the larger network?

Yes, it can be added to that. All of our site data begins in 1901 as noted -- except for Barro Colorado Island which begins in 1929.

Another difference is we computed these SPEI values -- whereas before they were coming directly from the DB - so we will now want to cite the above references instead of what you currently have (unless they are the same but the above citation is for the use of the R library so I don't think they are).

In computing our own SPEI we still generate standard Z scores, but to do this we compute the water balance before putting these values into the function. So pasted here below is all still relevant. I'm not sure how detailed we want to be about the function parameters (e.g., which equations are being used in the SPEI function)... but happy to also provide that info @teixeirak -- otherwise pointing people to the R function page should be enough (I think?) since we accepted all of the defaults but the time-scale parameter. https://github.com/sbegueria/SPEI

Calculate climatic water balance (BAL) or Precipitation - potential evapotranspiration
note that PET and PRE are computed from our CRU data process -- you could reference that readme.

from climate.

I did a quick comparison of SCBI results from the script and those downloaded directly from the website:

1mo:

6mo:

So,
1- the script is working as expected
2- the values calculated with CRU data are correlated--but not super closely--with what's available for download on their website. I'm not sure what they use for PRE and PET; that would be what's driving the difference.

from climate.

oh wait @forestgeoadm you may want to just make a second readme (copying the one you referenced) because Krista reorganized the data into data_downloaded and data_from_script ---> So we will want people to know these data are generated from the script via above process.

from climate.

@teixeirak it looks like they are using the most recent CRU so not sure why it would be different https://spei.csic.es/database.html

from climate.

@teixeirak it looks like they are using the most recent CRU so not sure why it would be different https://spei.csic.es/database.html

Maybe the version I downloaded is already dated?

from climate.

Great @forestgeoadm , I typed everything up pretty quickly, so let me know if you have any questions!

And @teixeirak yes, likely. Not sure when you downloaded though

from climate.

Hello, I try to download CRU TS4.05 PRE and PET data to calculate SPEI, but I find the units is different (PRE is mm/month and PET is mm/day), How did I transfer these to calculate SPEI?

from climate.

Hi @Yinshao-lzu ,

To do this, you need to multiply the daily PET by n days in each month. The code we have to do this here in this repo (probably this: https://github.com/forestgeo/Climate/blob/master/Climate_Data/SPEI/data_calculated_with_script/spei_forestGEO.R) should do that, although I don't have time at the moment to dig up the specific location.

Let me know if you have any further questions.

from climate.

Hi @Yinshao-lzu ,

To do this, you need to multiply the daily PET by n days in each month. The code we have to do this here in this repo (probably this: https://github.com/forestgeo/Climate/blob/master/Climate_Data/SPEI/data_calculated_with_script/spei_forestGEO.R) should do that, although I don't have time at the moment to dig up the specific location.

Let me know if you have any further questions.

Thank you very much!

from climate.