Giter Club home page Giter Club logo

egret's Introduction

EGRET EGRET

CRAN version

Exploration and Graphics for RivEr Trends (EGRET): An R-package for the analysis of long-term changes in water quality and streamflow, including the water-quality method Weighted Regressions on Time, Discharge, and Season (WRTDS).

Look for new and improved documentation here: https://rconnect.usgs.gov/EGRET/

The link for the official USGS publication user guide is here:

https://pubs.usgs.gov/tm/04/a10/

A companion package EGRETci implements a set of approaches to the analysis of uncertainty associated with WRTDS trend analysis.

If you are familiar with the traditional EGRET workflow, check out the Overview and Updates to see how all the latest updates relate.

Recent introduction to WRTDS and the EGRET package at the 12th National Monitoring Conference April 19, 2021:

<iframe width="560" height="315" src="https://www.youtube.com/embed/d58h3nIc1cc" frameborder="0" allowfullscreen> </iframe>

New capabilities

Package Installation

To install the EGRET package, you must be using R 3.0 or greater and run the following command:

install.packages("EGRET")

Background:

Evaluating long-term changes in river conditions (water quality and discharge) is an important use of hydrologic data. To carry out such evaluations, the hydrologist needs tools to facilitate several key steps in the process: acquiring the data records from a variety of sources, structuring it in ways that facilitate the analysis, routines that will process the data to extract information about changes that may be happening, and graphical techniques that can display findings about change. The R package EGRET (Exploration and Graphics for RivEr Trends) was developed for carrying out each of these steps in an integrated manner. It is designed to accept easily data from three sources: U.S. Geological Survey hydrologic data, Water Quality Portal Data (currently including U.S. Environmental Protection Agency (EPA) STORET data, and USDA STEWARDS data), and user-supplied flat files. The EGRET package has components oriented towards the description of long-term changes in streamflow statistics (high flow, average flow, and low flow) as well as changes in water quality. For the water-quality analysis, it uses Weighted Regressions on Time, Discharge and Season (WRTDS) to describe long-term trends in both concentration and flux. EGRET also creates a wide range of graphical presentations of the water-quality data and of the WRTDS results. The following report serves as a user guide, providing detailed guidance on installation and use of the software, documentation of the analysis methods used, as well as guidance on some of the kinds of questions and approaches that the software can facilitate.

EGRET includes statistics and graphics for streamflow history, water quality trends, and the statistical modeling algorithm Weighted Regressions on Time, Discharge, and Season (WRTDS). Please see the official EGRET User Guide for more information on the EGRET package:

https://doi.org/10.3133/tm4A10 The best ways to learn about the WRTDS approach is to read the User Guide and two journal articles. These articles are available, for free, from the journals in which they were published. The first relates to nitrate and total phosphorus data for 9 rivers draining to Chesapeake Bay. The URL is:

https://onlinelibrary.wiley.com/doi/full/10.1111/j.1752-1688.2010.00482.x.

The second is an application to nitrate data for 8 monitoring sites on the Mississippi River or its major tributaries. The URL is:

https://pubs.acs.org/doi/abs/10.1021/es201221s

For a thorough discussion of the generalized flow normalization method implemented in the EGRET enhancements, see the paper: “Tracking changes in nutrient delivery to western Lake Erie: Approaches to compensate for variability and trends in streamflow”:

(https://www.sciencedirect.com/science/article/pii/S0380133018302235).

Sample Workflow

WRTDS on the Choptank River at Greensboro MD, for Nitrate:

library(EGRET)

############################
# Gather discharge data:
siteID <- "01491000" #Choptank River at Greensboro, MD
startDate <- "" #Gets earliest date
endDate <- "2011-09-30"
# Gather sample data:
parameter_cd<-"00631" #5 digit USGS code
Sample <- readNWISSample(siteID,parameter_cd,startDate,endDate)
#Gets earliest date from Sample record:
#This is just one of many ways to assure the Daily record
#spans the Sample record
startDate <- min(as.character(Sample$Date)) 
# Gather discharge data:
Daily <- readNWISDaily(siteID,"00060",startDate,endDate)
# Gather site and parameter information:

# Here user must input some values for
# the default (interactive=TRUE)
INFO<- readNWISInfo(siteID,parameter_cd)
INFO$shortName <- "Choptank River at Greensboro, MD"

# Merge discharge with sample data:
eList <- mergeReport(INFO, Daily, Sample)
library(EGRET)
# Sample data included in package:
eList <- Choptank_eList

boxConcMonth(eList)

boxQTwice(eList)

plotConcTime(eList)

plotConcQ(eList)

multiPlotDataOverview(eList)

# Run WRTDS model:
eList <- modelEstimation(eList)
#> 
#>  first step running estCrossVal may take about 1 minute
#>  estCrossVal % complete:
#> 0    1   2   3   4   5   6   7   8   9   10  
#> 11   12  13  14  15  16  17  18  19  20  
#> 21   22  23  24  25  26  27  28  29  30  
#> 31   32  33  34  35  36  37  38  39  40  
#> 41   42  43  44  45  46  47  48  49  50  
#> 51   52  53  54  55  56  57  58  59  60  
#> 61   62  63  64  65  66  67  68  69  70  
#> 71   72  73  74  75  76  77  78  79  80  
#> 81   82  83  84  85  86  87  88  89  90  
#> 91   92  93  94  95  96  97  98  99  
#> Next step running  estSurfaces with survival regression:
#> Survival regression (% complete):
#> 0    1   2   3   4   5   6   7   8   9   10  
#> 11   12  13  14  15  16  17  18  19  20  
#> 21   22  23  24  25  26  27  28  29  30  
#> 31   32  33  34  35  36  37  38  39  40  
#> 41   42  43  44  45  46  47  48  49  50  
#> 51   52  53  54  55  56  57  58  59  60  
#> 61   62  63  64  65  66  67  68  69  70  
#> 71   72  73  74  75  76  77  78  79  80  
#> 81   82  83  84  85  86  87  88  89  90  
#> 91   92  93  94  95  96  97  98  99  
#> Survival regression: Done

#eList:
plotConcTimeDaily(eList)
#> plotGenConc = TRUE requires running WRTDSKalman
#>               on eList. Switching to WRTDS concentration.

plotFluxTimeDaily(eList)
#> plotGenConc = TRUE requires running WRTDSKalman
#>               on eList. Switching to WRTDS concentration.

plotConcPred(eList)

plotFluxPred(eList)

plotResidPred(eList)

plotResidQ(eList)

plotResidTime(eList)

boxResidMonth(eList)

boxConcThree(eList)

plotConcHist(eList)

plotFluxHist(eList)

# Multi-line plots:
date1 <- "1985-09-01"
date2 <- "1997-09-01"
date3 <- "2010-09-01"
qBottom<-0.2
qTop<-10
plotConcQSmooth(eList, date1, date2, date3, qBottom, qTop, 
                   concMax=2,legendTop = 0.85)

q1 <- 2
q2 <- 10
q3 <- 20
centerDate <- "07-01"
yearEnd <- 1980
yearStart <- 2010
plotConcTimeSmooth(eList, q1, q2, q3, centerDate, yearStart, yearEnd, legendTop = 0.55, legendLeft = 1990)

# Multi-plots:
fluxBiasMulti(eList)

#Contour plots:
clevel<-seq(0,2,0.5)
yearStart <- 1980
yearEnd <- 2010

plotContours(eList, yearStart,yearEnd,qBottom=0.5,
             qTop = 20, contourLevels = clevel)

plotDiffContours(eList, year0 = 1990,
                 year1 = 2010,
                 qBottom = 0.5,
                 qTop = 20,
                 maxDiff = 0.6)

Sample workflow for a flowHistory application for the entire record

library(EGRET)

# Flow history analysis
# Gather discharge data:
siteID <- "01491000" #Choptank River at Greensboro, MD
startDate <- "" # Get earliest date
endDate <- "" # Get latest date
Daily <- readNWISDaily(siteID, "00060", startDate, endDate)
#> There are 27427 data points, and 27427 days.
# Gather site and parameter information:
# Here user must input some values for
# the default (interactive=TRUE)
INFO <- readNWISInfo(siteID, "00060")
#> Your site for streamflow data is:
#>  01491000 .
#> Your site name is CHOPTANK RIVER NEAR GREENSBORO, MD 
#> but you can modify this to a short name in a style you prefer. 
#> This name will be used to label graphs and tables. 
#> If you want the program to use the name given above, just do a carriage return,
#> otherwise enter the preferred short name(no quotes):
#> 
#> The latitude and longitude of the site are:  38.99719 ,  -75.78581 (degrees north and west).
#> 
#> The drainage area at this site is  113 square miles
#>  which is being stored as 292.6687 square kilometers.
#> 
#> It is helpful to set up a station abbreviation when doing multi-site studies,
#> enter a unique id (three or four characters should work). It is case sensitive.  
#> Even if you don't feel you need an abbreviation for your site you need to enter something(no quotes):
#> 
#> Your water quality data are for parameter number:
#> 00060 
#> which has the name:' Discharge, cubic feet per second '.
#> Typically you will want a shorter name to be used in graphs and tables.
#> The suggested short name is:' Stream flow, mean. daily '.
#> If you would like to change the short name, enter it here, 
#> otherwise just hit enter (no quotes):
#> The units for the water quality data are:  ft3/s .
#> It is helpful to set up a constiuent abbreviation, enter a unique id 
#> three or four characters should work something like tn or tp or NO3).
#> Even if you don't feel you need an abbreviation you need to enter something (no quotes):
#> 
#> Required concentration units are mg/l. 
#> The INFO dataframe indicates: ft3/s 
#> Flux calculations will be wrong if units are not consistent.
INFO$shortName <- "Choptank River at Greensboro, MD"
eList <- as.egret(INFO, Daily, NA, NA)

# Check flow history data:
plotFlowSingle(eList, istat = 7,qUnit = "thousandCfs")

plotSDLogQ(eList)

plotQTimeDaily(eList, qLower = 1, qUnit = 3)

plotFour(eList, qUnit=3)

plotFourStats(eList, qUnit=3)

Model Archive

When using the WRTDS model, it is important to be able to reproduce the results in the future. The following version of R and package dependencies were used most recently to pass the embedded tests within this package. There is no guarantee of reproducible results using future versions of R or updated versions of package dependencies; however, we will make diligent efforts to test and update future modeling environments.

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.2.2 (2022-10-31 ucrt)
#>  os       Windows 10 x64 (build 19044)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  English_United States.utf8
#>  ctype    English_United States.utf8
#>  tz       America/Chicago
#>  date     2023-02-03
#>  pandoc   2.19.2 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package       * version date (UTC) lib source
#>  assertthat      0.2.1   2019-03-21 [1] CRAN (R 4.2.1)
#>  bit             4.0.5   2022-11-15 [1] CRAN (R 4.2.2)
#>  bit64           4.0.5   2020-08-30 [1] CRAN (R 4.2.2)
#>  class           7.3-20  2022-01-16 [2] CRAN (R 4.2.2)
#>  classInt        0.4-8   2022-09-29 [1] CRAN (R 4.2.1)
#>  cli             3.6.0   2023-01-09 [1] CRAN (R 4.2.2)
#>  colorspace      2.1-0   2023-01-23 [1] CRAN (R 4.2.2)
#>  crayon          1.5.2   2022-09-29 [1] CRAN (R 4.2.1)
#>  curl            5.0.0   2023-01-12 [1] CRAN (R 4.2.2)
#>  dataRetrieval   2.7.12  2023-02-01 [1] local
#>  DBI             1.1.3   2022-06-18 [1] CRAN (R 4.2.1)
#>  digest          0.6.31  2022-12-11 [1] CRAN (R 4.2.2)
#>  dotCall64       1.0-2   2022-10-03 [1] CRAN (R 4.2.1)
#>  dplyr           1.0.10  2022-09-01 [1] CRAN (R 4.2.1)
#>  e1071           1.7-12  2022-10-24 [1] CRAN (R 4.2.2)
#>  EGRET         * 3.0.8   2023-01-24 [1] local
#>  ellipsis        0.3.2   2021-04-29 [1] CRAN (R 4.2.1)
#>  evaluate        0.20    2023-01-17 [1] CRAN (R 4.2.2)
#>  fansi           1.0.4   2023-01-22 [1] CRAN (R 4.2.2)
#>  fastmap         1.1.0   2021-01-25 [1] CRAN (R 4.2.1)
#>  fields          14.1    2022-08-12 [1] CRAN (R 4.2.1)
#>  generics        0.1.3   2022-07-05 [1] CRAN (R 4.2.1)
#>  ggplot2         3.4.0   2022-11-04 [1] CRAN (R 4.2.2)
#>  glue            1.6.2   2022-02-24 [1] CRAN (R 4.1.3)
#>  gridExtra       2.3     2017-09-09 [1] CRAN (R 4.2.1)
#>  gtable          0.3.1   2022-09-01 [1] CRAN (R 4.2.1)
#>  highr           0.10    2022-12-22 [1] CRAN (R 4.2.2)
#>  hms             1.1.2   2022-08-19 [1] CRAN (R 4.2.1)
#>  htmltools       0.5.4   2022-12-07 [1] CRAN (R 4.2.2)
#>  httr            1.4.4   2022-08-17 [1] CRAN (R 4.2.2)
#>  KernSmooth      2.23-20 2021-05-03 [2] CRAN (R 4.2.2)
#>  knitr           1.42    2023-01-25 [1] CRAN (R 4.2.2)
#>  lattice         0.20-45 2021-09-22 [2] CRAN (R 4.2.2)
#>  lifecycle       1.0.3   2022-10-07 [1] CRAN (R 4.2.1)
#>  magrittr        2.0.3   2022-03-30 [1] CRAN (R 4.1.3)
#>  maps            3.4.1   2022-10-30 [1] CRAN (R 4.2.2)
#>  Matrix          1.5-3   2022-11-11 [1] CRAN (R 4.2.2)
#>  munsell         0.5.0   2018-06-12 [1] CRAN (R 4.2.1)
#>  pillar          1.8.1   2022-08-19 [1] CRAN (R 4.2.1)
#>  pkgconfig       2.0.3   2019-09-22 [1] CRAN (R 4.2.1)
#>  proxy           0.4-27  2022-06-09 [1] CRAN (R 4.2.1)
#>  R6              2.5.1   2021-08-19 [1] CRAN (R 4.2.1)
#>  Rcpp            1.0.10  2023-01-22 [1] CRAN (R 4.2.2)
#>  readr           2.1.3   2022-10-01 [1] CRAN (R 4.2.1)
#>  rlang           1.0.6   2022-09-24 [1] CRAN (R 4.2.1)
#>  rmarkdown       2.20    2023-01-19 [1] CRAN (R 4.2.2)
#>  rstudioapi      0.14    2022-08-22 [1] CRAN (R 4.2.1)
#>  scales          1.2.1   2022-08-20 [1] CRAN (R 4.2.1)
#>  sessioninfo     1.2.2   2021-12-06 [1] CRAN (R 4.2.1)
#>  sf              1.0-9   2022-11-08 [1] CRAN (R 4.2.2)
#>  spam            2.9-1   2022-08-07 [1] CRAN (R 4.2.1)
#>  survival        3.5-0   2023-01-09 [1] CRAN (R 4.2.2)
#>  tibble          3.1.8   2022-07-22 [1] CRAN (R 4.2.1)
#>  tidyselect      1.2.0   2022-10-10 [1] CRAN (R 4.2.1)
#>  tzdb            0.3.0   2022-03-28 [1] CRAN (R 4.2.1)
#>  units           0.8-1   2022-12-10 [1] CRAN (R 4.2.2)
#>  utf8            1.2.2   2021-07-24 [1] CRAN (R 4.2.1)
#>  vctrs           0.5.2   2023-01-23 [1] CRAN (R 4.2.2)
#>  viridis         0.6.2   2021-10-13 [1] CRAN (R 4.2.1)
#>  viridisLite     0.4.1   2022-08-22 [1] CRAN (R 4.2.1)
#>  vroom           1.6.1   2023-01-22 [1] CRAN (R 4.2.2)
#>  xfun            0.36    2022-12-21 [1] CRAN (R 4.2.2)
#>  yaml            2.3.7   2023-01-23 [1] CRAN (R 4.2.2)
#> 
#>  [1] C:/Users/ldecicco/Documents/R/win-library/4.2
#>  [2] C:/Program Files/R/R-4.2.2/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Reporting bugs

Please consider reporting bugs and asking questions on the Issues page: https://github.com/DOI-USGS/EGRET/issues

Subscribe

Please email questions, comments, and feedback to: [email protected]

Additionally, to subscribe to an email list concerning updates to these R packages, please send a request to [email protected].

Code of Conduct

We want to encourage a warm, welcoming, and safe environment for contributing to this project. See the code of conduct for more information.

Package Support

The Water Mission Area of the USGS has supported the development and maintenance of the EGRET R-package. Further maintenance is expected to be stable through October 2024. Resources are available primarily for maintenance and responding to user questions. Priorities on the development of new features are determined by the EGRET development team.

Sunset date

Funding for EGRET currently expires fall 2024. Expectations are that maintenance and customer service will continue to be supported past that date.

How to cite EGRET:

citation(package = "EGRET")
#> 
#> To cite EGRET in publications, please use:
#> 
#>   Hirsch, R.M., and De Cicco, L.A., 2015, User guide to Exploration and
#>   Graphics for RivEr Trends (EGRET) and dataRetrieval: R packages for
#>   hydrologic data (version 2.0, February 2015): U.S. Geological Survey
#>   Techniques and Methods book 4, chap. A10, 93 p., doi:10.3133/tm4A10
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @InBook{,
#>     author = {Robert M. Hirsch and Laura A. {De Cicco}},
#>     title = {User guide to Exploration and Graphics for RivEr Trends (EGRET) and dataRetrieval: R packages for hydrologic data},
#>     publisher = {U.S. Geological Survey},
#>     address = {Reston, VA},
#>     booktitle = {Techniques and Methods},
#>     institution = {U.S. Geological Survey},
#>     year = {2015},
#>     chapter = {A10},
#>     url = {https://pubs.usgs.gov/tm/04/a10/},
#>   }

References

See this list for WRTDS applications in print:

https://rconnect.usgs.gov/EGRET/articles/References_WRTDS.html

Disclaimer

This software is preliminary or provisional and is subject to revision. It is being provided to meet the need for timely best science. The software has not received final approval by the U.S. Geological Survey (USGS). No warranty, expressed or implied, is made by the USGS or the U.S. Government as to the functionality of the software and related material nor shall the fact of release constitute any such warranty. The software is provided on the condition that neither the USGS nor the U.S. Government shall be held liable for any damages resulting from the authorized or unauthorized use of the software.

egret's People

Contributors

aappling-usgs avatar actions-user avatar ayan-usgs avatar katrinleinweber avatar ldecicco-usgs avatar limnoliver avatar sckott avatar tcadkin-usgs avatar wdwatkins avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

egret's Issues

readWQPSample inserts mdl as value from STORET

The commands:
Sample <- readWQPSample('21FLEECO_WQX-IMPRGR80', 'Phosphorus', "", "")
eList <- mergeReport(INFO, Daily, Sample)
result in:
Discharge Record is 3994 days long, which is 11 years
First day of the discharge record is 2004-01-01 and last day is 2014-12-07
The water quality record has 124 samples
The first sample is from 2004-01-15 and the last sample is from 2014-03-19
Discharge: Minimum, mean and maximum 0.156 2.42 39.6
Concentration: Minimum, mean and maximum 0.006 0.031 0.073
Percentage of the sample values that are censored is 0%

There are censored values at multiple reporting levels. The readWQPSample function writes the correct mdl into the data value field.

A manual download using the generated url results in a dataset with blanks for the censored values. Something like:
http://www.waterqualitydata.us/Result/search?siteid=21FLEECO_WQX-IMPRGR80&characteristicName=Phosphorus&countrycode=US&mimeType=xlsx&zip=yes

Add flexible flow normalization to setupYears

OK... here it goes. We already have a function flexFN that let's you permanently set the flow-normalized calculations to some set of years. This issue is both a simplification of that function, and only temporary. It can either be called within setupYears, or before. It would probably be easiest to implement by putting it within setupYears, and setting a default argument to use the full period of record.

The function flexFN takes a data frame with 3 columns, the first represents where to split the surface, the second and 3rd where to split the discharge.

Instead of that workflow, I'd say we'll only concentrate on discharge, with a single vector of dates to normalize on.

  • Add more tests to setupYears
  • Improve documentation to estFNsegs
  • Create function to use vector of years.
  • Add it to functions before setupYears
  • Double-check all the output for those functions that call setupYears (many are graphing functions)
  • Add examples to functions that call setupYears
  • Write mini-vignette/blog.

@lindsaycarr I'm going to do a couple of these first tasks this afternoon to get it started. I'll have you review whatever PR I end up completing and then you take over from there.

fluxUnit suggestion

For your customers with native data units in ng/L, it would be helpful to have a fluxUnit option of kg/yr. After pulling the data from NWIS, I do the recommended conversion of:

Sample$ConcLow <- Sample$ConcLow / 1000000
Sample$ConcHigh <- Sample$ConcHigh / 1000000
Sample$ConcAve <- Sample$ConcAve / 1000000

to get the data into the expected units of mg/L. After progression through the WRTDS workflow and running tableChange(), I ignore the warning message at the end of the screen output regarding units of ng/L in INFO, but am hopeful that a units value for these smaller annual fluxes could be provided? If you would prefer users do the conversion themselves, no problem, only a suggestion.

tableChange(eList, fluxUnit=8, yearPoints=c(1998,2005,2013))

...
Flux Trends
time span change slope change slope
10^3 kg/yr 10^3 kg/yr /yr % %/yr
1998 to 2005 0.00076 0.00011 10 1.5
1998 to 2013 0.0052 0.00035 70 4.7
2005 to 2013 0.0044 0.00055 54 6.8

or changing fluxUnit=3 yields

tableChange(eList, fluxUnit=8, yearPoints=c(1998,2005,2013))
...
Flux Trends
time span change slope change slope
kg/day kg/day /yr % %/yr
1998 to 2005 0.0021 3e-04 10 1.5
1998 to 2013 0.014 0.00095 70 4.7
2005 to 2013 0.012 0.0015 54 6.8

ND's reported as 0's from WQ Warehouse

I am seeing data which is stored as less thens come up as 0's using dataRetrieval.

library(dataRetrieval)
library(EGRET)
siteNumber<-"MNPCA-S000-154"
characteristicName<-"Inorganic nitrogen (nitrate and nitrite)"
startDate<-"1976-01-01"
endDate<-"2010-12-31"
Sample<-readWQPSample(siteNumber,characteristicName,startDate,endDate)
Deleted Rows:
dateTime ConcLow ConcHigh Uncen
189 2004-10-11 0 0 1
190 2004-11-16 0 0 1
191 2005-02-16 0 0 1
192 2005-03-17 0 0 1
193 2005-04-27 0 0 1
194 2005-05-18 0 0 1
195 2005-06-06 0 0 1
196 2005-07-24 0 0 1
197 2005-08-17 0 0 1
198 2005-09-26 0 0 1
201 2007-02-12 0 0 1
204 2007-05-14 0 0 1
205 2007-06-24 0 0 1
206 2007-08-01 0 0 1
207 2007-08-27 0 0 1
208 2007-09-17 0 0 1
211 2009-04-07 0 0 1
212 2009-04-14 0 0 1
213 2009-04-22 0 0 1
215 2009-05-06 0 0 1
216 2009-05-13 0 0 1
217 2009-05-20 0 0 1
218 2009-06-11 0 0 1
219 2009-06-23 0 0 1
220 2009-07-14 0 0 1
221 2009-07-29 0 0 1
222 2009-08-11 0 0 1
223 2009-08-26 0 0 1
224 2009-09-15 0 0 1
225 2009-10-07 0 0 1
226 2009-10-13 0 0 1
227 2009-11-03 0 0 1
228 2009-11-10 0 0 1
229 2009-11-17 0 0 1
230 2009-12-01 0 0 1
232 2010-03-23 0 0 1
233 2010-04-19 0 0 1
234 2010-05-25 0 0 1
235 2010-06-14 0 0 1
236 2010-07-20 0 0 1
237 2010-08-23 0 0 1
238 2010-09-20 0 0 1
Warning message:
In compressData(data, interactive = interactive) :
Deleted 42 rows of data because concentration was reported as 0.0, the program is unable to interpret that result and is therefore deleting it.

installing EGRET

I received this response:

The downloaded binary packages are in C:\Users\10002932\AppData\Local\Temp\1\RtmpSKjO8I\downloaded_packages > library(dataRetrieval) Warning message: package ‘dataRetrieval’ was built under R version 3.3.3 > library(EGRET) Warning messages: 1: package ‘EGRET’ was built under R version 3.3.3 2: failed to assign RegisteredNativeSymbol for toeplitz to toeplitz since toeplitz is already defined in the ‘spam’ namespace

 

| >
Any suggestions on what to do next?

Rounding off of years in printSeries function

Laura: I just discovered a little glitch. I had set paStart=1, paLong=12 (so it is calendar year) and ran the function printSeries and the years printed on the left column of the output when like this: 1892, 1892, 1894, 1894, 1896,1896 etc. I looked at the annualSeries (as created by makeAnnualSeries) and I see that the year variables go 1891.5000, 1892.5000, 1893.5000.... so with rounding to integers they all round to the nearest even number. There should be an easy fix to the code although I'm not sure what it would be. It appears that it is just a problem when the year value is exactly an integer plus 0.50000.

Not a huge deal, but something that needs to be taken care of.

Thanks.

Bob

readWQPInfo not finding units

INFO <- readWQPInfo('21FLEECO_WQX-IMPRGR80', 'Phosphorus')

results in:
The units for the water quality data are: .

readNWISDaily oddity

As the attached screen capture will show, a gage in western WA (Gage ID = 12010000) apparently has two records under p-code 00060. The first has the nuance "STILLING WELL" and stops at the end of WY2003. The second instance of 00060 is ongoing. When I run readNWISDaily(), it grabs the dataset ending in WY2003. Is there a way to have it instead grab the dataset that is ongoing?

siteNumber <- "12010000"
startDate <- ""
endDate <- ""
Daily <- readNWISDaily(siteNumber,'00060',startDate,endDate)

two_00060_codes

Remove RColorBrewer dependency

It's only in the new function flexPlotAddOn . We could either make our own, or use a built-in one from base, or just make it an argument to the function (with some intelligent defaults)

Refactor interactive text in all of the `INFO` functions.

Probably just re-write the populateParameterINFO and populateSiteINFO functions so that all 3 INFO functions (readWQPInfo, readNWISInfo, and readUserInfo) perform the interactive bits the same (and, only one function needs to be maintained).

Improve help documentation for adjusting windowY, etc.

From an email:
" I wanted to officially request that EGRET documentation include examples of how to add the WRTDS estimate input variables shown in Table 16. Of particular interest are the windowY, windowQ, and windowS variables. "

Useful enhancement to plotContours and plotDiffContours

Laura: this is something that has come up in my interactions with some users and I share their concern. Right now, the y-axis of these two kinds of contour plots are constrained to start and end with discharge values such as 1, 2, 5, 10, 20, 50,... There are times when it would be nice to be able to pick any arbitrary value for the lower or upper bound on the axis. Sometimes when the selected limits are too wide apart, part of the figure plots as white space (because there are no data in that region in the surfaces object). That would be nice to avoid. But also, sometimes we have situations where perhaps a maximum of 20 cuts off too much of the relevant high discharge space, but 50 goes to far and shows results that are very unlikely and thus not very meaningful and the user may want to set the limit at say 30 or 40. I would still like to keep the tick mark values simple (1, 2, 5, 10, etc.) but allow it to go to any value. This is similar to a change you made a while back in plotQTimeDaily so that the lower bound could be at any value selected. Thanks. Bob

Add tests

The non-graphing functions could use better test coverage. Looks like a few of those functions could have a much more simple dplyr workflow.

fluxUnit for bacteria

There does not appear to be any defined fluxUnit for bacteria (concentration units = colonies/100mL). Any suggestions for a work around? or have I just missed something?

https changeover

Found these lines with http .gov URLs using https://github.com/wdwatkins/gran_http_gov

HTTPS is supposed be available for everything by the end of December, but may not be yet.

Some of these in .md and other documentation files won't show up on github unless you go to raw file. Line numbers reference the number in the raw file when opened in a text editor, so the link may not point you to the right spot in some cases if lines are wrapping, etc.

A small bug in the function wordLike in the EGRETci package

Laura: I was looking at a result the the Chesapeake Bay folks had and ran the wBT code on their data. What I found was a contradiction in the descriptions of the likelihood of uptrends and downtrends. What I got was "Upward trend in flux is about as likely as not" but "Downward trend is likely" -- this makes no sense. I see the error in the code. I'm quite sure it was MY error. There is a line in the code that says:

breaks <- c(0, 0.05, 0.1, 0.33, 0.66, 0.9, 0.95, 1)

It should say

breaks <- c(0, 0.05, 0.1, 0.33, 0.67, 0.9, 0.95, 1)

So, not a big deal, but I would like to have you make the change to EGRETci on CRAN and GRAN. Thanks.

Bob

How to handle a concentration of zero

modelEstimtion fails if there is a measurement that claims 0 in the concMin and concMax columns.

We could just check, if there are 0's in concAve, and error out with a useful message, leaving the process of what to do up to the user (ie, drop the rows? insert an artificial detection level? something else?)

readNWISDaily: Values not numeric

Hi,
Trying to download data using readNWISDaily. Running into the error
Error in rawData$value/qConvert : non-numeric argument to binary operator
This is because the data grabbed using dataRetrieval::importRDB1 has the values stored as characters. I think a simple as.numeric() would fix this issue. But there might be a better way.

Example:

readNWISDaily('05427850','00060')
Error in rawData$value/qConvert : non-numeric argument to binary operator

Add 'concMin' argument to plotConcHist()

There is a very subtle, seeming steady, possibly significant (statistically speaking) upward trend in pH (plot below). It is difficult to detect, however, because the y-axis lower limit defaults to 0. Would it be possible to add a concMin argument to plotConcHist(), kind of like what exists in plotConcQSmooth(eList, date1, date2, date3, qBottom, qTop, qUnit=1, printLegend=FALSE, concMin=7)

plotconchist

Discharge units in summary function

From an email:

The user’s guide indicates that the discharge values in the summary (i.e. summary(Daily))are automatically converted to cubic meter per second (m3/s).

Is there a way/ an option to change the units of discharge in the summary (summary(Daily)) to cfs?

plotFour & plotSDLogQ errors

Hello,

I tried plotFour & plotSDLogQ for my data but got the following errors

Error in seq.default(startDec, endDec - window, 0.1) : 
  wrong sign in 'by' argument

How should I fix this? Thank you!

Sample R code:

startDate <- '2007-10-01'
endDate <- '2017-09-30'
station_code <- "12488500"
data_code <- "00060"
dat <- readNWISDaily(station_code, data_code, startDate, endDate)
INFO <- readNWISInfo(station_code, data_code)
eList <- as.egret(INFO, dat , NA, NA)

#### Check flow history data: 
plotFourStats(eList,  qUnit  = 1)
plotSDLogQ(eList)
plotFour(eList, qUnit = 1)

Allow for right-censored data

Currently, only < is recognized as a censored value, and when > appears in the QW data, the value is documented as uncensored and ConcLow and ConcHigh are equal. Per survival package documentation, ConcLow should be the censored value, and ConcHigh should be NA.

This requires changes to populateConcentrations and populateSampleColumns.

Add mergeReport check in modelEstimation

The way modelEstimation fails if mergeReport is not run ahead of time is unclear. We could add a check to see if the proper columns are in Sample, and if not, just go ahead and run mergeReport.

Add tests to table output and other functions

  • calculateMonthlyResults
  • censoredSegments
  • checkStartEndDate
  • compressData
  • dateFormatCheck
  • flowDuration
  • formatCheckDate
  • formatCheckParameterCd
  • logPretty1
  • logPretty3
  • makeAnnualSeries
  • makeAugmentedSample this doesn't have an example either
  • setSeasonLabel
  • mergeReport
  • populateConcentrations better documentation - what actually are the columns?
  • selectDays
  • setPA
  • generalAxis
  • fluxBiasStat

Upon adding tests with the Choptank and/or Arkansas sample data, look for places to add intermediate functions. I expect fewer intermediate functions needed here than #102

as.qw freezes machine

I'm trying to convert some censored data into the qw object type so that rloadest handles it correctly. (I can't pull the data from NWIS). I've tried the code below but it repeatedly freezes my computer. I'm wondering if anyone has suggestions on what I'm doing wrong.
Thanks in advance-
Austin Baldwin

(careful- running this may freeze your machine!)

library(smwrQW)

conc <- c(0.07, 0.04, 0.025, 0.025)
rk <- c("", "", "<", "<")
rl <- c(0.025, 0.025, 0.025, 0.025)

DF <- data.frame(conc, rk, rl)

DF$rk <- as.character(DF$rk)

DF$qwconc <- as.qw(values=DF$conc, remark.codes=DF$rk, value.codes="",
               reporting.level=DF$rl, reporting.units = "none", analyte.method = "none",
               analyte.name = "none", unique.code = "none", value2="none")

Convert readme to readme.Rmd and get graphs to plot

I think you could take the readme as it is, save it as a .Rmd file, add the appropriate header, knit it, and we'd have a fancy readme with lots of graphs. This could then be the "testing platform" for our graphs:

---
title: "README"
author: "R"
date: "`r format(Sys.time(), '%d %B, %Y')`"
output:
  md_document:
    variant: markdown_github
---

Also would need to remember to add the .Rmd and corresponding images to the .Rbuildignore

what happens when the data don't exist?

Laura: I got into a problem today because I tried to use readNWISSample for a station and parameter combination that simply didn't exist. it was 09032000 and 70301 (someone was asking me about an analysis of this and that's what I saw in their INFO data frame). Jordan helped me figure out my problem. Them message I was getting was:

No sites/data found using the selection criteria specified

Error: file must be a string, raw vector or a connection.

Is there any way we can have EGRET (or dataRetrieval) figure out that there is nothing to send back to make the Sample data frame and just send a message that there are no such data?

Thanks.

Bob

blankTime and plotFluxHist or plotConcHist function

There is a problem when a user sets up a blankTime period that has a start and/or end date that is not alligned with the period of analysis for these graphs (it might also apply to some of the tables).

Say I do this:
plotFluxHist(eList)
startBlank <- "2000-01-01"
endBlank <- "2003-12-31"
eList <- blankTime(eList,startBlank,endBlank)
plotFluxHist(eList)

The result we would like is that the two graphs look exactly the same but the latter one will just have a part of the green line vanish and several dots vanish, but no values get changed. In fact, what happens is that the partial water years that haven't been blanked out still figure in so there can be a sharp bend in the green curves next to the break and the data points near the break have moved around.

What needs to happen is that these functions plotConcHist and plotFluxHist need to be modified so that if there are any NA values in a given year within the PA it won't compute a value for that year and it will be NA for purposes of the graph.

I've done a little test case and can see that the problem exists also in setupYears, tableResults and tableChange as well.

At first glance I thought the solution was just to make sure that the blankStart and blankEnd defined a water year, but then I realized that the user may want to look at calendar year or some other PA and the problem really wouldn't be solved.

I think all of this can be fixed with a fix to setupYears (because all of these functions that show the problem depend on setupYears for their annual values).

I think the fix is simply this: In setupYears the line that says:

good <- (sum(counter) > 25)

should actually say

good <- (sum(counter) == length(counter))

I think the sum(counter) > 25 is a relic from when we were doing this summing by month and I wanted there to be at least 25 valid days in the month to make it work).

Probably needs to be fixed on CRAN and on our internal version of EGRET

Thanks. Bob

Add tests to modeling functions

  • modelEstimation
  • estCrossVal
  • estSurfaces
  • runSurvReg
  • estDailyFromSurfaces

Upon adding tests with the Choptank and/or Arkansas sample data, look for places to add intermediate functions.

Helper functions

  • nObs <- nObservations(eList)

  • nCensoredVals(eList)

  • plot(eList)

others?

GRAN Install - package load error

I installed EGRET from the GRAN repo and when I load the package I get the following error:

library(EGRET)
Error : object ‘col_number’ is not exported by 'namespace:readr'
Error: package or namespace load failed for ‘EGRET’

bug in EGRET::plotConcTime when rResid=TRUE and qLower or qUpper is specified

Laura: We seem to have a bug in plotConcTime. It happens when rResid = TRUE and a qLower or qUpper value is specified. The command and message are:

plotConcTime(eList,qLower=20,rResid=TRUE)
Error in xy.coords(x, y, xlabel, ylabel, log) :
'x' and 'y' lengths differ

I can send you the workspace if you would like to have it. I looked at the code but couldn't figure out what is happening.

By the way, there is a little error in the help file for makeAugmentedSample. Under arguments it should say "eList named list with at least the Sample dataframe" -- note that all the processing here is on the Sample data frame so it wouldn't work if there was just the Daily dataframe. Then under "Value" it should say "eList named list with modified Sample dataframe."

I am running EGRET 2.5.1 which was built under R 3.3.0 on 2016-06-01. I'm using R 3.3.0.

Thanks. Bob

Data Retrieval not retrieving NDs

Data Retrieval is not retrieving NDs for any MN data stored after 2002 or so. I get the concentration reports as 0 message although the concentration was not reported as 0 and our statewide data retrieval has no problem getting it. Following is the sample retrieval code:

siteNumber <- "MNPCA-S002-010"

characteristicName <- "Inorganic nitrogen (nitrate and nitrite)"
startDate <- "1992-01-01"
endDate <- "2015-12-31"
Sample <- readWQPSample(siteNumber, characteristicName, startDate,endDate)
Deleted Rows:
dateTime ConcLow ConcHigh Uncen
4 2010-09-20 0 0 1
5 2010-10-20 0 0 1
8 2010-10-12 0 0 1
11 2010-07-13 0 0 1
13 2010-07-27 0 0 1
15 2010-04-21 0 0 1
17 2010-08-09 0 0 1
18 2010-05-19 0 0 1
19 2010-07-07 0 0 1
20 2011-11-01 0 0 1
22 2011-09-13 0 0 1
23 2011-11-16 0 0 1
25 2011-06-15 0 0 1
27 2011-05-18 0 0 1
33 2011-05-09 0 0 1
36 2011-06-08 0 0 1
45 2013-10-09 0 0 1
51 2013-08-06 0 0 1
53 2013-05-29 0 0 1
54 2013-08-27 0 0 1
58 2005-08-17 0 0 1
59 2010-12-06 0 0 1
60 2010-01-12 0 0 1
63 2010-08-23 0 0 1
64 2010-11-22 0 0 1
65 2010-06-08 0 0 1
67 2010-05-13 0 0 1
68 2010-05-26 0 0 1
69 2010-10-05 0 0 1
74 2004-10-20 0 0 1
84 2008-10-07 0 0 1
87 2008-09-09 0 0 1
88 2008-07-09 0 0 1
90 2008-11-04 0 0 1
92 2008-12-09 0 0 1
96 2007-05-14 0 0 1
97 2009-05-13 0 0 1
98 2009-04-22 0 0 1
101 2009-07-29 0 0 1
102 2009-09-15 0 0 1
105 2009-05-20 0 0 1
107 2009-08-11 0 0 1
109 2009-06-23 0 0 1
110 2009-08-26 0 0 1
111 2009-04-29 0 0 1
112 2009-07-14 0 0 1
113 2009-05-06 0 0 1
114 2009-12-01 0 0 1
115 2009-06-09 0 0 1
120 2013-06-17 0 0 1
123 2013-09-10 0 0 1
125 2013-09-25 0 0 1
128 2005-04-20 0 0 1
131 2005-08-31 0 0 1
132 2005-05-18 0 0 1
134 2005-11-02 0 0 1
136 2005-05-11 0 0 1
137 2005-09-14 0 0 1
138 2005-05-23 0 0 1
140 2005-08-03 0 0 1
141 2005-04-27 0 0 1
143 2005-05-04 0 0 1
149 2012-10-17 0 0 1
151 2012-06-25 0 0 1
152 2012-07-02 0 0 1
153 2012-04-10 0 0 1
154 2012-05-14 0 0 1
155 2012-06-05 0 0 1
162 2012-04-30 0 0 1
165 2012-05-22 0 0 1
167 2012-09-11 0 0 1
168 2012-11-07 0 0 1
170 2007-07-17 0 0 1
172 2007-05-21 0 0 1
173 2007-06-26 0 0 1
174 2007-08-27 0 0 1
184 2007-09-17 0 0 1
195 2015-05-11 0 0 1
198 2015-04-28 0 0 1
200 2015-10-27 0 0 1
201 2015-08-03 0 0 1
202 2015-04-13 0 0 1
204 2015-06-23 0 0 1
205 2015-09-22 0 0 1
207 2015-08-25 0 0 1
215 2015-11-04 0 0 1
258 2006-06-21 0 0 1
259 2006-09-27 0 0 1
260 2006-07-17 0 0 1
262 2006-08-23 0 0 1
264 2006-09-05 0 0 1
265 2006-06-07 0 0 1
266 2006-04-19 0 0 1
267 2006-05-10 0 0 1
270 2006-08-02 0 0 1
272 2009-10-07 0 0 1
275 2014-10-28 0 0 1
283 2014-05-27 0 0 1
284 2014-09-30 0 0 1
285 2014-09-08 0 0 1
289 2014-11-25 0 0 1
Warning message:
In compressData(data, interactive = interactive) :
Deleted 101 rows of data because concentration was reported as 0.0, the program is unable to interpret that result and is therefore deleting it.

Thank You

package or namespace load failed for ‘EGRET’

I'm having the following issue when loading EGRET. Something wrong related to dplyr?

> library(EGRET)
Error: package or namespace load failed for ‘EGRET’:
 object ‘rbind_all’ is not exported by 'namespace:dplyr'

> devtools::session_info()
Session info -----------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.4.0 (2017-04-21)
 system   x86_64, mingw32             
 ui       RStudio (1.0.143)           
 language (EN)                        
 collate  English_United States.1252  
 tz       America/Los_Angeles         
 date     2017-05-08                  

Packages ---------------------------------------------------------------------------------------------------------------------
 package    * version    date       source                           
 assertthat   0.2.0      2017-04-11 CRAN (R 3.4.0)                   
 curl         2.6        2017-04-27 CRAN (R 3.4.0)                   
 devtools     1.12.0     2016-12-05 CRAN (R 3.4.0)                   
 digest       0.6.12     2017-01-27 CRAN (R 3.4.0)                   
 dplyr        0.5.0.9004 2017-05-01 Github (hadley/dplyr@106070a)    
 glue         1.0.0      2017-05-01 Github (tidyverse/glue@650d0d3)  
 lubridate    1.6.0      2016-09-13 CRAN (R 3.3.1)                   
 magrittr     1.5        2014-11-22 CRAN (R 3.4.0)                   
 memoise      1.1.0      2017-04-21 CRAN (R 3.4.0)                   
 R6           2.2.0      2016-10-05 CRAN (R 3.4.0)                   
 Rcpp         0.12.10.2  2017-05-01 Github (RcppCore/Rcpp@c57b754)   
 rlang        0.0.0.9018 2017-05-01 Github (hadley/rlang@b22ce1c)    
 stringi      1.1.5      2017-04-07 CRAN (R 3.4.0)                   
 stringr      1.2.0      2017-02-18 CRAN (R 3.4.0)                   
 tibble       1.3.0.9001 2017-04-24 Github (tidyverse/tibble@08af6b0)
 withr        1.0.2      2016-06-20 CRAN (R 3.4.0) 

Thanks!

Tung

Problem with flow history calculations when there are missing years

Laura: I found a bug, fairly serious, in the makeAnnualSeries function that has consequences for a variety of other functions like plotFlowSingle, printSeries, plotFourStats and plotFour. I worked at it and found that I was able to write a fix for it (all the changes are in the loop near the bottom that goes from istat in 1:8. I've attached my code, but please look at it and also test it yourself. A good site to test it on is 11336580 (has a lot of missing data). Probably should be tested on one with no missing data (my changes should be of no consequence there.

So, this needs to go up on CRAN and GRAN fairly soon. It is a bad bug. You can see what I mean when you run it through plotFourStats without fixing makeAnnualSeries.

Thanks.

Bob

I'll send the code as an R file by email. Funny that they don't let me attach an R file in GitHub.

makeAnnualSeriesFixed.txt

Enhance plotQTimeDaily start/end points

From @rmhirsch49 :
One problem I've encountered in using plotQTimeDaily is that it isn't clear where the discharge record actually starts and ends. Take this one as an example.

image

The record actually starts at January 1, 1935 and ends at December 31, 2014. But people wouldn't know that from looking at the graph. We want to use this to make a point about what is happening to the frequency of high flow events and at first glance it suggests a big gap of events at the end. But the question is, can we make it an option to have the left and right edges of the graph be specified exactly by the user. Right now the two ends of the axis go beyond endYear and startYear. Perhaps an additional argument in the call to this function could be a logical variable that if set to TRUE will put the ends exactly at endYear and startYear (but if FALSE or if endYear and startYear are at the defaults of NA then it would work just like it does now). Then, there would need to be a fix to genericEGRETDotPlot to accomodate this. If we do it for plotQTimeDaily then we could probably do it for all the graphs that have time on the x axis (but not the plotContour functions).

No big rush on this, but I think it would be a nice enhancement.

Making simple maps of sites

It would be nice to have an ability to draw a quick map of the location of a site that is contained in eList$INFO using the latitude and longitude. Even nicer would be an ability to draw the watershed boundary shape file for that site as well. I know how to do the first, but we need to work on the second one. I plan to start up a vignette that will walk people through it.

Wording of a message in populateDaily

This seems like a small issue, but it is just a choice of words that means something very specific to USGS hydrologists. There is a message that populateDaily generates that says ""discharge measurements are not reported (NA's). \nMany of the EGRET functions will not work with missing discharge measurements." The word "measurements" should be replaced with the word "values" -- The reason is that in our lingo, the word "measurements" means the calibration measurements that are made at the streamgage -- there may be only a dozen or couple of dozen per year. What is delivered by the NWIS daily values service is the daily discharge value that is computed from the stage data. It doesn't deliver what we call "measurements".

Not a big deal, just something to fix the next time you make other changes to EGRET. Thanks. Bob

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.