02_Preliminary_analysis won't knit about mungbean_pm HOT 57 CLOSED

yes, a few

from mungbean_pm.

I am able to knit the whole book.
I pushed some changes that might have fixed it

from mungbean_pm.

from mungbean_pm.

Which branch are you on?
I don't have any issues on the master branch

from mungbean_pm.

I am/was on master when this issue occurred.

from mungbean_pm.

hmmm
I don't have any chunks names with bwplot in 01_TrialsForInclusionInAnalysis.Rmd
in 02_preliminary analysis.rmd there are 4 chunks with names starting with bwplot.

bwplot_yield
bwplot_sev
bwplot_yield_year &
bwplot_yield_sched

no other files contain the string bwplot

from mungbean_pm.

This was not 01_, this was 02_ where it barfed.

Now it hangs at ~74% fetching BoM data.

from mungbean_pm.

that may just be my flakey connection here in South Perth tho

from mungbean_pm.

It should not need to fetch the data from bom if the data already exists in the weather/ directory, which is contains the predownloaded data.

I am wondering if this is a problem between mac and windows.
does the file.exists() function work on macs?

from mungbean_pm.

Yes, that's a basic R function.

from mungbean_pm.

I ran the chunks up to this point then ran the code in this chunk step by step.

from mungbean_pm.

Have you made your cup of coffee yet ;)

from mungbean_pm.

Just wondering if you have a folder in the project directory weather/

from mungbean_pm.

yes

from mungbean_pm.

from mungbean_pm.

I am wondering if this might have something to do with the csv files starting with a - sign

from mungbean_pm.

I was able to run all the chunks including the season_rainfall chunk.
Is the macOS run on a UNIX shell ... ?

from mungbean_pm.

macOS is UNIX-like.

I might be able to run the chunk, will check again but the connection is poor here. However, I thought you said it was cached so it didn't run. If those CSV files (pictured) are the proper ones, why am I waiting on it to download the data again?

from mungbean_pm.

Yes have a look at the crop_rain function it checks to see if the files exist before downloading the data

from mungbean_pm.

I might get it to edit out any - from the lat long
I suspected it might cause issues when I wrote it

from mungbean_pm.

I have pushed the updated code so filenames are no longer saved with a -minus
Did you want to try rerunning the code?

from mungbean_pm.

I left and went to get ☕️, came back, it was still running .5hr later.

I pulled your new code just now. It's still trying to fetch the data...

from mungbean_pm.

Does the add_lat_longs.R function add the coordinates to columns in the PM_MB_dat data.frame?
It might be that it can't find the correct lat and longs in the data

Otherwise it might be your connection, can you run this code?
get_historical(stationid = "041359",type = "rain")

from mungbean_pm.

I know there's some issues with BOM servers timing out, etc. right now from Perth. I worked on it over the weekend, but that's not the issue here.

The issue as I see it, is that the code does not work properly. I shouldn't even be running into the bomrang/BOM issue if the data were properly cached. It seems that they aren't.

from mungbean_pm.

So were the columns lat and lon added to the data.frame?
Perhaps it is loading the wrong lat and longs and therefore not finding the cached files?

from mungbean_pm.

> head(PM_MB_dat)
    trial_ref year  location host_genotype row_spacing
1 mung1011/01 2011 Hermitage        Berken        0.75
2 mung1011/01 2011 Hermitage        Berken        0.75
3 mung1011/01 2011 Hermitage       Crystal        0.75
4 mung1011/01 2011 Hermitage       Crystal        0.75
5 mung1011/02 2011  Kingaroy        Berken        0.75
6 mung1011/02 2011  Kingaroy        Berken        0.75
  replicates planting_date first_sign_disease fungicide_ai
1          6    2011-01-24         2011-03-28      control
2          3    2011-01-24         2011-03-28 tebuconazole
3          6    2011-01-24         2011-03-28      control
4          3    2011-01-24         2011-03-28 tebuconazole
5          3    2011-02-02         2011-03-22      control
6          3    2011-02-02         2011-03-22 tebuconazole
  dose_ai.ha total_fungicide harvest_date final_assessment
1       0.00               0   2011-04-25       2011-04-11
2      62.35               1   2011-04-25       2011-04-11
3       0.00               0   2011-04-25       2011-04-11
4      62.35               1   2011-04-25       2011-04-11
5       0.00               0         <NA>       2011-05-11
6      62.35               1         <NA>       2011-05-11
  PM_final_severity disease_error D_error_type
1             4.667        0.5164        stdev
2             2.000        0.0000        stdev
3             5.000        0.0000        stdev
4             2.333        0.5774        stdev
5             8.000        0.0000        stdev
6             5.333        1.1547        stdev
  grain_yield.t.ha. yield_error Y_error_type spray_management
1            1.6619     0.11732        stdev          control
2            1.7693     0.19442        stdev      Recommended
3            1.3876     0.12447        stdev          control
4            1.5439     0.12833        stdev      Recommended
5            0.7985     0.15104        stdev          control
6            1.0586     0.08244        stdev      Recommended
     lat   lon
1 -28.21 152.1
2 -28.21 152.1
3 -28.21 152.1
4 -28.21 152.1
5 -26.58 151.8
6 -26.58 151.8

from mungbean_pm.

Perhaps it is downloading again because of the number of digits in the floating point number.
I'll specify that to in the function

from mungbean_pm.

where does it get the values from? that shouldn't happen.

from mungbean_pm.

See function add_lat_longs.R

# read in site location data, including lats and longs
experiment_sites <-
   read.csv("cache/Mungbean_experiment_sites.csv", stringsAsFactors = FALSE)

If your computer is downloading the data it must be returning false to the following if() statement

crop_rain <- function(latitude, longitude, start, end) {

   LL <- c(latitude, longitude)
   
      if (file.exists(paste("weather/", str_remove(paste(LL, collapse = ""),"-"), ".csv", sep = "")))

Does this line of code return TRUE or FALSE for you?
file.exists(paste("weather/28.21500152.1003.csv"))

from mungbean_pm.

> file.exists(paste("weather/28.21500152.1003.csv"))
[1] TRUE

from mungbean_pm.

I pushed some changes to format the numbers to a consistent length before being saved as file names.

crop_rain <- function(latitude, longitude, start, end) {
   # locations <- unique(data.frame(lat = latitude,
   #                                lon = longitude))
   
   LL <- format(round(as.numeric(c(latitude, longitude)),4), nsmall = 4)

Do you want to try it again?

from mungbean_pm.

Still fetching the data unless it just takes forever to import the cached data from the package?

from mungbean_pm.

It should take less than a minute.

from mungbean_pm.

yeah, nah. It's not importing the files. It's going to BOM or it's hanging importing files.

from mungbean_pm.

I am out of ideas, and not sure where to go from here given I can't reproduce it.

from mungbean_pm.

I should be able to pick up these files and knit them if it's truly reproducible. It's not.

from mungbean_pm.

I can't help anymore without more information.
Perhaps we could organise a time to video chat and get to the bottom of this.

from mungbean_pm.

I'm spending more time on this than I should be, but one thing that I see is that you download weather for every single line of the data.frame. Why? Just download for one lat/lon value once. Rather than fetching 161x, it should be only 15x.

After that's done, I'm still trying to figure out why it doesn't load the cached data.

from mungbean_pm.

If it loads the cached data it should only download 7 times, one for each weather station that is most proximal to the trial locations

from mungbean_pm.

But if you are able to figure out why the if statement is returning FALSE on your machine that will solve the problem.

if (file.exists(paste("weather/", str_remove(paste(LL, collapse = ""),"-"), ".csv", sep = "")))

from mungbean_pm.

Your apply statement runs over every line of the data.frame. That's 161 lines that it runs over. I don't see how that's only 7 times?

from mungbean_pm.

because the file name does not exist...

from mungbean_pm.

Yes the apply function does run over every line of the data.frame. Not the most efficient, I know.
but for each line it checks if there is a weather csv for the weather station that is closest to the trial. If the csv exists it will read it in with read.csv if not it downloads it from the web stores the data with file name of the lat and long of the station.
given there are multiple sites with multiple years it makes sense to only download it the minimum number of times
once per station closest to the trial (in some cases two sites might use the same station data).

from mungbean_pm.

> LL <- PM_MB_dat[1, 21:22]
> paste("weather/", str_remove(paste(LL, collapse = ""),"-"), ".csv", sep = "")
[1] "weather/28.215152.1003.csv"

28.2150152.1003.csv does exist but not 28.215152.1003.csv, which is what it's looking for

from mungbean_pm.

Yes, but it's much quicker/efficient to only look for the 15 unique lat/lon values to begin with.

from mungbean_pm.

I don't think the apply function is the problem here

from mungbean_pm.

I didn't look at any other lat/lon values.

from mungbean_pm.

I don't think so either. I'm just pointing out that I've found issues starting with the apply() function in how this was coded. I was going back to the beginning of where the issue starts. It starts there. That may not be what's causing the issue, but it is still inefficient and an issue that I've found.

from mungbean_pm.

Try

LL <- format(round(PM_MB_dat[1, 21:22],4,nsmall = 4))
paste("weather/", str_remove(paste(LL, collapse = ""),"-"), ".csv", sep = "")

from mungbean_pm.

> LL <- format(round(PM_MB_dat[1, 21:22],4,nsmall = 4))
Error in h(simpleError(msg, call)) : 
  error in evaluating the argument 'x' in selecting a method for function 'format': 3 arguments passed to 'round' which requires 1 or 2 arguments
> paste("weather/", str_remove(paste(LL, collapse = ""),"-"), ".csv", sep = "")
[1] "weather/28.215152.1003.csv"

from mungbean_pm.

arg, my bad, parenthesis in the wrong spot

LL <- format(round(PM_MB_dat[1, 21:22],4),nsmall = 4)
paste("weather/", str_remove(paste(LL, collapse = ""),"-"), ".csv", sep = "")

from mungbean_pm.

> LL <- format(round(PM_MB_dat[1, 21:22],4),nsmall = 4)
> paste("weather/", str_remove(paste(LL, collapse = ""),"-"), ".csv", sep = "")
[1] "weather/28.2150152.1003.csv"

I'd not be naming files in such a manner that it depends on rounding...

from mungbean_pm.

I know its horrible code, but I have been pushed for time.
For sure there is a better way to do it

from mungbean_pm.

Hi Adam,
I adapted the crop_rain code so it will not download the same station data more than once if file.exists() fails.

That said do you want to test the code to see if the amended function works on your computer?
If the file.exists() function still fails as it was before I may try a different work around like...

file %in% list.files(file.path) 

from mungbean_pm.

Starting at line 633 it fails.

> source("R/add_lat_longs.R") # code adds the latitude- and longitude to PM_MB_dat
> source("R/crop_rain.R")
> # Obtain rainfall from bomrang
> PM_MB_dat$sum_rain <-
+    crop_rain(location_name = PM_MB_dat$location,
+           latitude = PM_MB_dat$lat,
+           longitude = PM_MB_dat$lon,
+           first_day = PM_MB_dat$planting_date,
+           last_day = PM_MB_dat$harvest_date)
Error: Problem with `mutate()` input `stat_file`.
x object 'stat_file' not found
ℹ Input `stat_file` is `case_when(...)`.
Run `rlang::last_error()` to see where the error occurred.

from mungbean_pm.

Ok that problem should now be fixed.
Want to try again?

from mungbean_pm.

thanks, works now

from mungbean_pm.