gher-uliege / diva-workshops Goto Github PK

Code, data and instructions for the Diva workshops

Home Page: https://gher-uliege.github.io/Diva-Workshops/

Jupyter Notebook 97.41% Julia 0.12% HTML 2.44% Shell 0.03%

diva spatial-analysis data-analysis interpolation oceanography ocean-sciences seadatanet seadatacloud

diva-workshops's Introduction

DIVA Workshops and training

This repository provides a set of Jupyter notebooks (examples and exercises) for the DIVAnd user workshops and training sessions organised in the frame of H2020 SeaDataCloud project. The notebooks are also used in the FAIR-EASE project.

DIVA and DIVAnd are software tools designed to generate gridded fields from in-situ observations.

Workshops

Event	Location	Dates
1st workshop	Liège 🇧🇪	3-6 April 2018
2nd SeaDataCloud training course	Ostend 🇧🇪	19-26 June 2019
2nd workshop	Bologna 🇮🇹	27-30 January 2020

Installation

Jupyter

Jupyter has to be installed in order to have a notebook interface.
It can be installed and launched (in Julia) with the following commands

using Pkg
Pkg.add("IJulia")
using IJulia
notebook()

Extensions [optional]

It is also recommended to install the following modules which allow, for example, to have the sections automatically numbered:

Other relevant repositories

EMODnet-Chemistry-GriddedMaps

This repository aims to store the notebooks and the instructions to produce the EMODnet Chemistry products (climatologies).

Binder

Most notebooks need more resources that what is can currently available on Binder. The introduction notebooks (introduction to OI and variationa analysis) however work .

diva-workshops's People

Contributors

Stargazers

Watchers

Forkers

swatelet malek-belgacem msdevana celinednl stanleesocca yanchunhefork thomas-moore-creative mr-nobody-dey marie59

diva-workshops's Issues

Notebook clean-up

Are there any notebooks which are no longer be used (replaced e.g. by other functionalities)? If yes, I would propose to delete them.

Errors in the notebook 14b-errormaps-demo

The CI reported an error in notebook 14b-errormaps-demo so I tried to run it and got some issues:

In cell [15] (as per GitHub):

ploterr(aerrora)
title("Error using errormap version $method");

gives an error

PyError ($(Expr(:escape, :(ccall(#= /home/ctroupin/.julia/packages/PyCall/ygXW2/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'ValueError'>
ValueError('too many values to unpack (expected 2)')

In cell [16]:

aerror,method = DIVAnd_errormap(mask,(pm,pn),(xi,yi),(obslon,obslat),obsval.-mean(obsval),len,epsilon2,
    s;
    method = :precise,
    Bscale = false)
ploterr(aerror)
title("Error using automatic version $method");

gives the same error. In fact when I try to access the variable aerror I get this:

julia> aerror
'Y': ASCII/Unicode U+0059 (category Lu: Letter, uppercase)

Solutions

I could make it work with those changes below though I'm not sure why it failed?

Add an index [1] to the array:

ploterr(aerrora[1])
title("Error using errormap version $method");

Replace :precise by "precise" in the function call:

aerror,method = DIVAnd_errormap(mask,(pm,pn),(xi,yi),(obslon,obslat),obsval.-mean(obsval),len,epsilon2,
    s;
    method = "precise",
    Bscale = false)

Figures for readme

Horizontal correlation length fitting

In the previous version we were using

fithorzlen(x,v,z, nmean = 500, distbin = collect(0.:0.1:6))

to estimate the correlation length but the optional arguments nmean and distbin are not available anymore in fit.jl, so what would be the equivalent call to fithorzlen?

Or we leave it as:

fithorzlen(x,v,z)

Same for

lenz,infoz = fitvertlen(x,v,z, nmean = 500, distbin = collect([0.:50:400; 500:100:600]))

InitError: KeyError: key "get_cmap" not found during initialization of module PyPlot in CI

In the CI workflow

InitError: KeyError: key "get_cmap" not found
during initialization of module PyPlot
``

Removing notebook 05-DIVAnd-overview.ipynb

I propose we get rid of notebook 05-DIVAnd-overview.ipynb because it doesn't really need anything on the table (part of the content could be added to the README if needed).

I know this will cause a problem in the numbering of the notebooks (no number 5 anymore), but the numbering is already a bit messed up in some of the directories).

notebook 15-example-analysis: ArgumentError: invalid base 10 digit 'C'

Julia version: 1.4.0
DIVAnd version: [efc8151c] DIVAnd v2.5.1

I ran the notebooks successfully, until the last command:

divadoxml(filename,varname,project,cdilist,xmlfilename,ignore_errors = ignore_errors)

which returns this error:

ArgumentError: invalid base 10 digit 'C' in "Cruise:WOD05_BG000003"

Stacktrace:
 [1] tryparse_internal(::Type{Int64}, ::SubString{String}, ::Int64, ::Int64, ::Int64, ::Bool) at ./parse.jl:132
 [2] parse(::Type{Int64}, ::SubString{String}; base::Nothing) at ./parse.jl:238
 [3] parse at ./parse.jl:238 [inlined]
 [4] get_originators_from_obsid(::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Base.Iterators.Flatten{Array{Array{String,1},1}}; ignore_errors::Bool) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:328
 [5] getoriginators(::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String; ignore_errors::Bool) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:300
 [6] gettemplatevars(::Array{String,1}, ::String, ::String, ::String; errname::String, WMSlayername::Array{String,1}, previewindex::Int64, basemap::String, additionalcontacts::Array{Any,1}, ignore_errors::Bool) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:827
 [7] divadoxml(::Array{String,1}, ::String, ::String, ::String, ::String; ignore_errors::Bool, previewindex::Int64, basemap::String, additionalvars::Dict{String,Any}, additionalcontacts::Array{Any,1}, WMSlayername::Array{String,1}) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:1015
 [8] divadoxml(::String, ::String, ::String, ::String, ::String; WMSlayername::String, kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:ignore_errors,),Tuple{Bool}}}) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:1044
 [9] top-level scope at In[34]:1

Writing a huge number of observations to a netCDF

So I'm trying to write all the Atlantic observations inside the netCDF I've created with DIVAnd:

@show length(obslat);
length(obslat) = 31850848
DIVAnd.saveobs(filename,(obslon,obslat,obsdepth,obstime),obsid);

and the last command results in my browser to be killed.

Probably this part has to be somehow enhanced.

Use notebook extensions

Add to the doc/readme how to use the extensions:

https://github.com/ipython-contrib/jupyter_contrib_nbextensions
https://github.com/Jupyter-contrib/jupyter_nbextensions_configurator

so (for ex.) sections are automatically numbered

File download failure

The CI is failing (in all O.S.), at least one reason is that the file download (for example bathymetry) cannot be completed.

Origin of the problem

I guess it's because DoX URL:

This works:

julia> download("https://dox.uliege.be/index.php/s/k0f7FxA7l5FIgu9/download")
"/tmp/jl_onMfMLiNcN"

This fails:

julia> download("https://dox.ulg.ac.be/index.php/s/k0f7FxA7l5FIgu9/download")
ERROR: RequestError: Could not resolve host: dox.ulg.ac.be while requesting https://dox.ulg.ac.be/index.php/s/k0f7FxA7l5FIgu9/download
Stacktrace:
...

Getting rid of data files?

Most of the datafiles are stored in dox, it makes sense not to store datafiles, especially the netCDF, within github.

I'm checking if all of them are there.

"conversion to pointer not defined for StepRangeLen..." in notebook "90-full-analysis".

The cell

@time dbinfo = diva3d((lonr,latr,depthr,TS),
    (obslon,obslat,obsdepth,obstime), obsval,
    len, epsilon2,
    filename,varname,
    bathname=bathname,
    plotres = plotres,
    mask = mask_edit,
    fitcorrlen = false,
    niter_e = 2,
    ncvarattrib = ncvarattrib,
    ncglobalattrib = ncglobalattrib,
    surfextend = true
    );

yields this issue:

conversion to pointer not defined for StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}

Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] unsafe_convert(#unused#::Type{Ptr{Float64}}, a::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64})
    @ Base ./pointer.jl:67
  [3] unsafe_convert(#unused#::Type{Ptr{Nothing}}, a::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64})
    @ Base ./pointer.jl:66
  [4] nc_put_vara(ncid::Int32, varid::Int32, startp::Vector{Float64}, countp::Vector{Int64}, op::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/netcdf_c.jl:796
  [5] nc_put_var(ncid::Int32, varid::Int32, data::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/netcdf_c.jl:712
  [6] setindex!
    @ ~/.julia/packages/NCDatasets/c8XyT/src/variable.jl:342 [inlined]
  [7] setindex!(v::NCDatasets.CFVariable{Float64, 1, NCDatasets.Variable{Float64, 1, NCDataset{Nothing}}, NCDatasets.Attributes{NCDataset{Nothing}}, NamedTuple{(:fillvalue, :scale_factor, :add_offset, :calendar, :time_origin, :time_factor), NTuple{6, Nothing}}}, data::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, indexes::Colon)
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/cfvariable.jl:632
  [8] ncfile(ds::NCDataset{Nothing}, filename::String, xyi::Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, Vector{Float64}, Vector{DateTime}}, varname::String; ncvarattrib::OrderedDict{String, String}, ncglobalattrib::OrderedDict{String, String}, thresholds::Vector{Tuple{String, Float64}}, deflatelevel::Int64, chunksizes::Vector{Int64}, type_save::Type{Float32}, timeorigin::DateTime, checksum::Symbol, saveindex::Tuple{Colon, Colon, UnitRange{Int64}}, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:climatology_bounds, :relerr), Tuple{Matrix{DateTime}, Bool}}})
    @ DIVAnd ~/ULiege/Tools/DIVAnd.jl/src/DIVAnd_save.jl:322
  [9] (::DIVAnd.var"#447#463"{Vector{Float64}, OrderedDict{String, String}, OrderedDict{String, String}, typeof(DIVAnd.distfun_m), Nothing, Int64, Bool, Bool, Dict{Any, Any}, Dict{Any, Any}, Int64, Float64, Int64, Float64, Float64, Tuple{}, Bool, Vector{Tuple{String, Float64}}, typeof(DIVAndgo), Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, Vector{Float64}, String, String, Bool, Matrix{DateTime}, Dict{Symbol, Any}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Vector{Float64}, Int64, Tuple{Int64, Int64, Int64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}, DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, Vector{DateTime}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{Vector{Int64}}}, Vector{Float64}, Int64})(ds::NCDataset{Nothing})
    @ DIVAnd ~/ULiege/Tools/DIVAnd.jl/src/diva.jl:376
 [10] NCDataset(::DIVAnd.var"#447#463"{Vector{Float64}, OrderedDict{String, String}, OrderedDict{String, String}, typeof(DIVAnd.distfun_m), Nothing, Int64, Bool, Bool, Dict{Any, Any}, Dict{Any, Any}, Int64, Float64, Int64, Float64, Float64, Tuple{}, Bool, Vector{Tuple{String, Float64}}, typeof(DIVAndgo), Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, Vector{Float64}, String, String, Bool, Matrix{DateTime}, Dict{Symbol, Any}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Vector{Float64}, Int64, Tuple{Int64, Int64, Int64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}, DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, Vector{DateTime}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{Vector{Int64}}}, Vector{Float64}, Int64}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/dataset.jl:220
 [11] NCDataset(::Function, ::String, ::Vararg{String})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/dataset.jl:218
 [12] diva3d(xi::Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, Vector{Float64}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{Vector{Int64}}}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, value::Vector{Float64}, len::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Float64, filename::String, varname::String; datadir::String, bathname::String, bathisglobal::Bool, plotres::typeof(plotres), timeorigin::DateTime, moddim::Vector{Float64}, zlevel::Symbol, ncvarattrib::OrderedDict{String, String}, ncglobalattrib::OrderedDict{String, String}, transform::Tuple{DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}}, distfun::typeof(DIVAnd.distfun_m), mask::BitArray{3}, background::Nothing, background_epsilon2_factor::Nothing, background_lenz::Nothing, background_len::Nothing, background_lenz_factor::Int64, filterbackground::Int64, fitcorrlen::Bool, fithorzcorrlen::Bool, fitvertcorrlen::Bool, fithorz_param::Dict{Any, Any}, fitvert_param::Dict{Any, Any}, memtofit::Int64, overlapfactor::Float64, niter_e::Int64, minfield::Float64, maxfield::Float64, surfextend::Bool, velocity::Tuple{}, stat_per_timeslice::Bool, error_thresholds::Vector{Tuple{String, Float64}}, divamethod::typeof(DIVAndgo), kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ DIVAnd ~/ULiege/Tools/DIVAnd.jl/src/diva.jl:375
 [13] top-level scope
    @ ./timing.jl:220 [inlined]
 [14] top-level scope
    @ ./In[41]:0
 [15] eval
    @ ./boot.jl:373 [inlined]
 [16] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1196

Julia Version 1.7.0
Commit 3bf9d17731 (2021-11-30 12:12 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4

PosDefException in 13-processing-parameter-optimization

With imeth = 1,
this

bestfactorl[imeth+1],bestfactore[imeth+1], cvval,cvvalues, x2Ddata,y2Ddata,cvinter,xi2D,yi2D = DIVAnd_cv(mask,(pm,pn),(xi,yi),(obslon,obslat),obsval.-mean(obsval),len,epsilon2*rdiag,2,3,imeth);

gives a PosDefException error (see below).
Any hint?

PosDefException: matrix is not positive definite; Cholesky factorization failed.

Stacktrace:
 [1] #cholesky!#6 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SuiteSparse/src/cholmod.jl:1284 [inlined]
 [2] #cholesky! at ./none:0 [inlined]
 [3] #cholesky#8(::Float64, ::Bool, ::Nothing, ::typeof(LinearAlgebra.cholesky), ::SuiteSparse.CHOLMOD.Sparse{Float64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SuiteSparse/src/cholmod.jl:1322
 [4] #cholesky#9 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SuiteSparse/src/cholmod.jl:1315 [inlined]
 [5] cholesky at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SuiteSparse/src/cholmod.jl:1434 [inlined]
 [6] factorize!(::CovarIS{Float64,SparseArrays.SparseMatrixCSC{Float64,Int64}}) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/special_matrices.jl:94
 [7] DIVAnd_factorize!(::DIVAnd.DIVAnd_struct{Float64,Int64,2,SparseArrays.SparseMatrixCSC{Float64,Int64}}) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/DIVAnd_factorize.jl:37
 [8] #DIVAndrun#267(::Tuple{}, ::Bool, ::Bool, ::Float64, ::Int64, ::Int64, ::Tuple{}, ::Symbol, ::Array{Any,1}, ::Array{Float64,2}, ::Array{Int64,1}, ::Int64, ::typeof(DIVAnd_pc_none), ::DIVAnd.var"#269#271", ::Array{Float64,2}, ::Array{Float64,1}, ::Int64, ::Bool, ::Array{Any,1}, ::Float64, ::Tuple{}, ::Tuple{}, ::Int64, ::Tuple{}, ::Tuple{}, ::Array{Float64,1}, ::Array{Float64,1}, ::typeof(DIVAndrun), ::Type, ::BitArray{2}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Tuple{Float64,Float64}, ::Array{Float64,1}) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/DIVAndrun.jl:97
 [9] (::DIVAnd.var"#kw##DIVAndrun")(::NamedTuple{(:alpha, :alphabc),Tuple{Array{Int64,1},Int64}}, ::typeof(DIVAndrun), ::BitArray{2}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Tuple{Float64,Float64}, ::Array{Float64,1}) at ./none:0
 [10] #DIVAnd_cv#308(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DIVAnd_cv), ::BitArray{2}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Int64, ::Array{Float64,1}, ::Int64, ::Int64, ::Int64) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/DIVAnd_cv.jl:310
 [11] DIVAnd_cv(::BitArray{2}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Int64, ::Array{Float64,1}, ::Int64, ::Int64, ::Int64) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/DIVAnd_cv.jl:73
 [12] top-level scope at ./In[12]:5

Analysis notebook

In the previous version, we start making analysis with notebook 10-processing-quality-check.ipynb which is about quality check.

I wonder if we should change the order, i.e., present QC based on the analysis later in the course.

Warning: "Preconditioned conjugate gradients method did not converge" in notebook 90-full-analysis

Hi,
I am using DIVAnd for the first time and I have been working on an analysis based on the notebook 90-full-analysis, I got the following warning in output:

@time dbinfo = diva3d((lonr,latr,depthr,TS), (obslon,obslat,obsdepth,obstime), obsval, len, epsilon2, filename2,varname, bathname=bath_name, mask = mask, fitcorrlen = false, niter_e = 2, );

Output:
[ Info: Creating netCDF file /~/data/julia/fromjulia/gam_sal_processed.nc
[ Info: Time step 1 / 4
[ Info: scaled correlation length (min,max) in dimension 1: (100000.0, 100000.0)
[ Info: scaled correlation length (min,max) in dimension 2: (100000.0, 100000.0)
[ Info: scaled correlation length (min,max) in dimension 3: (25.0, 25.0)
[ Info: number of windows: 6
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
[ Info: number of windows: 6
[ Info: Time step 2 / 4
[ Info: scaled correlation length (min,max) in dimension 1: (100000.0, 100000.0)
[ Info: scaled correlation length (min,max) in dimension 2: (100000.0, 100000.0)
[ Info: scaled correlation length (min,max) in dimension 3: (25.0, 25.0)
[ Info: number of windows: 6
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
[ Info: number of windows: 6
[ Info: Time step 3 / 4
┌ Warning: no data at 3
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/diva.jl:432

I'd like to understand the reason and how to solve this warning.
The resulting netcdf file looks good but I'd prefer to correct this before going further.

I hope you can help me to better understand how it works.

Cheers,
Javiera

Remove NaN values

Hi !

After I make an analysis, I got some warnings in output:
dbinfo = diva3d((lonr,latr,depthr,TS), (lon,lat,depth,obstime), value, (lenx,leny,lenz2), 0.1 * eps1 ./ diagR, filename,varname, bathname = bathname, bathisglobal = bathisglobal, plotres = plotres2, timeorigin = timeorigin, transform = Anam.loglin(slog), minfield = 0., memtofit = 100, solver = :direct, ) # Add used observations in the netcfd file DIVAnd.saveobs(filename,(lon,lat,depth,obstime),ids,used = dbinfo[:used])

Output:
┌ Info: starting DIVA computations for Autumn
└ @ Main In[20]:12
┌ Info: 2020-01-29T11:35:36.05
└ @ Main In[20]:13
┌ Info: Will write results in /home/george/my_project_dir/DIN_Sel5_LCH_LCV_propuse_Autumn//Water_body_dissolved_inorganic_nitrogen_(DIN)_Autumn.4Danl.nc
└ @ Main In[20]:34
┌ Info: Creating netCDF file
└ @ DIVAnd /home/george/.julia/packages/DIVAnd/BGj1J/src/diva.jl:297
┌ Info: Time step 1 / 4
└ @ DIVAnd /home/george/.julia/packages/DIVAnd/BGj1J/src/diva.jl:333
┌ Warning: Observations equal to NaN: 3
└ @ DIVAnd /home/george/.julia/packages/DIVAnd/BGj1J/src/DIVAnd_obs.jl:45
┌ Info: number of windows: 11
└ @ DIVAnd /home/george/.julia/packages/DIVAnd/BGj1J/src/DIVAndgo.jl:89

So, I was wondering, if it's possible to remove all NaN values from dbinfo ( which is a dictionary) ?
Because, the resulting netcdf file, it's almost full with NaN values.

Cheers !
George

fithorzlen fails in 12-correlation-length

@time lenxy,infoxy = fithorzlen(x,v,z; distfun = DIVAnd.distfun_m, searchz = z -> (z/4+10));

gives

TypeError: in keyword argument searchz, expected Float64, got var"#5#6"

Stacktrace:
 [1] (::DIVAnd.var"#kw##fithorzlen")(::NamedTuple{(:distfun, :searchz),Tuple{typeof(DIVAnd.distfun_m),var"#5#6"}}, ::typeof(fithorzlen), ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Array{Float64,1}) at ./none:0
 [2] top-level scope at util.jl:155
 [3] top-level scope at In[11]:1

Broken links to download data

Some notebooks (or at least 20-Turtles-Density-map.ipynb) still contains URLs related to B2drop, which don't seem to work anymore.

Solution: replace with DoX link (easy)

Future: let's hope DoX will be maintained because it is used now in several of our projects to stored:

data files (so we can download them within a Julia session)
small files to perform unit tests

Integrate 5-AdvancedTopics/time-aggregation.ipynb into 18-defining-time-periods.ipynb

These notebooks are quite similar:

https://github.com/gher-ulg/Diva-Workshops/blob/master/notebooks/5-AdvancedTopics/18-defining-time-periods.ipynb

https://github.com/gher-ulg/Diva-Workshops/blob/master/notebooks/5-AdvancedTopics/time-aggregation.ipynb

If I am not mistaken, @ctroupin will discuss them on Monday PM. Would it be useful to combine them?

Alternative grid for interpolation (non lat-lon)

Hi all,

I am working with datasets in polar regions and would like to interpolate to a grid which is different from the lat-lon grid. I see that in the notebook the grid coordinates are defined as
lonr = lon_beg:dx:lon_end
Is there a way, instead, to give in input the grid points in another coordinate system? The locations would always be expressed in lat and lon, but they would not be organised in a regular lat-lon grid. It would be something similar to interpolate to scattered points.

Thank you very much in advance for your help.
Francesca

TEST ISSUE :) Cannot open the notebooks with mac OS

Ideally:

configuration: versions of the tools, OS
packages
copy of the message

"Kernelspec name julia-1.100 cannot be found!" during CI

During the CI workflow the code cannot be run:

...
Available kernels:
  julia-1.10    /home/runner/.local/share/jupyter/kernels/julia-1.10
  python3       /home/runner/.local/share/jupyter/kernels/python3
JULIA_VERSION 1.10.4 1.10
/home/runner/work/Diva-Workshops/Diva-Workshops/utils/../notebooks/1-Intro/01-notebooks-basics.ipynb 1.10.4
[NbConvertApp] Converting notebook /home/runner/work/Diva-Workshops/Diva-Workshops/utils/../notebooks/1-Intro/01-notebooks-basics.ipynb to html
[NbConvertApp] WARNING | Kernelspec name julia-1.100 cannot be found!
[NbConvertApp] ERROR | No such kernel named julia-1.100
...

Problem to write the observations to the netCDF

Hello,

in 90-full-analysis, the command to write the obs DIVAnd.saveobs(filename,(obslon,obslat,obsdepth,obstime),obsid); returns the following error.

The checkobs((obslon,obslat,obsdepth,obstime),obsval,obsid); seems to provide a correct output:

              minimum and maximum of obs. dimension 1: (11.716666221618652, 20.11)
              minimum and maximum of obs. dimension 2: (39.0, 45.755)
              minimum and maximum of obs. dimension 3: (0.0, 2508.516357421875)
              minimum and maximum of obs. dimension 4: (1911-08-17T09:24:00, 2015-02-10T10:43:00)
                          minimum and maximum of data: (25.0, 39.700001)

Any hint?

Julia Version 1.0.0 (2018-08-08)
DIVAnd v2.1.1+ [~/.julia/dev/DIVAnd]
NCDatasets v0.5.1+ [~/.julia/dev/NCDatasets]

BoundsError: attempt to access 1805275-element Array{Float64,1} at index [Base.LogicalIndex(Bool[true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, 
......
true, true, true, true, true, true, true, true])]

Stacktrace:
 [1] throw_boundserror(::Array{Float64,1}, ::Tuple{Base.LogicalIndex{Int64,BitArray{1}}}) at ./abstractarray.jl:484
 [2] checkbounds at ./abstractarray.jl:449 [inlined]
 [3] _getindex at ./multidimensional.jl:588 [inlined]
 [4] getindex at ./abstractarray.jl:905 [inlined]
 [5] #356 at ./none:0 [inlined]
 [6] iterate at ./generator.jl:47 [inlined]
 [7] collect(::Base.Generator{Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}},getfield(DIVAnd, Symbol("##356#357")){BitArray{1}}}) at ./array.jl:619
 [8] #saveobs#355(::Type, ::DateTime, ::BitArray{1}, ::Symbol, ::Function, ::String, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{String,1}) at /home/ctroupin/.julia/dev/DIVAnd/src/DIVAnd_save.jl:389
 [9] saveobs(::String, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{String,1}) at /home/ctroupin/.julia/dev/DIVAnd/src/DIVAnd_save.jl:387
 [10] top-level scope at In[38]:1