Comments (5)
Weiming-Hu
commented on July 28, 2024
Sorry for my late reply. While I'm check on this issue, could you run the program again with --verbose 4 and paste the log?
from analogsensemble.
Weiming-Hu
commented on July 28, 2024
I have received the input_pred2.nc file but not the input_obs2.nc file. Could you also share this file? Thanks.
from analogsensemble.
Weiming-Hu
commented on July 28, 2024
When I run the following command, I received an error message saying that the test time is not found.
> anen_netcdf --forecast-file input_pred2.nc --observation-file input_obs2.nc --out out.nc --test-start "2021-06-06 00:00:00" --test-end "2021-09-29 23:00:00" --search-start "2018-05-01 00:00:00" --search-end "2020-10-31 23:00:00" --ai-embedding embedding_epoch-00010.pt -v 4
Reading forecast file (/home/weh012/singularity_wd/input_pred2.nc) ...
entire: 1
start:
count:
Reading parameters ...
Reading stations ...
Reading times ...
Reading times ...
Updating dimensions for 37 parameters, 364 stations, 628 times, and 24 lead times ...
Reading variable Data ...
Reading observation file (/home/weh012/singularity_wd/input_obs2.nc) ...
entire: 1
start:
count:
Reading parameters ...
Reading stations ...
Reading times ...
Updating dimensions ...
Reading variable Data ...
Transforming forecasts variables to latent features with AI ...
Reading the embedding model ...
Embedding type is 1: 2-dimensional embedding [parameters, lead times]
Lead time radius for embedding: 1
Populating the tensor with 2-dimensional embeddings (parameters with lead times) ...
Feature transformation complete!36
Saving [20] latent features ...
Copying latent feature values ...
Forecast variables have been transformed to latent features!
Initialize weights to all 1s because weights in latent space do not matter!
Start AnEnIS generation ...
terminate called after throwing an instance of 'std::runtime_error'
what(): No test indices provided. Do forecasts actually cover the test period?
I noticed that your observations are from 2017/12/31 to 2021/10/12 but your forecasts are from 2018/3/31 to 2020/10/30. Apparently, you can only generate analogs for the time period that you have forecasts and search through the period when you have both forecasts and observations. I hope this makes sense.
I changed the test period to the following.
> anen_netcdf --forecast-file input_pred2.nc --observation-file input_obs2.nc --out out.nc --test-start "2020-06-06 00:00:00" --test-end "2020-09-29 23:00:00" --search-start "2018-05-01 00:00:00" --search-end "2020-06-01 23:00:00" --ai-embedding embedding_epoch-00010.pt -v 4
Reading forecast file (/home/weh012/singularity_wd/input_pred2.nc) ...
entire: 1
start:
count:
Reading parameters ...
Reading stations ...
Reading times ...
Reading times ...
Updating dimensions for 37 parameters, 364 stations, 628 times, and 24 lead times ...
Reading variable Data ...
Reading observation file (/home/weh012/singularity_wd/input_obs2.nc) ...
entire: 1
start:
count:
Reading parameters ...
Reading stations ...
Reading times ...
Updating dimensions ...
Reading variable Data ...
Transforming forecasts variables to latent features with AI ...
Reading the embedding model ...
Embedding type is 1: 2-dimensional embedding [parameters, lead times]
Lead time radius for embedding: 1
Populating the tensor with 2-dimensional embeddings (parameters with lead times) ...
Feature transformation complete!36
Saving [20] latent features ...
Copying latent feature values ...
Forecast variables have been transformed to latent features!
Initialize weights to all 1s because weights in latent space do not matter!
Start AnEnIS generation ...
Computing standard deviation ...
Allocating memory ...
************** AnEn Configuration Summary **************
verbose: 4
num_analogs: 1
num_similarity: 1
observation_id: 0
max_par_nan: 0
max_flt_nan: 0
flt_radius: 1
save_analogs: 1
save_analogs_time_index: 0
save_similarity: 0
save_similarity_time_index: 0
operation: 0
quick: 0
prevent_search_future: 1
no_norm: 0
Use AI similarity: 0
weights: 1,1,1,1,1,... [total: 20]
standard deviation array dimensions: [20,364,24,1]
observations time index table dimensions: [448,24]
analogs arary dimensions: [364,116,24,1]
*********** End of AnEn Configuration Summary **********
************** AnEn Computation Summary **************
Number of stations: 364
Number of test times: 116
Number of lead times: 24
Number of search times: 448
Number of threads to be created: 80
*********** End of AnEn Computation Summary **********
Computing analogs ...
Progress: 100%
AnEnIS generation done!
Writing AnEn ...
anen_netcdf complete!
The program generates the following output file.
$ ncdump -h out.nc
netcdf out {
dimensions:
num_stations = 364 ;
num_test_times = 116 ;
num_flts = 24 ;
num_analogs = 1 ;
num_parameters = 20 ;
num_search_times = 448 ;
variables:
double analogs(num_analogs, num_flts, num_test_times, num_stations) ;
double weights(num_parameters) ;
double Xs(num_stations) ;
double Ys(num_stations) ;
uint64 test_times(num_test_times) ;
uint64 search_times(num_search_times) ;
uint64 FLTs(num_flts) ;
string ParameterNames(num_parameters) ;
// global attributes:
:_NCProperties = "version=1|netcdflibversion=4.6.0|hdf5libversion=1.10.0" ;
:num_analogs = 1 ;
:num_similarity = 1 ;
:observation_id = 0 ;
:max_par_nan = 0 ;
:max_flt_nan = 0 ;
:flt_radius = 1 ;
:operation = 0 ;
:quick = 0 ;
:prevent_search_future = 1 ;
:no_norm = 0 ;
:Institute = "GEOlab @ Penn State" ;
:Institute\ Link = "http://geolab.psu.edu" ;
:Package = "Parallel Analog Ensemble" ;
:Package\ Version = "v 4.4.4" ;
:Package\ Link = "https://weiming-hu.github.io/AnalogsEnsemble" ;
:Report\ Issues = "https://github.com/Weiming-Hu/AnalogsEnsemble/issues" ;
Additional to the time setting, you might also want to check your parameters in the forecast file. It might surprise you.
$ ncdump -v ParameterNames input_pred2.nc
netcdf input_pred2 {
dimensions:
num_parameters = 37 ;
num_circulars = 1 ;
num_stations = 364 ;
num_times = 628 ;
num_flts = 24 ;
variables:
string ParameterNames(num_parameters) ;
string ParameterCirculars(num_circulars) ;
float Xs(num_stations) ;
Xs:_FillValue = NaNf ;
float Ys(num_stations) ;
Ys:_FillValue = NaNf ;
float Times(num_times) ;
Times:_FillValue = NaNf ;
float FLTs(num_flts) ;
FLTs:_FillValue = NaNf ;
float Data(num_flts, num_times, num_stations, num_parameters) ;
Data:_FillValue = NaNf ;
// global attributes:
:_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6" ;
data:
ParameterNames =
"<xarray.DataArray \'ParameterNames\' ()>\narray(1., dtype=float32)\nCoordinates:\n ParameterNames float32 1.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(2., dtype=float32)\nCoordinates:\n ParameterNames float32 2.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(3., dtype=float32)\nCoordinates:\n ParameterNames float32 3.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(4., dtype=float32)\nCoordinates:\n ParameterNames float32 4.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(5., dtype=float32)\nCoordinates:\n ParameterNames float32 5.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(6., dtype=float32)\nCoordinates:\n ParameterNames float32 6.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(7., dtype=float32)\nCoordinates:\n ParameterNames float32 7.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(8., dtype=float32)\nCoordinates:\n ParameterNames float32 8.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(9., dtype=float32)\nCoordinates:\n ParameterNames float32 9.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(10., dtype=float32)\nCoordinates:\n ParameterNames float32 10.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(11., dtype=float32)\nCoordinates:\n ParameterNames float32 11.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(12., dtype=float32)\nCoordinates:\n ParameterNames float32 12.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(13., dtype=float32)\nCoordinates:\n ParameterNames float32 13.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(14., dtype=float32)\nCoordinates:\n ParameterNames float32 14.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(15., dtype=float32)\nCoordinates:\n ParameterNames float32 15.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(16., dtype=float32)\nCoordinates:\n ParameterNames float32 16.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(17., dtype=float32)\nCoordinates:\n ParameterNames float32 17.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(18., dtype=float32)\nCoordinates:\n ParameterNames float32 18.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(19., dtype=float32)\nCoordinates:\n ParameterNames float32 19.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(20., dtype=float32)\nCoordinates:\n ParameterNames float32 20.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(21., dtype=float32)\nCoordinates:\n ParameterNames float32 21.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(22., dtype=float32)\nCoordinates:\n ParameterNames float32 22.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(23., dtype=float32)\nCoordinates:\n ParameterNames float32 23.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(24., dtype=float32)\nCoordinates:\n ParameterNames float32 24.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(25., dtype=float32)\nCoordinates:\n ParameterNames float32 25.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(26., dtype=float32)\nCoordinates:\n ParameterNames float32 26.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(27., dtype=float32)\nCoordinates:\n ParameterNames float32 27.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(28., dtype=float32)\nCoordinates:\n ParameterNames float32 28.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(29., dtype=float32)\nCoordinates:\n ParameterNames float32 29.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(30., dtype=float32)\nCoordinates:\n ParameterNames float32 30.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(31., dtype=float32)\nCoordinates:\n ParameterNames float32 31.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(32., dtype=float32)\nCoordinates:\n ParameterNames float32 32.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(33., dtype=float32)\nCoordinates:\n ParameterNames float32 33.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(34., dtype=float32)\nCoordinates:\n ParameterNames float32 34.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(35., dtype=float32)\nCoordinates:\n ParameterNames float32 35.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(36., dtype=float32)\nCoordinates:\n ParameterNames float32 36.0\nAttributes:\n units: non",
"<xarray.DataArray \'ParameterNames\' ()>\narray(37., dtype=float32)\nCoordinates:\n ParameterNames float32 37.0\nAttributes:\n units: non" ;
}
And I have just made the program available through singularity/docker. Please see the pre-built images here. Checkout the one tagged with torch and everything will be ready to go.
from analogsensemble.
lovechang1986
commented on July 28, 2024
Thank you very much. On the question of data file time, I want to make sure that the forecast file time and the real-time data time are exactly the same?
As for the data files themselves, I write NC files using xarray, but for some reason using NC dump directly causes this strange problem, but it seems to come out as if the data is usable.
from analogsensemble.
Weiming-Hu
commented on July 28, 2024
Thank you very much. On the question of data file time, I want to make sure that the forecast file time and the real-time data time are exactly the same?
No. Forecast file time and the real-time data time do NOT need to be the same. The program is able to figure out the overlapping period of time and check whether your testing (e.g. search and test period) is valid. The program will also take care of aligning forecast initialization time and lead time with valid time.
As for the data files themselves, I write NC files using xarray, but for some reason using NC dump directly causes this strange problem, but it seems to come out as if the data is usable.
I noticed the format. Not sure why xarray does this. But if results come out fine, this would not be a problem.
from analogsensemble.
Related Issues (20)
- MPI output for only one process
- Forecast dimensions check
- Use archived Boost library HOT 1
- StationName order when using formatObservations HOT 4
- RAnEn installation failed on Windows HOT 3
- Installation error on Mac OS Catalina HOT 3
- Data downloading issue HOT 2
- Issue with swift and Xcode HOT 5
- Outdated instruction
- Work with timeseries of observations and forecast HOT 3
- Unable to find method "generateAnalogs" HOT 1
- Linking error when integrating PyTorch HOT 3
- Installation problems HOT 4
- Self-built NetCDF C++4 include path is not added to NETCDF_INCLUDE automatically HOT 1
- Installation problem? HOT 3
- Integration with Conda HOT 4
- Installation issue on a local machine without Internet connection HOT 5
- Get NA when using formatForecasts function to generate the forecasts list HOT 3
- How to use your AnEn model in Python？ HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from analogsensemble.