Reading cycle number from GCPL mpr file about yadg HOT 7 CLOSED

I had a look and tried to process the *_04_GCPL_C16.mpr file with both the development version of yadg as well as with yadg-4.2.4, see attached picture:

I don't think the cycle number is stored in the .mpr file. Which is also why the NetCDF has NaNs. It might be calculated by EC-Lab once the .mpr file is opened there; see #41.

Could you please clarify if this is a regression introduced in yadg-5.x or whether it's also missing from the json files on your end with yadg-4.2.x?

from yadg.

@PeterKraus
I have updated all of my environment to yadg5 since we transitioned to yadg5 last week. I tried to make a new conda environment with version 4.2 and tried to run this file but stragely got the following error

Traceback (most recent call last):
  File "C:\Users\plnu\Anaconda3\envs\yadg4\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\plnu\Anaconda3\envs\yadg4\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\plnu\Anaconda3\envs\yadg4\Scripts\yadg.exe\__main__.py", line 4, in <module>
  File "C:\Users\plnu\Anaconda3\envs\yadg4\lib\site-packages\yadg\__init__.py", line 2, in <module>
    from .main import run_with_arguments
  File "C:\Users\plnu\Anaconda3\envs\yadg4\lib\site-packages\yadg\main.py", line 8, in <module>
    from . import subcommands
  File "C:\Users\plnu\Anaconda3\envs\yadg4\lib\site-packages\yadg\subcommands.py", line 8, in <module>
    from dgbowl_schemas import to_dataschema, DataSchema
ImportError: cannot import name 'to_dataschema' from 'dgbowl_schemas' (C:\Users\plnu\Anaconda3\envs\yadg4\lib\site-packages\dgbowl_schemas\__init__.py)

Anyhow, since you have tried on your end with both stable and development version of yadg 4.2 and found that there is no record of cycle number, I doubt it would be different on my end.

I investigated this matter a bit further. I used EC-lab (version<11.5) to export this *_04_GCPL_C16.mpr file into .mpt and found that indeed this very mpr file was re-written. The size of the file increases from 14 mb to 23 mb suggesting that EC-lab may have add some columns into this. You can find the file after conversion here I used yadg5 to process this new mpr file again, but still the cycle number still remain to be 1 and NaN.

When I used EC-lab (version >11.5) to perform mpr to mpt converion, the resulting mpr file after conversion can no longer be processed using yadg5. Despite this file being recorded using EC-lab version <11.5. This strongly suggest that EC-lab does re-write the mpr file after mpt conversion.

I have sent an email to Biologic contact person (Corsin was pissed about them changing mpr format without informing us, so we organized a meeting once). I asked him about whether or not Cycle number is recorded in the original mpr file. I will get back to you If I got any further information.

from yadg.

The Biologic representative has responded my email and said that he cannot comment on the content of the mpr files as they are intellectual property of Biologic SAS.

from yadg.

@NukP Alright. I'll close the issue as there's nothing we can do about "missing" data in yadg. Perhaps the cycle numbers could be computed (e.g. from the half cycle column), but that should then be implemented in dgpost, as it's post-processing, not extraction. Feel free to open a feature request there.

With regards to installing yadg-4.2, I suggest you try and install yadg-4.2.5 (which is the latest in that branch) - I just tried it on my machine and it works without issues; if you run into problems, please open a new issue.

from yadg.

@PeterKraus Everything seems to point into the direction that the cycle number is not recorded in mpr file in the first place. One small thing that I don't get is that if the cycle number is not there in the first place, why does yadg read this column name with some of the row having value 1? If this is being recorded, then cycle number should not even be appeared in the extracted column at all.

Anyhow, there is nothing we can do about this and I agree this would be best resolved by calculating the cycle number in dgpost as you suggested.

from yadg.

You're probably merging different techniques into one NetCDF file in your preset (perhaps even with --ignore-merge-errors). Missing values get back-filled with np.NaN.

from yadg.

Ah I see, that's probably the reason why. Thanks.

from yadg.