[str(n) for n in res1d.data.Reaches]

may give something like

['Res1DReach: B4.1200l1-1 (0-479.998999014497)',
 'Res1DReach: B4.1502l1-12 (0-209.999271030538)',
 'Res1DReach: B4.1520l1-13 (0-235.000625581015)',
 'Res1DReach: Weir:B4.1480w1-14 (0-1)',
 'Res1DReach: Weir:B4.1510w1-15 (0-1)',
 'Res1DReach: Pump:B4.1510p1-16 (0-80.0006092712283)',
 'Res1DReach: Pump:B4.1510p2-17 (0-80.0006092712283)']

Is there a way to get the reach type, which is named 'Weir' or 'Pump' in the above list, and probably 'Link' for the other lines?

I've created a function, but I don't know which values are possible beside 'Weir' and 'Pump'.

def get_reach_type(reach):
    try:
        # get the full identifier string:
        fullstring = str(reach)

        # get the second part after the ':'
        structureReach = fullstring.split(':')[1].lstrip()

        # if the second part is Pump or Weir, use Pump or Weir:
        if structureReach in ['Pump','Weir']:
            reach_type = structureReach

        # in any other case this must be a normal Link (Pipe or Canal or River)
        else:
            reach_type = 'Link'
    
    except:
        reach_type = None

    return reach_type

I'm thinking of something simpler, similar to ''Reach.Name'', wich extracts the MUID. This is a property important for any further processing.

Thank you,
Thomas

For collection system modelling, the results saved in .res1d sometimes need to be transformed based on the properties of the node/reach/catchment. In Mike Urban and MIKE+, they're called derived results. For example, the .res1d might only contain water level for nodes, whereas someone might be interested in node flooding, calculated as the difference of the water level and the top of the node. Other examples are pipe filling and pipe flooding, which relate water level in the pipe to its dimensions and ground level (in this case, I assume ground level is obtained from the adjoining nodes). A comparable feature would be useful for mikeio1d.

from mikeio1d.res1d import Res1D, QueryDataReach
Traceback (most recent call last):

File "", line 1, in
from mikeio1d.res1d import Res1D, QueryDataReach

File "C:\Anaconda3\lib\site-packages\mikeio1d\res1d.py", line 25, in
from DHI.Mike1D.ResultDataAccess import ResultData, ResultDataQuery

ImportError: cannot import name 'ResultDataQuery' from 'DHI.Mike1D.ResultDataAccess' (unknown location)

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.
I want to use xns11 to create a new xns11
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.
[email protected]

Trying to install MIKE IO 1D on windows pip install mikeio1d but keep getting this error

`
Running setup.py install for pythonnet ... error
error: subprocess-exited-with-error

× Running setup.py install for pythonnet did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

  error: option --single-version-externally-managed not recognized
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pythonnet

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
`

Discussed in #49

I'm trying to read pump discharge but the quantity is not given in the res1d object. The quantities list in res1d object are given below. The pump discharge is not listed. Same res1d file given more quantities in MIKE+ (see attached screenshot). I have uploaded the res1d file to the ftp (ftp://[email protected]/pub/zhyu/richmond_2041OCP_24hr_2.4m_Upgrade2041OCP-24hr-2.4m-UpgradeRichmondHD.res1d) . Thanks for your help.

['WaterLevel',
'WaterVolumeAboveGround',
'WaterFlowRateAboveGround',
'WaterVolumeRetainedInMaxInflow',
'Discharge',
'FlowVelocity',
'WaterLevelSlope',
'EnergyLevelSlope']

When I tried to read the water volume above ground for all nodes, I got following error. The code I'm using is given below. I also tried C# query class, it also give errors (given below). I have uploaded res1d file to the ftp: ftp://[email protected]/pub/ZHYU/richmond_20200923_2020-09-23RichmondHD.res1d

I'm also wondering if there is function to read all max water volume above ground values for all nodes.

allnode_ids = [n.Id for n in res1d.data.Nodes]

queries = []
for n in allnode_ids:
queries.append(QueryDataNode('WaterVolumeAboveGround', n))

max_level_of_all = res1d.read(queries).max()
max_level_of_all

TypeError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_35324/2335864540.py in
5 queries.append(QueryDataNode('WaterVolumeAboveGround', n))
6
----> 7 max_level_of_all = res1d.read(queries).max()
8 max_level_of_all

~\AppData\Roaming\Python\Python37\site-packages\mikeio1d\res1d.py in read(self, queries)
163 df = pd.DataFrame(index=self.time_index)
164 for query in queries:
--> 165 df[str(query)] = query.get_values(self)
166
167 return df

~\AppData\Roaming\Python\Python37\site-packages\mikeio1d\res1d.py in get_values(self, res1d)
126 def get_values(self, res1d):
127 values = res1d.query.GetNodeValues(self._name, self._quantity)
--> 128 return self.from_dotnet_to_python(values)
129
130

~\AppData\Roaming\Python\Python37\site-packages\mikeio1d\res1d.py in from_dotnet_to_python(array)
44 def from_dotnet_to_python(array):
45 """Convert .NET array to numpy."""
---> 46 return np.fromiter(array, np.float64)
47
48 @Property

TypeError: 'NoneType' object is not iterable

Then I used C# query class, it gives the error below.

allnode_ids = [n.Id for n in res1d.data.Nodes]

max_water_volume_above_ground = {}
for n in allnode_ids:
max_water_volume_above_ground[n] = to_numpy(res1d.query.GetNodeValues(n,'WaterVolumeAboveGround')).max

max_water_volume_above_ground

TypeError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_35324/535727106.py in
3 max_water_volume_above_ground = {}
4 for n in allnode_ids:
----> 5 max_water_volume_above_ground[n] = to_numpy(res1d.query.GetNodeValues(n,'WaterVolumeAboveGround')).max
6
7 max_water_volume_above_ground

~\AppData\Roaming\Python\Python37\site-packages\mikeio1d\dotnet.py in to_numpy(src)
156 try:
157 src_ptr = src_hndl.AddrOfPinnedObject().ToInt64()
--> 158 bufType = ctypes.c_float * len(src)
159 cbuf = bufType.from_address(src_ptr)
160 d = np.frombuffer(cbuf, dtype=cbuf.type)

TypeError: object of type 'NoneType' has no len()

Hi, when I read the res1d file using following codes, I got such error. It's weird because in other computers It did work successfully. Do you know how to fix it? is it something related to the environment setting？

from mikeio1d.res1d import Res1D, QueryDataReach
Res1D('HD_test.res1d').read()

Thanks

Is it possible to read a res1d file only between a certain timestamp or for certain catchments? As far as I can tell, it's only possible to query reaches and manholes, but not catchments or time steps. This is useful when dealing with large files >1gb.

Hi,

Trying to read my own (and several others) .res1d I get the following error:

res1d = Res1D(file_path)
df = res1d.read()
Traceback (most recent call last):

File "", line 2, in
df = res1d.read()

File "C:\Users\aac\Anaconda3\lib\site-packages\mikeio1d\res1d.py", line 159, in read
return self.read_all()

File "C:\Users\aac\Anaconda3\lib\site-packages\mikeio1d\res1d.py", line 174, in read_all
for values, col_name in Res1D.get_values(

File "C:\Users\aac\Anaconda3\lib\site-packages\mikeio1d\res1d.py", line 188, in get_values
col_name = col_name_delimiter.join([data_item.Quantity.Id, name])

TypeError: sequence item 1: expected str instance, NoneType found

I've tried with the .res1d file which you have in the test folder, worked fine.
I'm using MIKE URBAN 2020 Update 1

It could be great, if we could in the future write pd.DataFrames as .res1d files.

For a model I'm working on, I found the discharge read with the MIKEIO1D and is different than the one read in M+/MIKE VIEW (see the two plots below). The code I'm using is given below. The version of MIKEIO1D is 0.4.0. The link id is 183211. The res1d file is here. Any ideas?

result = Res1D(r"WWF_2023-08-03Runoff_2022-2023Default_Network_HD.res1d", reaches = ["183211"])
result = result.read()
result = result[[column for column in result.columns.to_list() if "Discharge" in column]]
result.plot(figsize = (21,10))

Hi,

I received hundreds of such PerformanceWarning:

PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use newframe = frame.copy()
df[col_name] = values

Is there any way to read res1d files and overcome these performance warnings?

The cross-section properties can be exported to text files through MIKE Zero, I can also import the text files to MIKE Zero. The question is can I convert the exported txt file to an xns11 file using methods in MIKE 1D python API, like DHI.Mike1D.CrossSectionModule.

I can use DHI.Mike1D.CrossSectionModule.CrossSectionDataFactory.Open function to open the xns11 file and read the cross-section information, but how can I save a txt file to xns11 and use it by MIKE Zero directly in Python?

MIKE Zero version: 2019
Python version: 3.7

Adding this issue from an external request:

Has the MIKEIO 1D for xns11 file been improved as for the result files res1d?

For instance, can it be extracted the markers (e.g.1, 2, 3) from raw data, plot them in a shapefile (via GeoPandas) and set them, back into the XNS11 file?

This can be important as a pre-processing work for coupling with 2D and MSHE and other applications.

I had a Mike+-simulation with a fixed time step of 0.01 s, and chose results to be saved every 3 time step. When I opened the res1d-file in Mikeio1d, the DateTime index has a resolution of 1 s, without any decimals. A time series plot from Python therefore creates a stepped plot instead of a continuous plot. It can be fixed by writing, e.g.

df['proper_time'] = pd.date_range(start='2023-01-01 00:00:00',end='2023-01-01 00:10:00',periods=20001)
df = df.set_index('proper_time')

but this should be correct when the series is opened in Python.

The performance of reading result files could be considerably improved.

Here are two examples of reading files on my machine (16 GB of RAM, 4 cores 2.7 GHz, Samsung MZVLW512HMJP drive):

• A ~1.1 GB network result file with ~58000 times series and ~4300 time steps currently takes the following time:

  1. Load data into ResultData object takes ~20 seconds and ~1.1 GB of memory
  2. Reading in memory from ResultData to data frame takes ~220 seconds and ~2.2 GB of memory

• A ~1.5 GB catchment result file with ~27000 time series and ~15000 time steps currently takes the following time:

  1. Load data into ResultData object takes ~12 seconds and ~1.5 GB of memory
  2. Reading in memory from ResultData to data frame takes ~210s and ~3.0 GB of memory

I think the ii. step should be at least a factor of 10 faster, because it deals with copying data in memory. I suspect there is a problem with Python to C# interop.

I have a .res1D file from a model simulation. The file is large, its about 200+ GB large.
Opening this file on MIKE VIEW was painfully slow and I have been testing the MIKEIO1D python tools developed by DHI but that was also taking so long.

When trying the following code, python says there is not enough memory on the computer, even though I have a 64GB RAM computer

`

from mikeio1d.res1d import Res1D, QueryDataReach
df = Res1D(filename).read()
`

Can you suggest a better/faster way to read the .res1D file and extract the results?

Thanks,
Michael

Hi,

Is it possible to extract from a res1d only the WaterLevel and drop the DischargeToSurface for example?

thanks for the help

Hi!
I had a customer solving his own request, I thaught it's worth sharing:

He tried to read a LTS extreme statistics *.res1d file via mikeio1d and got the following error:

df = Res1D(‘../Var0LTS_extreme_statistics.res1d’).read()
OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 100-01-01 00:00:00

The Pandas documentation gave him the following explenation:
„Since pandas represents timestamps in nanosecond resolution, the timespan that can be represented using a 64-bit integer is limited to approximately 584 years
pd.Timestamp.min
Out[54]: Timestamp('1677-09-22 00:12:43.145225')

In [55]: pd.Timestamp.max
Out[55]: Timestamp('2262-04-11 23:47:16.854775807')

After some back and forth, his final mail was, that the file he tried to read had too many data points. By changing the time index via the code in line 18 of the Jupyter Notebook, he could read the LTS:

time_stamps = [datetime.datetime(1900, 1, 1, 0, 0, 1)+datetime.timedelta(days = i) for i in range(number_of_event_entries)]

The maximum date stayed inside the boundaries needed and the LTS was readable.
Hope this helps someone else too as a quick fix.

Can we write or modify the xns11 file?

I have a res1d file which contains the results from a simulation in Mike+. I imported that file with
df = mikeio1d.Res1D("model_m1d - Result Files/modelBaseDefault_Network_HD.res1d")
One of the reaches in the model is a tunnel which I earlier opened with
discharge = df.read(mikeio1d.res1d.QueryDataReach("Discharge", 'Fictious outlet Valldalen-Basin'))
But I get an error:
NoDataForQuery: Invalid query Discharge:Fictious outlet Valldalen-Basin

It works properly in mikeio1d version 0.3.0, but not in 0.4.0. Is it a bug?

Don´t have permission to push to a new branch, otherwise I would have added a failing test in branch bug/convertRes11ToRes1d.
Conversion fails saying the res11 bridge is not for writing. I would expect it tries writing using the res1d bridge.

Is there any functionality to read the markers of cross sections, e.g. left levee bank and lowest point?

https://github.com/DHI/mikeio1d/blob/edacdc3d2fa4dacf71fa3da21dfac4a1dcda39e6/mikeio1d/result_network/various.py#L15C13-L15C13
I come from China, sometime I use Chinese characters for river name.

    # Replace all non alpha numeric characters by an underscore.
    string = re.sub(r"[^a-zA-Z0-9]", "_", string)
    # Replace more than two underscores with a single underscore.
    string = re.sub(r"_{2,}", "_", string)
    # Add an extra string is the string starts with a number.
    string = extra_string_before_digit + string if string and string[0].isdigit() else string
    # Remove a starting underscore
    string = string[1:] if string[0] == "_" else string

After above code run, string's length will be zero，at last,
string = string[:-1] if string[-1] == "_" else string will raise an exception.

I noticed this error when querying a reach. I've had success querying reaches with chainages set to 0 or 10, but this one gives troubles. Similar errors happen if I use reaches or catchments as a param in Res1D, and then try to read it without a query (e.g. res1d.read(). For rivers it may be okay to have the chainage as mandatory, but for sewer networks where a "reach" is actually a "pipe", the chainage is more an inconvenience than anything. For that reason it would be nice to have chainge be optional, with the default returning values for all of the chainages. A convenient way to list chainages connected to a reach would also be useful.

A common workflow in collection system modelling is to visualize the results on a map with GIS. Modelling software like Mike Urban / MIKE+ allow results to be exported to shapefile format (.shp). The attributes in the shapefile contain the results, which are for either a specific timestep or a statistic of the results (e.g. max/min). It would be nice to be able to export nodes / reaches / catchments in a similar way with mikeio1d.

We have a list of upgrades (~400) list in a Excel file. Each upgrade has a from node and to node. We want to get select the pipes in between and get their size. Please let me know if it's possible to do this.

Describe the bug
mikeio1d.res1d.Res1D module conflicts with mikeplus.Engine1D module.
Currently the mikeplus package is in beta and contains limited functionality, when I loaded Engine1D form mikeplus, I noticed that mikeio1d could not read the results properly. The following error is returned:

Python.Runtime.PythonException: DHI.Mike1D.ResultDataAccess.ResultData value cannot be converted to DHI.Mike1D.ResultDataAccess.ResultData

The above exception was the direct cause of the following exception:

System.ArgumentException: DHI.Mike1D.ResultDataAccess.ResultData value cannot be converted to DHI.Mike1D.ResultDataAccess.ResultData in method Void .ctor(DHI.Mike1D.ResultDataAccess.ResultData) ---> Python.Runtime.PythonException: DHI.Mike1D.ResultDataAccess.ResultData value cannot be converted to DHI.Mike1D.ResultDataAccess.ResultData   
   --- End of inner exception stack trace ---

The above exception was the direct cause of the following exception:

System.AggregateException: One or more errors occurred. ---> System.ArgumentException: DHI.Mike1D.ResultDataAccess.ResultData value cannot be converted to DHI.Mike1D.ResultDataAccess.ResultData in method Void .ctor(DHI.Mike1D.ResultDataAccess.ResultData) ---> Python.Runtime.PythonException: DHI.Mike1D.ResultDataAccess.ResultData value cannot be converted to DHI.Mike1D.ResultDataAccess.ResultData
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.ArgumentException: DHI.Mike1D.ResultDataAccess.ResultData value cannot be converted to DHI.Mike1D.ResultDataAccess.ResultData in method Void .ctor(DHI.Mike1D.ResultDataAccess.ResultData) ---> Python.Runtime.PythonException: DHI.Mike1D.ResultDataAccess.ResultData value cannot be converted to DHI.Mike1D.ResultDataAccess.ResultData
   --- End of inner exception stack trace ---<---

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "d:/****/****/mikepluspy-main/notebooks/test.py", line 21, in <module>
    result_ = mikeio1d.res1d.Res1D(r'D:\****\****\mikepluspy-main\tests\testdata\Db\Sirius\Sirius_m1d - Result Files\Sirius_1_DEMOBaseDefault_Network_HD.res1d')    
  File "C:\Users\****\.conda\envs\python38\lib\site-packages\mikeio1d\res1d.py", line 113, in __init__
    self.result_reader = ResultReaderCreator.create(
  File "C:\Users\****\.conda\envs\python38\lib\site-packages\mikeio1d\result_reader_writer\result_reader_creator.py", line 41, in create
    return reader(
  File "C:\Users\****\.conda\envs\python38\lib\site-packages\mikeio1d\result_reader_writer\result_reader_copier.py", line 59, in __init__
    self.result_data_copier = ResultDataCopier(self.data)
TypeError: No method matches given arguments for ResultDataCopier..ctor: (<class 'DHI.Mike1D.ResultDataAccess.ResultData'>)

To Reproduce
Steps to reproduce the behavior:
You can download the attached file which is test data for mikeplus.
Sirius.zip

1. import mikeplus
2. import mikeio1d
3. read results file using Res1D

Code

from mikeplus import DataTableAccess
from mikeplus.engines import Engine1D
import mikeio1d
'''
db_path = r'D:\Drainage\MIKEpy\mikepluspy-main\tests\testdata\Db\Sirius\Sirius.sqlite'
data_access = DataTableAccess(db_path)
data_access.open_database()

print(data_access.is_database_open())
data_access.close_database()
'''
# replace the result file path
result = mikeio1d.res1d.Res1D(r'D:\Drainage\MIKEpy\mikepluspy-main\tests\testdata\Db\Sirius\Sirius_m1d - Result Files\Sirius_1_DEMOBaseDefault_Network_HD.res1d')
print(result)

Screenshots
If applicable, add screenshots to help explain your problem.


Sirius.zip

System information:

• Python version: [Python 3.8] / [Python 3.12]
• MIKE IO version [MIKE IO 1D v0.6]

The current mikeio1d is fairly primitive and not that accessible for beginner python users. Examples of potential improvements include:

• Reading results: currently requires making a query object. Simplify so that in the read function itself you can specify ID(s), item(s), time(s), etc.
• Loading results: should only be done upon reading, not opening a .res1d file. For large result files it's nice to load the file to see what's in it before reading (e.g. result types, time frames). Right now there's some functionality to limit what's read to specific items, but it's specified as arguments when first opening the res1d file. The ideal is to load nothing, until the read function where you specify precisely what you want to load (object ID, result item, time steps)
• Result objects: a more granular definition of these would help. For example, a modeller might be looking for a 'weir', or an 'orifice', or a 'pump', or a plain 'conduit'. In mikeio1d, all of these are considered 'reaches'. I get the history behind it, but for ui/ux it could be improved here, even if only as abstractions.
• Result processing: other issues opened include features to help working with the results, like exporting to shapefiles or making derived results. Adding some flexibility here would lower the bar for users to do some useful things on their own.

How can I get the maximum water level from at large res1d file of all nodes from a simulation using mikeio with python?