Comments (4)
FabianHofmann
commented on September 20, 2024
Hey @TohokuEva, I am wondering what your model looks like that you have 10425 different arrays of constraints. This looks suboptimal, and part of the reason why the writing time is high. Perhaps you make sure that define arrays on constraints with coordinates? This should speed up things.
from linopy.
TohokuEva
commented on September 20, 2024
Thank you very mush for your answer, @FabianHofmann.
While defining KCL and KVL constraints I have used loops. This wad indeed the origin of the problem.
My previous code was this one:
for i in coords_nodes:
for t in coords_time:
lhs = ((var_gen_dispatch.sel(dim_nodes=i,dim_time=t)).sum()
+ (var_stor_discharging.sel(dim_nodes=i,dim_time=t)).sum()
- (var_stor_charging.sel(dim_nodes=i,dim_time=t)).sum()
+ (network_matrix_K.sel(dim_nodes=i) * var_line_flow.sel(dim_time=t)).sum())
model.add_constraints(lhs, '=', demand.sel(dim_nodes=i,dim_time=t),
name = 'KCL_{}_{}'.format(i,t))
for c in coords_cycles:
for t in coords_time:
lhs = (network_matrix_C.sel(dim_cycles=c) * line_reactance * var_line_flow.sel(dim_time=t)).sum()
model.add_constraints(lhs, '=', 0,
name = 'KVL_{}_{}'.format(c,t))
After you pointed the problem I have tried to fix it by deleting the loops and specifying on which dimension the variables should be summed. I did not really know what is the good formulation, so I am not sure if I did it correctly. The new code is this one:
lhs = ((var_gen_dispatch).sum('dim_gen_techno')
+ (var_stor_discharging).sum('dim_stor_techno')
- (var_stor_charging).sum('dim_stor_techno')
+ (network_matrix_K * var_line_flow).sum('dim_lines'))
model.add_constraints(lhs, '=', demand, name = 'KCL')
lhs = (network_matrix_C * var_line_flow * line_reactance).sum('dim_lines')
model.add_constraints(lhs, '=', 0, name = 'KVL')
I obtained the same result, but with a number of constraints drastically decreased:
Writing constraints.: 100%|██████████| 11/11 [00:00<00:00, 37.37it/s]
Writing variables.: 100%|██████████| 8/8 [00:00<00:00, 121.19it/s]Read LP format model from file C:\Users\Eva\AppData\Local\Temp\linopy-problem-tx0z8oe8.lp
Reading time = 0.20 seconds
obj: 126481 rows, 91595 columns, 367979 nonzeros
Gurobi Optimizer version 10.0.0 build v10.0.0rc2 (win64)
CPU model: 12th Gen Intel(R) Core(TM) i9-12900K, instruction set [SSE2|AVX|AVX2]
Thread count: 16 physical cores, 24 logical processors, using up to 24 threads
Optimize a model with 126481 rows, 91595 columns and 367979 nonzeros
Model fingerprint: 0xaa3f369a
Coefficient statistics:
Matrix range [1e-06, 2e+02]
Objective range [1e-02, 1e+05]
Bounds range [4e+02, 1e+07]
RHS range [6e+02, 6e+07]
Presolve removed 7713 rows and 7715 columns
Presolve time: 0.15s
Presolved: 118768 rows, 83880 columns, 354826 nonzeros
Concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...
Ordering time: 0.01s
Barrier statistics:
Dense cols : 81
Free vars : 6695
AA' NZ : 3.834e+05
Factor NZ : 1.991e+06 (roughly 100 MB of memory)
Factor Ops : 1.134e+08 (less than 1 second per iteration)
Threads : 14
Objective Residual
Iter Primal Dual Primal Dual Compl Time
0 2.05908200e+14 -3.84291651e+15 6.96e+10 4.15e+03 5.26e+12 0s
1 2.37145030e+14 -2.07731861e+15 5.70e+10 2.37e+05 2.48e+12 0s
2 1.39281994e+14 -1.68120405e+15 3.02e+10 8.45e+04 1.17e+12 0s
3 7.48499079e+13 -1.75250575e+15 1.44e+10 4.79e+04 6.36e+11 0s
4 3.63519649e+13 -1.80307024e+15 4.90e+09 1.02e+04 1.71e+11 0s
5 2.55046947e+13 -1.28282703e+15 2.44e+09 4.28e+03 8.11e+10 1s
6 2.08641032e+13 -9.86927082e+14 1.45e+09 2.37e+03 4.72e+10 1s
7 1.79799724e+13 -6.62212421e+14 8.66e+08 9.19e+02 2.36e+10 1s
8 1.62996655e+13 -5.56145461e+14 5.71e+08 6.13e+02 1.58e+10 1s
9 1.48920139e+13 -4.10745223e+14 3.43e+08 2.68e+02 8.71e+09 1s
10 1.27036089e+13 -2.36710884e+14 5.17e+07 7.19e+01 2.35e+09 1s
11 1.08221505e+13 -2.72790232e+13 1.94e+07 4.03e+00 3.41e+08 1s
12 7.81385424e+12 -1.25459861e+13 7.89e+06 1.70e+00 1.56e+08 1s
13 4.64090193e+12 -9.21035431e+12 1.41e+06 1.26e+00 8.41e+07 1s
14 3.66071977e+12 -7.80788460e+12 9.67e+05 2.21e+00 6.75e+07 1s
15 2.15706917e+12 -2.33085366e+12 3.45e+05 3.11e+00 2.49e+07 1s
16 1.25829759e+12 -1.34067622e+12 1.44e+05 1.40e+00 1.40e+07 1s
17 5.96802858e+11 -6.02885956e+11 5.01e+04 1.09e+00 6.36e+06 1s
18 3.09261641e+11 -1.91234391e+11 3.61e+03 4.14e-01 2.63e+06 1s
19 1.95736031e+11 -1.03025108e+11 4.22e+00 2.24e-01 1.57e+06 1s
20 1.27169044e+11 -7.91083101e+10 2.62e+00 3.63e-01 1.08e+06 1s
21 7.53853579e+10 -2.61494560e+10 1.44e+00 3.89e-01 5.32e+05 1s
22 5.26230399e+10 -6.21750537e+09 9.20e-01 3.65e-01 3.08e+05 1s
23 3.10837474e+10 1.47160160e+09 3.70e-01 2.20e-01 1.55e+05 1s
24 1.77664833e+10 2.31139980e+09 1.35e-01 8.76e-02 8.08e+04 1s
25 1.38019323e+10 5.99811191e+09 7.58e-02 3.60e-02 4.08e+04 1s
26 1.17194986e+10 6.98805479e+09 4.39e-02 2.79e-02 2.47e+04 1s
27 1.08168370e+10 7.57637732e+09 2.98e-02 6.76e-02 1.69e+04 1s
28 1.05474688e+10 7.79530339e+09 2.54e-02 5.62e-02 1.44e+04 1s
29 1.02049470e+10 7.98688715e+09 2.01e-02 4.73e-02 1.16e+04 1s
30 1.01387929e+10 8.07818319e+09 1.89e-02 4.35e-02 1.08e+04 1s
31 1.00498951e+10 8.25265912e+09 1.70e-02 3.60e-02 9.40e+03 1s
32 9.90306459e+09 8.39870399e+09 1.44e-02 4.86e-02 7.87e+03 1s
33 9.83678361e+09 8.44772011e+09 1.31e-02 4.52e-02 7.26e+03 2s
34 9.78729955e+09 8.50823320e+09 1.22e-02 4.19e-02 6.69e+03 2s
35 9.68576617e+09 8.66392022e+09 1.03e-02 3.33e-02 5.34e+03 2s
36 9.52086005e+09 8.74498175e+09 7.33e-03 2.71e-02 4.06e+03 2s
37 9.39270813e+09 8.81342891e+09 5.24e-03 2.11e-02 3.03e+03 2s
38 9.31977537e+09 8.89093165e+09 4.05e-03 1.42e-02 2.24e+03 2s
39 9.28747268e+09 8.93515235e+09 3.52e-03 1.50e-02 1.84e+03 2s
40 9.19958341e+09 8.95987749e+09 2.13e-03 1.39e-02 1.25e+03 2s
41 9.18346274e+09 9.00597495e+09 1.88e-03 7.92e-03 9.28e+02 2s
42 9.15370864e+09 9.01542616e+09 1.40e-03 5.50e-03 7.23e+02 2s
43 9.13921012e+09 9.02475199e+09 1.17e-03 6.53e-03 5.99e+02 2s
44 9.12361713e+09 9.03483949e+09 9.28e-04 6.32e-03 4.64e+02 2s
45 9.10934181e+09 9.04070645e+09 7.15e-04 5.89e-03 3.59e+02 2s
46 9.10230029e+09 9.04544674e+09 5.92e-04 5.14e-03 2.97e+02 2s
47 9.09216631e+09 9.04981166e+09 4.21e-04 4.29e-03 2.21e+02 2s
48 9.09061697e+09 9.05313908e+09 3.97e-04 3.69e-03 1.96e+02 2s
49 9.08751203e+09 9.05537159e+09 3.45e-04 3.11e-03 1.68e+02 2s
50 9.08094569e+09 9.05785298e+09 2.36e-04 2.60e-03 1.21e+02 2s
51 9.07781389e+09 9.06134599e+09 1.85e-04 2.08e-03 8.61e+01 2s
52 9.07379618e+09 9.06239848e+09 1.26e-04 1.89e-03 5.96e+01 2s
53 9.07197046e+09 9.06312530e+09 9.57e-05 1.65e-03 4.63e+01 2s
54 9.07025077e+09 9.06423894e+09 6.54e-05 1.27e-03 3.14e+01 2s
55 9.06867877e+09 9.06500298e+09 9.89e-04 9.11e-04 1.92e+01 2s
56 9.06777841e+09 9.06548237e+09 3.38e-03 5.81e-04 1.20e+01 2s
57 9.06723025e+09 9.06563339e+09 4.55e-03 4.88e-04 8.35e+00 2s
58 9.06704033e+09 9.06573719e+09 3.70e-03 3.80e-04 6.82e+00 3s
59 9.06692864e+09 9.06581696e+09 3.30e-03 2.92e-04 5.82e+00 3s
60 9.06667971e+09 9.06594734e+09 2.32e-03 3.39e-04 3.83e+00 3s
61 9.06658607e+09 9.06598391e+09 1.88e-03 3.72e-04 3.15e+00 3s
62 9.06644750e+09 9.06602358e+09 1.25e-03 4.01e-04 2.22e+00 3s
63 9.06636073e+09 9.06608835e+09 8.89e-04 4.11e-04 1.43e+00 3s
64 9.06629124e+09 9.06610711e+09 6.18e-04 4.12e-04 9.63e-01 3s
65 9.06624721e+09 9.06612531e+09 4.15e-04 3.51e-04 6.38e-01 3s
66 9.06622989e+09 9.06613225e+09 3.26e-04 3.04e-04 5.11e-01 3s
67 9.06621536e+09 9.06613682e+09 2.52e-04 2.65e-04 4.11e-01 3s
68 9.06619440e+09 9.06614879e+09 1.48e-04 1.61e-04 2.39e-01 3s
69 9.06618170e+09 9.06615589e+09 8.75e-05 2.05e-04 1.35e-01 3s
70 9.06617465e+09 9.06615858e+09 5.45e-05 2.31e-04 8.41e-02 3s
71 9.06616944e+09 9.06616019e+09 3.05e-05 2.51e-04 4.84e-02 3s
72 9.06616482e+09 9.06616151e+09 9.28e-06 2.50e-04 1.73e-02 3s
73 9.06616330e+09 9.06616218e+09 3.03e-06 2.09e-04 5.85e-03 4s
74 9.06616274e+09 9.06616242e+09 9.11e-07 1.58e-04 1.69e-03 4s
75 9.06616248e+09 9.06616245e+09 5.20e-08 1.71e-04 1.66e-04 4s
76 9.06616245e+09 9.06616246e+09 2.44e-07 1.80e-04 7.73e-07 4s
77 9.06616245e+09 9.06616246e+09 1.25e-08 1.83e-04 2.84e-10 4s
Barrier solved model in 77 iterations and 3.83 seconds (4.14 work units)
Optimal objective 9.06616245e+09
Crossover log...
60897 DPushes remaining with DInf 2.3298577e-03 4s
0 DPushes remaining with DInf 2.3298577e-03 4s
Warning: Markowitz tolerance tightened to 0.5
5854 PPushes remaining with PInf 0.0000000e+00 4s
0 PPushes remaining with PInf 0.0000000e+00 4s
Push phase complete: Pinf 0.0000000e+00, Dinf 3.6874893e-10 4s
Solved with barrier
Iteration Objective Primal Inf. Dual Inf. Time
47816 9.0661625e+09 0.000000e+00 0.000000e+00 4s
Solved in 47816 iterations and 4.43 seconds (5.24 work units)
Optimal objective 9.066162454e+09
And this time, the real solving time (I calculated with time.time()) is 5.6 seconds instead of more than one hour.
Thank you very much for your help!
from linopy.
FabianHofmann
commented on September 20, 2024
Great! There you go :)
from linopy.
FabianHofmann
commented on September 20, 2024
I'll close this, reopen if anything related comes up again.
from linopy.
Related Issues (20)
- MPS IO changes with highspy >= 1.7
- Re-solving a changed model with "direct" api results in wrong model being solved HOT 1
- add_constraints lhs eats constant HOT 2
- Explore PytOptInferface for writing matrix over native C API HOT 9
- Improve workflow for releases HOT 1
- Pass parameters to the Gurobi solver HOT 1
- Docs errata: LinearExpression `expand_dims` HOT 2
- How are "0" coefficient passed to gurobi? HOT 3
- Multiple coordinates for one dimension HOT 2
- groupby sum not allowing multidimensional grouping per default HOT 6
- Creating variable for each linear expression HOT 4
- `xarray` DeprecationWarning: Deleting a single level of a MultiIndex is deprecated. Previously
- masking vs selecting HOT 3
- Slicing selection not aligned on "Python standards" HOT 3
- sol file is not dumped when using gurobi
- 0.3.12 ModuleNotFoundError: No module named 'highspy' HOT 3
- Update xarray package requirements HOT 1
- Overhaul CI and CD workflows HOT 6
- 0.3.3 release onwards have broken Gurobi solves for large models. HOT 5
- Serializing and deserializing linopy.Model HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from linopy.