opencobra / optlang Goto Github PK
View Code? Open in Web Editor NEWoptlang - sympy based mathematical programming language
Home Page: http://optlang.readthedocs.org/
License: Apache License 2.0
optlang - sympy based mathematical programming language
Home Page: http://optlang.readthedocs.org/
License: Apache License 2.0
Hi,
I think it would be helpful if travis could also run the tests for symengine and Python 3.6.
The indicator_variable
and active_when
arguments for Constraints provide a straightforward way to define conditional constraints. Currently this is only supported by cplex, which has a native implementation for these.
However the functionality could be extended so other solvers fall back on a general big-M based implementation.
For example a constraint Constraint(expr, ub=a, indicator_variable=y, active_when=0)
can be rewritten as Constraint(expr - M * y, ub=a)
where M is a sufficiently large constant and y is a binary variable
When running the nosetests locally many cplex tests fail with cplex 12.7.1.
This is not caught by the continuous integration because this tests are skipped on travis and appveyor.
======================================================================
14) ERROR: test_netlib_check_objective_value_25FV47 (test_netlib_cplex_interface.pyc)
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python2.7/dist-packages/nose/case.py line 197 in runTest
self.test(*self.arg)
optlang/tests/test_netlib_cplex_interface.py line 60 in check_objval
if problem.solution.get_status() == cplex.Cplex.solution.status.optimal:
AttributeError: type object 'Cplex' has no attribute 'solution'
======================================================================
15) ERROR: test_netlib_check_objective_value_80BAU3B (test_netlib_cplex_interface.pyc)
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python2.7/dist-packages/nose/case.py line 197 in runTest
self.test(*self.arg)
optlang/tests/test_netlib_cplex_interface.py line 60 in check_objval
if problem.solution.get_status() == cplex.Cplex.solution.status.optimal:
AttributeError: type object 'Cplex' has no attribute 'solution'
======================================================================
16) ERROR: test_netlib_check_objective_value_ADLITTLE (test_netlib_cplex_interface.pyc)
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python2.7/dist-packages/nose/case.py line 197 in runTest
self.test(*self.arg)
optlang/tests/test_netlib_cplex_interface.py line 60 in check_objval
if problem.solution.get_status() == cplex.Cplex.solution.status.optimal:
AttributeError: type object 'Cplex' has no attribute 'solution'
======================================================================
17) ERROR: test_netlib_check_objective_value_AFIRO (test_netlib_cplex_interface.pyc)
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python2.7/dist-packages/nose/case.py line 197 in runTest
self.test(*self.arg)
optlang/tests/test_netlib_cplex_interface.py line 60 in check_objval
if problem.solution.get_status() == cplex.Cplex.solution.status.optimal:
AttributeError: type object 'Cplex' has no attribute 'solution'
======================================================================
18) ERROR: test_netlib_check_objective_value_AGG2 (test_netlib_cplex_interface.pyc)
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python2.7/dist-packages/nose/case.py line 197 in runTest
self.test(*self.arg)
optlang/tests/test_netlib_cplex_interface.py line 60 in check_objval
if problem.solution.get_status() == cplex.Cplex.solution.status.optimal:
AttributeError: type object 'Cplex' has no attribute 'solution'
======================================================================
19) ERROR: test_netlib_check_objective_value_AGG3 (test_netlib_cplex_interface.pyc)
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python2.7/dist-packages/nose/case.py line 197 in runTest
self.test(*self.arg)
optlang/tests/test_netlib_cplex_interface.py line 60 in check_objval
if problem.solution.get_status() == cplex.Cplex.solution.status.optimal:
AttributeError: type object 'Cplex' has no attribute 'solution'
======================================================================
20) ERROR: test_netlib_check_objective_value_BANDM (test_netlib_cplex_interface.pyc)
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python2.7/dist-packages/nose/case.py line 197 in runTest
self.test(*self.arg)
optlang/tests/test_netlib_cplex_interface.py line 60 in check_objval
if problem.solution.get_status() == cplex.Cplex.solution.status.optimal:
AttributeError: type object 'Cplex' has no attribute 'solution'
======================================================================
21) ERROR: test_netlib_check_objective_value_BEACONFD (test_netlib_cplex_interface.pyc)
----------------------------------------------------------------------
...
Currently, _round_primal_to_bounds
(called by .primal_values
in some interfaces) throws an error if the primal values lie outside the bounds. However, that only makes sense for 'optimal' solutions. If the solver declared the solution as 'infeasible' that means by definition that one of the primal values violates a constraint (equality or inequality) so it is "ok" for infeasible solutions to lie outside of the bounds and no error should be thrown in that case.
Also creates quite a headache for cobrapy code since the Solution object is created regardless of the solver status and thus throws errors for infeasible solutions.
swiglpk
requires that a problem created with glp_problem
is eventually freed with glp_delete_prob
. Otherwise only the pointer to the problem will be deleted but not the actual problem matrices.
When populating the solver for the first time we can use the matrix initialization in GPLK
Example from https://github.com/opencobra/cobrapy/blob/devel/cobra/solvers/cglpk.pyx#L248
glp_load_matrix(glp, n_values, c_rows, c_cols, c_values).
Hi all,
could you make a new release with the latest develop changes.
It is much easier to reference a release than the develop branch in setup.py
, requirements.txt
and tox.ini
in depending projects.
Thanks so much
M
Getting the following for an infeasible problem with the cplex interface:
/home/cdiener/.local/lib/python3.5/site-packages/optlang/interface.py in primal(self)
290 """The primal of variable (None if no solution exists)."""
291 if self.problem:
--> 292 primal = self._get_primal()
293 if primal is not None:
294 if self.type in ("integer", "binary"):
/home/cdiener/.local/lib/python3.5/site-packages/optlang/cplex_interface.py in _get_primal(self)
170
171 def _get_primal(self):
--> 172 primal_from_solver = self.problem.problem.solution.get_values(self.name)
173 return primal_from_solver
174
/home/cdiener/Downloads/ibm/cplex/python/3.5/x86-64_linux/cplex/_internal/_subinterfaces.py in get_values(self, *args)
7566 return CPX_PROC.getx(self._env._e, self._cplex._lp, a, b)
7567 return apply_freeform_two_args(
-> 7568 getx, self._cplex.variables.get_indices, args)
7569
7570 def get_reduced_costs(self, *args):
/home/cdiener/Downloads/ibm/cplex/python/3.5/x86-64_linux/cplex/_internal/_aux_functions.py in apply_freeform_two_args(fn, convert, args)
67 conarg0 = con(args[0])
68 if isinstance(conarg0, six.integer_types):
---> 69 return fn(conarg0, conarg0)[0]
70 else:
71 raise TypeError("expecting name or index")
/home/cdiener/Downloads/ibm/cplex/python/3.5/x86-64_linux/cplex/_internal/_subinterfaces.py in getx(a, b)
7564 """
7565 def getx(a, b = self._cplex.variables.get_num() - 1):
-> 7566 return CPX_PROC.getx(self._env._e, self._cplex._lp, a, b)
7567 return apply_freeform_two_args(
7568 getx, self._cplex.variables.get_indices, args)
/home/cdiener/Downloads/ibm/cplex/python/3.5/x86-64_linux/cplex/_internal/_procedural.py in getx(env, lp, begin, end)
1510 x = _safeDoubleArray(xlen)
1511 status = CR.CPXXgetx(env, lp, x, begin, end)
-> 1512 check_status(env, status)
1513 return LAU.array_to_list(x, xlen)
1514
/home/cdiener/Downloads/ibm/cplex/python/3.5/x86-64_linux/cplex/_internal/_procedural.py in __call__(self, env, status, from_cb)
222 else:
223 error_string = geterrorstring(env, status)
--> 224 raise CplexSolverError(error_string, env, status)
225
226 check_status = StatusChecker()
CplexSolverError: CPLEX Error 1217: No solution exists.
Actually it is a bit weird, because it happens after solving once and modifying the model by changing bounds. In some instances that seems to cause model.problem
to reset.
Feel free to close if it is not reproducible.
Getting lots of errors with Cplex 12.7:
self = <optlang.cplex_interface.Configuration object at 0x7fc07085c320>, value = 'auto'
@solution_target.setter
def solution_target(self, value):
if self.problem is not None:
if value is None:
self.problem.problem.parameters.solutiontarget.reset()
else:
try:
solution_target = _SOLUTION_TARGETS.index(value)
except ValueError:
raise ValueError(
"%s is not a valid solution target. Choose between %s" % (value, str(_SOLUTION_TARGETS)))
> self.problem.problem.parameters.solutiontarget.set(solution_target)
E AttributeError: 'RootParameterGroup' object has no attribute 'solutiontarget'
Hey Optlang-Devs,
I had to rename a few reactions in the model I am working on, and then realized that I had to remove one of the renamed reactions. This lead to the following error:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-49-97e3d9b3721c> in <module>()
----> 1 print model.solve().f
/Users/clie/.virtualenvs/modeling/lib/python2.7/site-packages/cameo/core/solver_based_model.pyc in solve(self, solution_type, *args, **kwargs)
493 Solution or LazySolution
494 """
--> 495 solution = self.optimize(solution_type=solution_type, *args, **kwargs)
496 if solution.status is not 'optimal':
497 raise exceptions._OPTLANG_TO_EXCEPTIONS_DICT.get(solution.status, SolveError)(
/Users/clie/.virtualenvs/modeling/lib/python2.7/site-packages/cameo/core/solver_based_model.pyc in optimize(self, objective_sense, solution_type, **kwargs)
476 self.objective.direction = original_direction
477 else:
--> 478 self.solver.optimize()
479 solution = solution_type(self)
480 self.solution = solution
/Users/clie/.virtualenvs/modeling/lib/python2.7/site-packages/optlang/interface.pyc in optimize(self)
1343 Solution status.
1344 """
-> 1345 self.update()
1346 status = self._optimize()
1347 if status != OPTIMAL and self.configuration.presolve == "auto":
/Users/clie/.virtualenvs/modeling/lib/python2.7/site-packages/optlang/interface.pyc in update(self, callback)
1319 rm_var = self._pending_modifications.rm_var
1320 if len(rm_var) > 0:
-> 1321 self._remove_variables(rm_var)
1322 self._pending_modifications.rm_var = []
1323 callback()
/Users/clie/.virtualenvs/modeling/lib/python2.7/site-packages/optlang/cplex_interface.pyc in _remove_variables(self, variables)
739 self.problem.variables.delete([variable.name for variable in variables])
740 for variable in variables:
--> 741 del self._variables_to_constraints_mapping[variable.name]
742 variable.problem = None
743 del self._variables[variable.name]
KeyError: u'PAH'
I found this bug:
from cameo.util import ProblemCache
from cameo.flux_analysis import moma
model = models.bigg.iJO1366
ref = model.solve().fluxes
model.reactions.PGI.knock_out()
cache = ProblemCache(model)
res = moma(model, reference=ref, cache=cache) # first run is fine
ref['ACALD'] = -1
res = moma(model, reference=ref, cache=cache) # second run is boom
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-1-879c5ea1306a> in <module>()
8 res = moma(model, reference=ref, cache=cache) # first run is fine
9 ref['ACALD'] = -1
---> 10 res = moma(model, reference=ref, cache=cache) # second run is boom
/Users/joao/Documents/repos/cameo/cameo/flux_analysis/simulation.py in moma(model, reference, cache, reactions, *args, **kwargs)
186 reaction = model.reactions.get_by_id(rid)
187 cache.add_constraint(constraint_id, create_constraint, update_constraint,
--> 188 cache.variables[var_id], reaction, flux_value)
189
190 def create_objective(model, variables):
/Users/joao/Documents/repos/cameo/cameo/util.py in add_constraint(self, constraint_id, create, update, *args, **kwargs)
145 if constraint_id in self.constraints:
146 if update is not None:
--> 147 update(self.model, self.constraints[constraint_id], *args, **kwargs)
148 else:
149 self.time_machine(
/Users/joao/Documents/repos/cameo/cameo/flux_analysis/simulation.py in update_constraint(model, constraint, var, reaction, flux_value)
180 def update_constraint(model, constraint, var, reaction, flux_value):
181 if constraint.lb != flux_value:
--> 182 constraint.ub = flux_value
183 constraint.lb = flux_value
184
/Users/joao/.virtualenvs/marsi/lib/python3.4/site-packages/optlang-0.6.3-py3.4.egg/optlang/cplex_interface.py in ub(self, value)
271 @interface.Constraint.ub.setter
272 def ub(self, value):
--> 273 self._check_valid_upper_bound(value)
274 if getattr(self, 'problem', None) is not None:
275 if self.indicator_variable is not None:
/Users/joao/.virtualenvs/marsi/lib/python3.4/site-packages/optlang-0.6.3-py3.4.egg/optlang/interface.py in _check_valid_upper_bound(self, value)
594 raise TypeError("Constraint bounds must be numeric or None, not {}".format(type(value)))
595 if value is not None and getattr(self, "lb", None) is not None and value < self.lb:
--> 596 raise ValueError("Cannot set an upper bound that is less than the lower bound.")
597
598 @classmethod
ValueError: Cannot set an upper bound that is less than the lower bound.
I tried to change the ub and lb the other way around and I still get the same problem.
When constructing a constraint with an expression that adds or subtracts a constant (i.e. x-1), I get an error when setting the lower and upper bounds. I think this is probably due to converting the expression to standard form and not correctly modifying the upper/lower bounds in the correct order, but I haven't been able to pin it down.
Below is a simple example with the error. While c1, c2, and c3 are equivalent. c1 and c2 work. c3 throws an error.
>>> from optlang import *
>>> v=Variable('v')
>>> c1=Constraint(v,lb=5,ub=5)
>>> c2=Constraint(-v+5,lb=0,ub=0)
>>> c3=Constraint(v-5,lb=0,ub=0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/long/cobrapy/optlang/optlang/glpk_interface.py", line 151, in __init__
super(Constraint, self).__init__(expression, sloppy=sloppy, *args, **kwargs)
File "/home/long/cobrapy/optlang/optlang/interface.py", line 678, in __init__
super(Constraint, self).__init__(expression, *args, **kwargs)
File "/home/long/cobrapy/optlang/optlang/interface.py", line 431, in __init__
self._expression = self._canonicalize(expression)
File "/home/long/cobrapy/optlang/optlang/interface.py", line 749, in _canonicalize
self.lb = self.lb - float(coeff)
File "/home/long/cobrapy/optlang/optlang/glpk_interface.py", line 173, in lb
super(Constraint, Constraint).lb.fset(self, value)
File "/home/long/cobrapy/optlang/optlang/interface.py", line 691, in lb
self._check_valid_lower_bound(value)
File "/home/long/cobrapy/optlang/optlang/interface.py", line 631, in _check_valid_lower_bound
raise ValueError("Cannot set a lower bound that is greater than the upper bound.")
ValueError: Cannot set a lower bound that is greater than the upper bound.
Run in Python 3.4.3 on Linux64 with optlang.version.==1.1.2
Done: When symbolic bounds are merged into optlang devel for the glpk interface.
Just to keep track: would be awesome if optlang would use the new helper functions from swiglpk to get primals/duals and create intArrays/doubleArrays. This should give some speed boost.
There seems to be some issue with the sign of the dual for constraints.
If the definition of the dual is the change in the objective that would occur with a one unit increase in the right hand side of the constraint, then for the following problem:
from optlang import *
m=Model()
v=Variable('v')
c=Constraint(v,lb=0,ub=0)
m.add([v,c])
c.dual should be +1 when maximizing v and should be -1 when minimizing v. Instead, I get +1 in both instances. Here's the rest of the code with output:
m.objective=Objective(v,direction='max')
m.optimize()
# c.dual == 1
m.objective=Objective(v,direction='min')
m.optimize()
# c.dual == 1
This is a very simplified example, but I am having trouble figuring out the sign for the shadow price in more complicated examples.
From @matthiaskoenig on March 11, 2017 23:18
I just tried to use the cobrapy 0.6.0a2 with cplex and got the following errors on cobra import with cplex (see below).
I am using cplex 12.6.0. I assume that the new optlang implementation uses some features which are in cplex 12.7.0. The old 0.5.* cobrapy versions worked without problems with cplex 12.6.0.
I could not find any information which solver versions are supported by the new 0.6.0. I.e. which cplex, glpk and gurobi work with 0.6.0? It would be nice to have this information in the installation information or the documentation.
import cobra
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-1-da884c934071> in <module>()
----> 1 import cobra
/home/mkoenig/git/cobrapy/cobra/__init__.py in <module>()
9 from os.path import dirname as _dirname
10
---> 11 from cobra import design, flux_analysis, io
12 from cobra.core import (
13 DictList, Gene, Metabolite, Model, Object, Reaction, Species)
/home/mkoenig/git/cobrapy/cobra/design/__init__.py in <module>()
3 from __future__ import absolute_import
4
----> 5 from cobra.design.design_algorithms import *
/home/mkoenig/git/cobrapy/cobra/design/design_algorithms.py in <module>()
3 from __future__ import absolute_import
4
----> 5 from cobra.exceptions import DefunctError
6
7 __all__ = ("set_up_optknock", "dual_problem")
/home/mkoenig/git/cobrapy/cobra/exceptions.py in <module>()
3 from __future__ import absolute_import, print_function
4
----> 5 import optlang.interface
6
7
/usr/local/lib/python2.7/dist-packages/optlang/__init__.pyc in <module>()
31 # Load classes from preferred solver interface
32 if available_solvers['CPLEX']:
---> 33 from optlang.cplex_interface import Model, Variable, Constraint, Objective
34 elif available_solvers["GUROBI"]:
35 from optlang.gurobi_interface import Model, Variable, Constraint, Objective
/usr/local/lib/python2.7/dist-packages/optlang/cplex_interface.py in <module>()
108 cplex.Cplex.solution.status.solution_limit: interface.SPECIAL,
109 cplex.Cplex.solution.status.unbounded: interface.UNBOUNDED,
--> 110 cplex.Cplex.solution.status.relaxation_unbounded: interface.UNBOUNDED,
111 # 102: interface.OPTIMAL # The same as cplex.Cplex.solution.status.optimal_tolerance
112 }
AttributeError: SolutionStatus instance has no attribute 'relaxation_unbounded'
Copied from original issue: opencobra/cobrapy#453
My current problem in cobrapy is that when Reaction.flux
is accessed before optimization, I get solver-specific errors, like CplexSolverError
or GurobiError
. These are raised when accessing optlang.Variable.primal
, for example. I was wondering whether optlang
, which is aware of the solvers available, could catch these and raise a unified optlang.SolverError
(or similar) instead.
The problem is that cobrapy is not explicitly aware of the solvers available so I cannot import the errors in order to catch them specifically.
I tried install optlang using pip and I got an error about "requirements.txt" missing. This is the error:
Downloading/unpacking optlang
Downloading optlang-0.0.2.tar.gz
Running setup.py (path:/tmp/pip_build_root/optlang/setup.py) egg_info for package optlang
No handlers could be found for logger "optlang"
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/optlang/setup.py", line 27, in <module>
with open('requirements.txt') as fhandle:
IOError: [Errno 2] No such file or directory: 'requirements.txt'
Complete output from command python setup.py egg_info:
No handlers could be found for logger "optlang"
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/optlang/setup.py", line 27, in <module>
with open('requirements.txt') as fhandle:
IOError: [Errno 2] No such file or directory: 'requirements.txt'
Hi all,
I have to run a lot of FBA simulations and looking for places to speed them up. I am mainly interested in getting the reduced_costs, constraint_values, shadow_prices, ... as fast as possible from the model.
I had look at the interface.Model and wondered myself why are there OrderedDicts used for this values (in comparison to simple dict). This creates unnecessary overhead, costs performance and I don't see any benefits to it.
Or are there any algorithms relying on the order of the variables or constraint ids ?
If not all the OrderedDicts should be replaced with simple dicts in the interface.
In addition you should hash the names of your variables, constraints, ...
On every query of reduced costs, ... you reiterate and recalculate for instance the list of variable names
(variable.name for variable in self.variables)
There should be some property/field
variable_names
, constraint_names
on the Model class which are just updated when new variables are added/removed.
@property
def reduced_costs(self):
if self.is_integer:
raise ValueError("Dual values are not well-defined for integer problems")
try:
return collections.OrderedDict(
zip((variable.name for variable in self.variables), self.problem.solution.get_reduced_costs()))
except CplexSolverError as err:
raise SolverError(str(err))
@property
def constraint_values(self):
try:
return collections.OrderedDict(
zip((constraint.name for constraint in self.constraints), self.problem.solution.get_activity_levels()))
except CplexSolverError as err:
raise SolverError(str(err))
M
On the first try installing I get a print statement that says "No installers available." on module import:
moorepants@garuda:~$ conda create -n optlang-review ipython
Fetching package metadata ...........
Solving package specifications: ..........
Package plan for installation in environment /home/moorepants/miniconda3/envs/optlang-review:
The following NEW packages will be INSTALLED:
decorator: 4.0.10-py35_1
ipython: 5.1.0-py35_0
ipython_genutils: 0.1.0-py35_0
openssl: 1.0.2j-0
path.py: 8.2.1-py35_0
pexpect: 4.0.1-py35_0
pickleshare: 0.7.4-py35_0
pip: 9.0.1-py35_1
prompt_toolkit: 1.0.9-py35_0
ptyprocess: 0.5.1-py35_0
pygments: 2.1.3-py35_0
python: 3.5.2-0
readline: 6.2-2
setuptools: 27.2.0-py35_0
simplegeneric: 0.8.1-py35_1
six: 1.10.0-py35_0
sqlite: 3.13.0-0
tk: 8.5.18-0
traitlets: 4.3.1-py35_0
wcwidth: 0.1.7-py35_0
wheel: 0.29.0-py35_0
xz: 5.2.2-0
zlib: 1.2.8-3
Linking packages ...
[ COMPLETE ]|############################################################################| 100%
#
# To activate this environment, use:
# > source activate optlang-review
#
# To deactivate this environment, use:
# > source deactivate optlang-review
#
moorepants@garuda:~$ act optlang-review
(optlang-review) moorepants@garuda:~$ pip install optlang
Collecting optlang
Using cached optlang-1.0.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.9.0 in ./miniconda3/envs/optlang-review/lib/python3.5/site-packages (from optlang)
Collecting sympy>=1.0.0 (from optlang)
Collecting mpmath>=0.19 (from sympy>=1.0.0->optlang)
Installing collected packages: mpmath, sympy, optlang
Successfully installed mpmath-0.19 optlang-1.0.0 sympy-1.0
(optlang-review) moorepants@garuda:~$ ipython -c "import optlang"
No solvers available.
(optlang-review) moorepants@garuda:~$ ipython
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:53:06)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import optlang
No solvers available.
In [2]:
Do you really want to exit ([y]/n)?
(optlang-review) moorepants@garuda:~$ python -c "import optlang"
No solvers available.
I'd suggest having all required dependencies installed automatically via the pip dependency resolution. Also, shouldn't import optlang
raise and ImportError if the dependencies are not installed?
In the following model (zipped JSON) you have a constraint "test" whose primal will have the wrong sign after optimizing.
File: bad.zip
Basically you get stuff like this when having constraints with only a lower bound:
In [95]: mod.solver.optimize()
'optimal'
In [96]: print(mod.constraints.test)
test: 0.0 <= -1.0*Biomass_Ecoli_core_reverse_2cdba + 1.0*Biomass_Ecoli_core
In [97]: mod.constraints.test.primal
Out[97]: -0.8739215069684309
I think the problem is that optlang returns the slack variable from the associated optlang interface, however those change their sign depending on the type of constraint (negative for >= and positive for <=)...
So the following code will raise an "invalid expression" error with the GLPK interface but run fine with CPLEX and give a wrong result.
In [11]: model.objective.expression
Out[11]: -1.0*Biomass_Ecoli_core_reverse_2cdba + 1.0*Biomass_Ecoli_core
In [13]: model.optimize()
Out[13]: 'optimal'
In [15]: model.objective.value
Out[15]: 0.8739215069684304
In [17]: model.objective += 1.0 # error with glpk_interface
In [18]: model.optimize()
Out[18]: 'optimal'
In [19]: model.objective.value
Out[19]: 0.8739215069684304
In [20]: model.objective.expression
Out[20]: -1.0*Biomass_Ecoli_core_reverse_2cdba + 1.0*Biomass_Ecoli_core + 1.0
What is the motivation to prohibit adding constants to expressions in optlang? The case here is pretty obvious but there are some more subtle ones like optimizing distances to a reference solution. For instance
minimizing (x_known - x)**2) won't work in optlang since the expanded expression includes adding a constant (this objective is used in the method of global criterion for instance).
Like there are set_linear_coefficients it might be useful to have a get_linear_coefficients method.
So adding integer variables to a Cplex Model and removing them afterwards kind of breaks the optlang Model:
In [26]: mod.solver.is_integer
Out[26]: False
In [27]: mod.solver.optimize()
Out[27]: 'optimal'
In [28]: mod.solver.reduced_costs
---------------------------------------------------------------------------
CplexSolverError Traceback (most recent call last)
~/.local/lib/python3.6/site-packages/optlang/cplex_interface.py in _get_reduced_costs(self)
758 try:
--> 759 return self.problem.solution.get_reduced_costs()
760 except CplexSolverError as err:
~/.local/lib/python3.6/site-packages/cplex/_internal/_subinterfaces.py in get_reduced_costs(self, *args)
7715 return apply_freeform_two_args(
-> 7716 getdj, self._cplex.variables.get_indices, args)
7717
~/.local/lib/python3.6/site-packages/cplex/_internal/_aux_functions.py in apply_freeform_two_args(fn, convert, args)
116 elif len(args) == 0:
--> 117 return fn(0)
118 else:
~/.local/lib/python3.6/site-packages/cplex/_internal/_subinterfaces.py in getdj(a, b)
7713 def getdj(a, b = self._cplex.variables.get_num() - 1):
-> 7714 return CPX_PROC.getdj(self._env._e, self._cplex._lp, a, b)
7715 return apply_freeform_two_args(
~/.local/lib/python3.6/site-packages/cplex/_internal/_procedural.py in getdj(env, lp, begin, end)
1602 status = CR.CPXXgetdj(env, lp, dj, begin, end)
-> 1603 check_status(env, status)
1604 return LAU.array_to_list(dj, djlen)
~/.local/lib/python3.6/site-packages/cplex/_internal/_procedural.py in __call__(self, env, status, from_cb)
269 error_string = geterrorstring(env, status)
--> 270 raise CplexSolverError(error_string, env, status)
271
CplexSolverError: CPLEX Error 1017: Not available for mixed-integer problems.
During handling of the above exception, another exception occurred:
SolverError Traceback (most recent call last)
<ipython-input-28-4f8266bcf3ca> in <module>()
----> 1 mod.solver.reduced_costs
~/.local/lib/python3.6/site-packages/optlang/interface.py in reduced_costs(self)
1229 """
1230 return collections.OrderedDict(
-> 1231 zip(self._get_variables_names(), self._get_reduced_costs())
1232 )
1233
~/.local/lib/python3.6/site-packages/optlang/cplex_interface.py in _get_reduced_costs(self)
759 return self.problem.solution.get_reduced_costs()
760 except CplexSolverError as err:
--> 761 raise SolverError(str(err))
762
763 def _get_constraint_values(self):
SolverError: CPLEX Error 1017: Not available for mixed-integer problems.
To support bi-level MILPs like OptKnock, it would be nice to add a convenience method for taking the dual of a problem and combining it with the original problem.
See here for COBRApy implementation:
Doing this in COBRApy, I would up with a bug that prevents me from implementing the 3-level optimization RobustKnock, and I gave up on it. Maybe OptLang provides a core library that will be easier to debug.
Hi!
I'm stuck with a "maximum recursion depth exceeded in comparison" when creating a quadratic objective.
Code:
from cameo import models
from cameo.flux_analysis import moma
model = models.bigg.iJO1366
ref = model.solve().fluxes
model.reactions.PGI.knock_out()
res = moma(model, reference=ref)
Error:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-8-fecd89f0a3e4> in <module>()
----> 1 res = moma(model, reference=ref)
/Users/joao/Documents/repos/cameo/cameo/flux_analysis/simulation.py in moma(model, reference, cache, reactions, *args, **kwargs)
193 sloppy=True)
194
--> 195 cache.add_objective(create_objective, None, cache.variables.values())
196
197 solution = model.solve()
/Users/joao/Documents/repos/cameo/cameo/util.py in add_objective(self, create, update, *args)
184 def add_objective(self, create, update, *args):
185 if self.objective is None:
--> 186 self.objective = create(self.model, *args)
187 self.time_machine(
188 do=partial(setattr, self.model, 'objective', self.objective),
/Users/joao/Documents/repos/cameo/cameo/flux_analysis/simulation.py in create_objective(model, variables)
191 return model.solver.interface.Objective(Add(*[FloatOne * var ** 2 for var in variables]),
192 direction="min",
--> 193 sloppy=True)
194
195 cache.add_objective(create_objective, None, cache.variables.values())
/Users/joao/.virtualenvs/cameo-py3/lib/python3.4/site-packages/optlang/cplex_interface.py in __init__(self, *args, **kwargs)
302 super(Objective, self).__init__(*args, **kwargs)
303 self._expression_expired = False
--> 304 if not (self.is_Linear or self.is_Quadratic):
305 raise ValueError("Cplex only supports linear and quadratic objectives.")
306
/Users/joao/.virtualenvs/cameo-py3/lib/python3.4/site-packages/optlang/interface.py in is_Linear(self)
464 return True
465 else:
--> 466 poly = self.expression.as_poly(*self.variables)
467 if poly is not None:
468 return poly.is_linear
/Users/joao/.virtualenvs/cameo-py3/lib/python3.4/site-packages/sympy/core/basic.py in as_poly(self, *gens, **args)
695
696 try:
--> 697 poly = Poly(self, *gens, **args)
698
699 if not poly.is_Poly:
/Users/joao/.virtualenvs/cameo-py3/lib/python3.4/site-packages/sympy/polys/polytools.py in __new__(cls, rep, *gens, **args)
87 return cls._from_poly(rep, opt)
88 else:
---> 89 return cls._from_expr(rep, opt)
90
91 @classmethod
/Users/joao/.virtualenvs/cameo-py3/lib/python3.4/site-packages/sympy/polys/polytools.py in _from_expr(cls, rep, opt)
198 """Construct a polynomial from an expression. """
199 rep, opt = _dict_from_expr(rep, opt)
--> 200 return cls._from_dict(rep, opt)
201
202 def _hashable_content(self):
/Users/joao/.virtualenvs/cameo-py3/lib/python3.4/site-packages/sympy/polys/polytools.py in _from_dict(cls, rep, opt)
147 rep[monom] = domain.convert(coeff)
148
--> 149 return cls.new(DMP.from_dict(rep, level, domain), *gens)
150
151 @classmethod
/Users/joao/.virtualenvs/cameo-py3/lib/python3.4/site-packages/sympy/polys/polyclasses.py in from_dict(cls, rep, lev, dom)
257 def from_dict(cls, rep, lev, dom):
258 """Construct and instance of ``cls`` from a ``dict`` representation. """
--> 259 return cls(dmp_from_dict(rep, lev, dom), dom, lev)
260
261 @classmethod
/Users/joao/.virtualenvs/cameo-py3/lib/python3.4/site-packages/sympy/polys/densebasic.py in dmp_from_dict(f, u, K)
1013
1014 if coeff is not None:
-> 1015 h.append(dmp_from_dict(coeff, v, K))
1016 else:
1017 h.append(dmp_zero(v))
... last 1 frames repeated, from the frame below ...
/Users/joao/.virtualenvs/cameo-py3/lib/python3.4/site-packages/sympy/polys/densebasic.py in dmp_from_dict(f, u, K)
1013
1014 if coeff is not None:
-> 1015 h.append(dmp_from_dict(coeff, v, K))
1016 else:
1017 h.append(dmp_zero(v))
RuntimeError: maximum recursion depth exceeded in comparison
Otherwise there will be cyclic referencing. Adding __del__
methods to handle solver-specific cleanup routines will prevent the garbage collector from collecting these cycles.
Should solve #125 and other problems.
Hi,
the appended MIP problem solves fine with the glpk or cplex interface but is declared infeasible when using the gurobi interface. Tested Gurobi version was 7.02.
This seems like a perfectly valid problem setup:
Maximize x
subject to:
x<= 3.5
x must be integer
Of course the solution is 3... but optlang will tell you it is not a valid setup. perhaps this should be changed to a warning?
Best,
Josh
I don't see a suggestive branch name but has there been any preliminary work to support the MOSEK Python API? Otherwise I figure there's no better way for me to get to know the optlang interface than implementing one myself ๐
Hi,
I am regularly getting the following errors:
---------------------------------------------------------------------------
RecursionError Traceback (most recent call last)
<timed exec> in <module>()
~/code/micom/micom/media.py in minimal_medium(community, community_growth, min_growth, minimize_components, open_exchanges)
159 medium[rxn.id] = -flux
160 elif not export and flux > 0:
--> 161 medium[rxn.id] = flux
162
163 return medium
~/code/cobrapy/cobra/core/model.py in __exit__(self, type, value, traceback)
971 """Pop the top context manager and trigger the undo functions"""
972 context = self._contexts.pop()
--> 973 context.reset()
974
975 def merge(self, right, prefix_existing=None, inplace=True,
~/code/cobrapy/cobra/util/context.py in reset(self)
34 while self._history:
35 entry = self._history.pop()
---> 36 entry()
37
38
~/code/cobrapy/cobra/util/solver.py in reset()
157 if context:
158 def reset():
--> 159 model.solver.objective = reverse_value
160 model.solver.objective.direction = reverse_value.direction
161
~/.local/lib/python3.6/site-packages/optlang/gurobi_interface.py in objective(self, value)
551 @objective.setter
552 def objective(self, value):
--> 553 super(Model, self.__class__).objective.fset(self, value)
554 expression = self._objective._expression
555
~/.local/lib/python3.6/site-packages/optlang/interface.py in objective(self, value)
1180 @objective.setter
1181 def objective(self, value):
-> 1182 self.update()
1183 # try: # Is this except ever needed?
1184 for atom in sorted(value.expression.atoms(Variable), key=lambda v: v.name):
~/.local/lib/python3.6/site-packages/optlang/gurobi_interface.py in update(self)
581
582 def update(self):
--> 583 super(Model, self).update(callback=self.problem.update)
584
585 def _optimize(self):
~/.local/lib/python3.6/site-packages/optlang/interface.py in update(self, callback)
1441 rm_var = self._pending_modifications.rm_var
1442 if len(rm_var) > 0:
-> 1443 self._remove_variables(rm_var)
1444 self._pending_modifications.rm_var = []
1445 callback()
~/.local/lib/python3.6/site-packages/optlang/gurobi_interface.py in _remove_variables(self, variables)
611 def _remove_variables(self, variables):
612 # Not calling parent method to avoid expensive variable removal from sympy expressions
--> 613 self.objective._expression = self.objective.expression.xreplace({var: 0 for var in variables})
614 for variable in variables:
615 internal_variable = variable._internal_variable
~/.local/lib/python3.6/site-packages/optlang/interface.py in expression(self)
453 def expression(self):
454 """The mathematical expression defining the objective/constraint."""
--> 455 return self._get_expression()
456
457 @property
~/.local/lib/python3.6/site-packages/optlang/gurobi_interface.py in _get_expression(self)
363 terms = []
364 for i in range(grb_obj.size()):
--> 365 terms.append(grb_obj.getCoeff(i) * self.problem.variables[grb_obj.getVar(i).getAttr('VarName')])
366 expression = sympy.Add._from_args(terms)
367 # TODO implement quadratic objectives
~/.local/lib/python3.6/site-packages/optlang/interface.py in variables(self)
1199 def variables(self):
1200 """The model variables."""
-> 1201 self.update()
1202 return self._variables
1203
... last 6 frames repeated, from the frame below ...
~/.local/lib/python3.6/site-packages/optlang/gurobi_interface.py in update(self)
581
582 def update(self):
--> 583 super(Model, self).update(callback=self.problem.update)
584
585 def _optimize(self):
RecursionError: maximum recursion depth exceeded while calling a Python object
Happens when adding MIP variables and constraints/objectives to a model followed by removing all of those again after solving.
There seems to be an issue regarding the order of rows/cols provided for GLPK.
Running the exakt same input twice, GLPK guarantees to give the exact same output. This applies to the objective value as well as the variable values.
Anyway, feeding optlang/GLPK with the exact same data twice does in fact result in the same objective value, but the variable values differ over runs if there is more than one combination of variable values leading to an optimal objective value.
See this gist as an example binary problem where any combination of '1's in a matrix (subject to any row's sum being <= 1) leads to an optimal solution. Running this sample code several times shows how the result matrix differs.
I think, regarding determinism, optlang should provide the same output given the same input across runs.
Any thoughts on this?
Hi, I experienced that modifying the objective in the cplex interface scales badly.
For a model with about variables setting the objective takes about half of the time when optimizing (see "cumtime" column which is the time the function call and all sub-calls takes):
1174604 function calls (1050578 primitive calls) in 0.698 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.698 0.698 {built-in method builtins.exec}
1 0.000 0.000 0.698 0.698 <string>:1(<module>)
1 0.000 0.000 0.698 0.698 parsimonious.py:27(pfba)
1 0.003 0.003 0.698 0.698 parsimonious.py:138(_pfba_optlang)
1 0.000 0.000 0.471 0.471 parsimonious.py:102(add_pfba)
2 0.000 0.000 0.415 0.207 cplex_interface.py:713(objective)
4 0.000 0.000 0.295 0.074 _subinterfaces.py:3852(set_linear)
However for a model with ~200.000 variables it now takes 95% of the time.
40679344 function calls (36037663 primitive calls) in 257.649 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 257.649 257.649 {built-in method builtins.exec}
1 0.000 0.000 257.649 257.649 <string>:1(<module>)
1 0.000 0.000 257.649 257.649 community.py:275(optimize)
2 0.015 0.007 249.575 124.788 cplex_interface.py:713(objective)
4 0.009 0.002 244.498 61.125 _subinterfaces.py:3852(set_linear)
I did not observe the same effect with the other solver interfaces.
Hi all,
I just installed the latest cplex 12.7.1.
Somehow optlang does not work with it. Looks like some attributes changed on the Cplex object.
In [2]: import optlang
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-2-b90a99bbabb7> in <module>()
----> 1 import optlang
/usr/local/lib/python2.7/dist-packages/optlang/__init__.pyc in <module>()
31 # Load classes from preferred solver interface
32 if available_solvers['CPLEX']:
---> 33 from optlang.cplex_interface import Model, Variable, Constraint, Objective
34 elif available_solvers["GUROBI"]:
35 from optlang.gurobi_interface import Model, Variable, Constraint, Objective
/usr/local/lib/python2.7/dist-packages/optlang/cplex_interface.py in <module>()
116 _CPLEX_STATUS_TO_STATUS = {}
117 for status_name, optlang_status in _STATUS_MAP.items():
--> 118 cplex_status = getattr(cplex.Cplex.solution.status, status_name, None)
119 if cplex_status is not None:
120 _CPLEX_STATUS_TO_STATUS[cplex_status] = optlang_status
AttributeError: type object 'Cplex' has no attribute 'solution'
py2.7, linux x64, optlang 1.1.3
The best
Matthias
For objectives it returns zero coefficients for the passed variables for Constraints it does not. I think the problem is that glp_get_mat_row
does not return indices or values for zero entries.
In [20]: model.constraints[0].get_linear_coefficients(model.variables)
Out[20]:
{0 <= PGK <= 1000.0: 1.0,
0 <= GAPD_reverse_459c1 <= 1000.0: -1.0,
0 <= PGK_reverse_02696 <= 1000.0: -1.0,
0 <= GAPD <= 1000.0: 1.0}
In [21]: model.objective.get_linear_coefficients(model.variables)
Out[21]:
{0.0 <= PFL <= 1000.0: 0.0,
0 <= ATPS4r <= 1000.0: 0.0,
...,
0 <= D_LACt2 <= 1000.0: 0.0}
Things that work on Linux and Mac might fail on Windows :)
It seems like symengine can be installed quite easily with conda now.
Good day optlang people !
I stumbled upon something that I might call an issue, but it might as well be a design choice from your side. I use MILPs, which contain binary variables. Sometimes, I need to constrain my model by assigning the value of the binary variable to either 1, or 0. However, in optlang.interface
, lines 111-134, the methods __test_valid_lower_bound
and __test_valid_upper_bound
forbid such a thing by throwing a ValueError
.
So far, my fix has been to use integer variables in the stead of binary variables, but I fear it might have an impact on the solver behavior, as some heuristics are specifically tailored to binary problems.
What say you, would it be a design choice or a small issue to fix ?
(That being said, I really love your software!)
Cheers,
Pierre
Hello,
I started using cplex with cameo.py two month ago, and pickled a lot of objects. It seems that cplex_interface.py use the functions getstate and setstate to transfer a cplex problem to/from bites, creating temp files .sav in the process. But these .sav files are not deleted afterward and using cameo for two month get me 200 000 files. sav, taking a total space of 409 Go in my hard drive.
Unless I misunderstand the use of these temp files, you should probably erase them after use?
when having cplex in site-packages on py35,36 optlang tries to create a cplex interface but fails and throws exception
Traceback (most recent call last):
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/_pytest/config.py", line 325, in _getconftestmodules
return self._path2confmods[path]
KeyError: local('/home/travis/build/biosustain/cameo/tests')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/_pytest/config.py", line 356, in _importconftest
return self._conftestpath2mod[conftestpath]
KeyError: local('/home/travis/build/biosustain/cameo/tests/conftest.py')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/_pytest/config.py", line 362, in _importconftest
mod = conftestpath.pyimport()
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/py/_path/local.py", line 650, in pyimport
__import__(modname)
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/_pytest/assertion/rewrite.py", line 216, in load_module
py.builtin.exec_(co, mod.__dict__)
File "/home/travis/build/biosustain/cameo/tests/conftest.py", line 3, in <module>
import cobra.test
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/cobra/__init__.py", line 11, in <module>
from cobra import design, flux_analysis, io
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/cobra/design/__init__.py", line 5, in <module>
from cobra.design.design_algorithms import *
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/cobra/design/design_algorithms.py", line 5, in <module>
from cobra.exceptions import DefunctError
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/cobra/exceptions.py", line 5, in <module>
import optlang.interface
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/optlang/__init__.py", line 33, in <module>
from optlang.cplex_interface import Model, Variable, Constraint, Objective
File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/optlang/cplex_interface.py", line 118, in <module>
cplex_status = getattr(cplex.Cplex.solution.status, status_name, None)
AttributeError: module 'cplex' has no attribute 'Cplex'
ERROR: could not load /home/travis/build/biosustain/cameo/tests/conftest.py
Better behaviour would be to not make the cplex interface at all
_round_primal_to_bounds
uses its own absolute tolerance of 1e-5 but the solvers usually use a relative and absolute tolerance around 1e-9. For the case of large upper bounds the solver tolerance will be rtol*ub + atol which is 1e-9 * 1e6 + 1e-9 ~ 1e-3 > 1e-5. However the solution is completely feasible given the set solver tolerances.
_round_primal_to_bounds
should either respect the solver tolerances or simply not check bounds feasibility since this is done by all lower level solver interfaces anyways.
With appended model the scipy interface declares the problem as infeasible even if all other interfaces find the correct optimal solution (0.874).
import json
import optlang
with open("textbook.json") as f:
m = optlang.scipy_interface.Model.from_json(json.load(f))
m.optimize()
esolver is an exact solver and cobrapy currently supports an interface to solve metabolic models using it. Since we are shifting to only using optlang interfaces in cobrapy, I raise this issue to help people comment and upvote this issue if they feel they would benefit from esolver in optlang as well.
When formulating a problem with Variable objects of type integer, optimized outputted values for those variables are returned as floating point representations of the integer results. Is this behavior expected (as opposed to outputting integer values)?
Thanks!
~ Matej
Caused by the tolerances which contain lambda functions (lambdas can never be pickled in Python).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.