Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to run multi_obj_with_constraints.ipynb #28

Open
R-M-Lee opened this issue Nov 23, 2023 · 5 comments
Open

Unable to run multi_obj_with_constraints.ipynb #28

R-M-Lee opened this issue Nov 23, 2023 · 5 comments

Comments

@R-M-Lee
Copy link

R-M-Lee commented Nov 23, 2023

Hi Nathan @spiralulam

I cloned the master branch and did
pip install .
then installed glpk and scip via conda as written in the docs. Then I realized I still did not have pyomo or gurobipy, so I did
pip install -r requirements.txt
and everything seemed good to go. But when I run multi_obj_with_constraints.ipynb (the first one I tried) I got errors with both the Gurobi and pyomo versions at the lines
res_gur = opt_gur.solve(enting, model_core=model_gur)
and / or
res_pyo = opt_pyo.solve(enting, model_core=model_pyo)

The error (for the pyomo version) is:

AttributeError                            Traceback (most recent call last)
Untitled-1 in line 1
----> [88](untitled:Untitled-1?line=87) res_pyo = opt_pyo.solve(enting, model_core=model_pyo)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\optimizers\pyomo_opt.py:96](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py:96), in PyomoOptimizer.solve(self, tree_model, model_core, weights)
     [93](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=92)         opt.options[k] = v
     [95](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=94) # build pyomo model using information from tree model
---> [96](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=95) tree_model.add_to_pyomo_model(opt_model)
     [98](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=97) # Solve optimization model
     [99](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=98) opt.solve(opt_model, tee=True)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\enting.py:209](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py:209), in Enting.add_to_pyomo_model(self, core_model, weights)
    [204](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=203)     core_model.constraint_link_mu_auxmu = pyo.Constraint(
    [205](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=204)         expr=core_model._aux_mu[obj_name] == core_model._mu
    [206](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=205)     )
    [207](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=206) else:
    [208](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=207)     # multi-objective case
--> [209](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=208)     self.mean_model.add_to_pyomo_model(
    [210](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=209)         core_model, add_mu_var=True, normalize_mean=True
    [211](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=210)     )
    [212](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=211)     if weights is not None:
    [213](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=212)         moo_weights = weights

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\tree_ensemble.py:402](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py:402), in TreeEnsemble.add_to_pyomo_model(self, model, add_mu_var, normalize_mean)
    [397](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=396) def add_to_pyomo_model(
    [398](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=397)     self, model, add_mu_var: bool = True, normalize_mean: bool = False
    [399](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=398) ):
    [400](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=399)     import pyomo.environ as pyo
--> [402](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=401)     self._update_meta_tree_dict()
    [404](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=403)     # attach tree info
    [405](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=404)     model._num_trees = lambda obj_name: self._meta_tree_dict[obj_name].num_trees

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\tree_ensemble.py:132](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py:132), in TreeEnsemble._update_meta_tree_dict(self)
    [126](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=125)     raise IOError(
    [127](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=126)         "Parameter 'train_lib' for tree ensembles needs to be "
    [128](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=127)         "in '('lgbm')'."
    [129](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=128)     )
    [131](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=130) # order tree_model_dict
--> [132](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=131) ordered_tree_model_dict = read_lgbm_tree_model_dict(
    [133](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=132)     lib_out, cat_idx=self._problem_config.cat_idx
    [134](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=133) )
    [136](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=135) # populate meta_tree_model
    [137](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=136) self._meta_tree_dict[obj.name] = MetaTreeModel(ordered_tree_model_dict)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:11](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:11), in read_lgbm_tree_model_dict(tree_model_dict, cat_idx)
      [8](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=7) node_list = []
     [10](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=9) # populate node_list and add to ordered_tree_list if non-empty
---> [11](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=10) add_next_nodes(node_list=node_list, node=root_node, cat_idx=cat_idx)
     [13](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=12) if node_list:
     [14](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=13)     ordered_tree_list.append(node_list)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:56](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:56), in add_next_nodes(node_list, node, cat_idx)
     [53](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=52) except KeyError:
     [54](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=53)     pass
---> [56](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=55) add_next_nodes(node_list, node, cat_idx=cat_idx)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:56](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:56), in add_next_nodes(node_list, node, cat_idx)
     [53](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=52) except KeyError:
     [54](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=53)     pass
---> [56](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=55) add_next_nodes(node_list, node, cat_idx=cat_idx)

    [... skipping similar frames: add_next_nodes at line 56 (3 times)]

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:56](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:56), in add_next_nodes(node_list, node, cat_idx)
     [53](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=52) except KeyError:
     [54](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=53)     pass
---> [56](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=55) add_next_nodes(node_list, node, cat_idx=cat_idx)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:32](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:32), in add_next_nodes(node_list, node, cat_idx)
     [29](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=28)     new_node["split_code_pred"] = temp_node_val
     [30](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=29) else:
     [31](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=30)     # read categorical variables
---> [32](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=31)     cat_set = node[-1]["threshold"].split("||")
     [33](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=32)     temp_node_val = [int(cat) for cat in cat_set]
     [34](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=33)     new_node["split_code_pred"] = temp_node_val

AttributeError: 'float' object has no attribute 'split'`

Any idea what is missing here?
@spiralulam
Copy link
Collaborator

spiralulam commented Nov 23, 2023 via email

@spiralulam
Copy link
Collaborator

I fixed now that you had to install gurobipy/pyomo. This has to be specified in setup.py, I thought there would be a reference to requirements.txt, but there is a very lengthy and well rated post (that I haven't read) on stackoverflow where someone explains that this duplication makes sense because requirements.txt and setup.py serve different purposes.

@spiralulam
Copy link
Collaborator

It should work now, please test again, @R-M-Lee . The problem was the lightgbm version which is now fixed to 4.0.0. Will open an issue for that one.

@R-M-Lee
Copy link
Author

R-M-Lee commented Nov 29, 2023

Thanks for looking into this. Now it runs with glpk and gurobi, but there is something wrong with the scip installation (I did it via conda as described in the docs)

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
c:\Users\user\dev\entmoot_test\entmoot\docs\notebooks\multi_obj_with_constraints.ipynb Cell 15 line 1
     14 params_pyomo = {"solver_name": "SCIP"}
     15 opt_pyo = PyomoOptimizer(problem_config, params=params_pyomo)
---> 17 res_pyo = opt_pyo.solve(enting, model_core=model_pyo)
     18 x_opt, y_opt, z_opt = res_pyo.opt_point[3:]
     20 assert round(x_opt, 5) == round(y_opt, 5) and round(y_opt, 5) == round(z_opt, 5)

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\entmoot\optimizers\pyomo_opt.py:99, in PyomoOptimizer.solve(self, tree_model, model_core, weights)
     96 tree_model.add_to_pyomo_model(opt_model)
     98 # Solve optimization model
---> 99 opt.solve(opt_model, tee=True)
    101 # update current solution
    102 self._curr_sol, self._active_leaves = self._get_sol(opt_model)

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\opt\base\solvers.py:635, in OptSolver.solve(self, *args, **kwds)
    629 if self._report_timing:
    630     print(
    631         "      %6.2f seconds required for solver"
    632         % (solve_completion_time - presolve_completion_time)
    633     )
--> 635 result = self._postsolve()
    636 result._smap_id = self._smap_id
    637 result._smap = None

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\solvers\plugins\solvers\ASL.py:230, in ASL._postsolve(self)
    228 #
    229 self._instance = None
--> 230 return SystemCallSolver._postsolve(self)

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\opt\solver\shellcmd.py:291, in SystemCallSolver._postsolve(self)
    288 results = None
    290 if self._results_format is not None:
--> 291     results = self.process_output(self._rc)
    292     #
    293     # If keepfiles is true, then we pop the
    294     # TempfileManager context while telling it to
    295     # _not_ remove the files.
    296     #
    297     if not self._keepfiles:
    298         # in some cases, the solution filename is
    299         # not generated via the temp-file mechanism,
    300         # instead being automatically derived from
    301         # the input lp/nl filename. so, we may have
    302         # to clean it up manually.

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\opt\solver\shellcmd.py:396, in SystemCallSolver.process_output(self, rc)
    389     results = self._results_reader(
    390         self._results_file,
    391         res=results,
    392         soln=results.solution(0),
    393         suffixes=self._suffixes,
    394     )
    395 else:
--> 396     results = self._results_reader(
    397         self._results_file, res=results, suffixes=self._suffixes
    398     )
    399 results_reader_completion_time = time.time()
    400 if self._report_timing is True:

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\opt\plugins\sol.py:40, in ResultsReader_sol.__call__(self, filename, res, soln, suffixes)
     36 """
     37 Parse a *.sol file
     38 """
     39 try:
---> 40     with open(filename, "r") as f:
     41         return self._load(f, res, soln, suffixes)
     42 except ValueError as e:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Local\\Temp\\tmpi0kx1olr.pyomo.sol'

@spiralulam
Copy link
Collaborator

There seems to be no way to use SCIP after a conda installation without changing some code snippets. I traced it down to this part of Pyomo's code:
image
Up to now, we use SolverFactory("gurobi"), SolverFactory("glpk") and so on... BUT scip requires the use of a different interface, namely SolverFactory(solver='scip', solver_io='nl'), see here (comment of Matthias Fripp https://stackoverflow.com/questions/56885640/problems-in-interfacing-scip-with-pyomo/56886542#56886542) and this is unfortunately handled in a very different way (see screenshot above). So unfortunately no SCIP for now...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@spiralulam @R-M-Lee