Related: openjournals/joss-reviews#2974 and #23

The __repr__ of OneDimensionalSolution currently only shows the objective values without any indication it is a more complex object.

This does not follow Python's recommended usage of the __repr__ method and goes against the Zen of Python's line:

Explicit is better than implicit

I would strongly recommend changing it such that it can be round-tripped using eval(repr(x)) for any x = OneDimensionalSolution(...)

Related: openjournals/joss-reviews#2974

Currently the required packages for building the documentation are only manually specified in the github actions docs.yml file.
A simple requirements-dev.txt with those packages (and corresponding install instruction) would help potential contributors to work on the project locally.

Related: openjournals/joss-reviews#2974, #10

Related: openjournals/joss-reviews#2974 and #23

The Solution-class used in this framework as a default output is not so intuitive/standard that it can be included unexplained.
A (simple) example of how to extract and use each type of returned/stored data for typical uses would be very helpful, so something like

• plotting a typical 2D pareto-front
• obtaining the particular solution for some f (to use your nomenclature)
• using the included distance attribute

Using https://documentation.divio.com/ terminology, ideally you would also include some explanation in a 'tutorial', but I think that's more optional than adding the list above as a set of 'how-to's.

Related: openjournals/joss-reviews#2974 and #15

See e.g. the difference between

Related: openjournals/joss-reviews#2974

Some modules have pretty low coverage. Since you are working in a research context consider to increase the test coverage so the software could be trusted. It would be nice to provide a badge in the read me regarding the code coverage.

Below I list some files:

Name                                                                                  Stmts   Miss  Cover   Missing
-------------------------------------------------------------------------------------------------------------------
moead_framework\tool\result.py                                                          249    151    39%   5-24, 28-38, 42-52, 56-69, 73-75, 80-89, 93-100, 104-109, 146, 175, 179, 195-243, 265, 268, 288, 299-310, 314, 318-324, 328-333, 348-355, 362-366
moead_framework\core\genetic_operator\numerical\moead_de_operators.py                    14      7    50%   12-22
moead_framework\tool\mop.py                                                              68     24    65%   25-27, 40, 78-88, 110-124
moead_framework\solution\one_dimension_solution.py                                       15      4    73%   13, 16, 19, 22
moead_framework\core\sps_strategy\sps_dra.py                                             34      9    74%   37-47
moead_framework\aggregation\functions.py                                                  8      2    75%   19, 29
moead_framework\problem\problem.py                                                       13      3    77%   12, 16, 20
moead_framework\core\genetic_operator\abstract_operator.py                                9      2    78%   11, 15

Related: openjournals/joss-reviews#2974

For example here is the instruction

init()
Initialize self. See help(type(self)) for accurate signature.

which gives the following unrelated result in a terminal:

>>> from moead_framework.aggregation import Tchebycheff
>>> help(type(Tchebycheff))
Help on class ABCMeta in module abc:

class ABCMeta(builtins.type)
[...]

Related: openjournals/joss-reviews#2974

Docs should state the minimum (and the expected maximum) required versions so the package would be able to work.

Related: openjournals/joss-reviews#2974

The word 'generate' is currently used with different meanings.

Definitions

Let me first clarify how I typically interpret certain words in the context of evolutionary optimization:

• Generation: to newly create some candidate or feasible solution x according to some procedure, such as a new random sample
• Evaluation: to determine/calculate the fitness f(x) according to one or more objectives f

My confusion

For example in the Problem tutorial where it introduces the example function as:

The function generate_solution(array, evaluate) allows to generate a solution.

To me, this function creates a Solution object, containing x and additionally the evaluation f(x) if evaluate=True is given, but no x is generated, since the x is given as the array parameter, which has to have been generated beforehand.

However, the generate_random_solution(evaluate) method listed further on does generate a new x, in addition to performing the other two steps.

A potential solution

I think it would be clearest if it were refactored to something like this:

class Problem():
    def __init__(self, ...):
        ... # unchanged from current implementation

    def f(self, function_id, solution):
        ... # unchanged from current implementation

    def evaluate(self, solution):
        """evaluate a feasible solution on all objectives"""
        if not isinstance(solution, OneDimensionSolution):
            solution = OneDimensionSolution(solution)
        solution.F = [self.f(j, solution) for j in range(self.number_of_objective)]
        return solution
    
    def generate_random_solution(self):
        """create new random feasible solution according to problem domain"""
        ...  # only create the new feasible solution x
        return OneDimensionSolution(x)

This way, each of the methods evaluate and generate_random_solution does exactly one thing without 'overloading' the word 'generate'. Importantly, the encapsulation of x in a OneDimensionSolution object should either be automatic so a user does not have to think about it, or should be so explicit (if e.g. the user wants to use a different Solution class) that it does not belong inside the definition of a Problem.

In the latter case, solution = OneDimensionSolution(solution) should be replaced with a raise ValueError('evaluate() expects a Solution-derived object as input') and generate_random_solution() should just return x.

Discussion

I obviously don't know how large the ripple effects of such a refactor would be throughout the rest of the codebase, but I feel this is the most long-term solution to the confusion I encountered regarding the use of the word 'generate'.

I don't want to be too pedantic about this, so @geoffreyp and @chkoar please let me know if anything is unclear, what you think of this suggestion and if you think this could more easily be cleared up by clarifying relevant documentation instead.

Related: openjournals/joss-reviews#2974

See here.

Related: openjournals/joss-reviews#2974

Currently, the statement of need centers on other (Python) packages pymoo, pygmo and jMetal not providing a modular interface, and R package MOEADr only doing so in R. However, this doesn't answer the question:

'If MOEADr already exists and provides the desired functionality, why is it not sufficient?'

Some more detailed questions to guide the level of detail for what I think should be in the statement of need:

• Is there a fundamental problem with the MOEADr R package?
• Or is the main goal to bring it's functionality to Python?
  • If so, what are the (significant) benefits over using MOEADr in combination with e.g. rpy2?
  • Or is there a significant difference in support in the wider ecosystem such as benchmark problem implementations?
• Does moead-framework fundamentally improve on MOEADr somehow, e.g. by implementing more modules?
• ...

Just to note: I much prefer Python over R myself, so I think it's perfectly fine if the desire to program in Python is the main reason, but that should be clearly stated and motivated in the paper. Otherwise, (further) development effort might be better spent on creating/improving a Python interface for an existing implementation such as MOEADr.

Related: openjournals/joss-reviews#2974

As a quick experiment, I tried to run all possible combinations of components, but many fail due to code incompatibliities.

Looking at the documentation, it is not immediately clear if some components cannot work together with others. For example, for moead_framework.core.sps_strategy.sps_dra.SpsDra, the description is 'The strategy used in MOEA/D-DRA', but does that mean it exclusively works with that algorithm?

Can you explain to what extent it is the goal of your framework to enable all such combinations?

from moead_framework.aggregation import Tchebycheff, WeightedSum
aggregations = [Tchebycheff, WeightedSum]

from moead_framework.core.genetic_operator.combinatorial import BinaryMutation, Crossover, CrossoverAndMutation
genetic_operators = [CrossoverAndMutation, BinaryMutation, Crossover]

from moead_framework.core.parent_selector import TwoRandomParentSelector, TwoRandomAndCurrentParentSelector, OneRandomAndCurrentParentSelector
parent_selectors = [TwoRandomParentSelector, TwoRandomAndCurrentParentSelector, OneRandomAndCurrentParentSelector]

from moead_framework.core.selector import DeltaSelector, ClosestNeighborsSelector
selectors = [ClosestNeighborsSelector, DeltaSelector]

from moead_framework.core.sps_strategy import SpsAllSubproblems, SpsDra, SpsRandomAndBoundaries
sps_strategies = [SpsAllSubproblems, SpsDra, SpsRandomAndBoundaries]

from moead_framework.algorithm.combinatorial import Moead, MoeadDeltaNr, MoeadSPSRandom, MoeadDRA
comb_algorithms = [Moead] #, MoeadDRA, MoeadDeltaNr, MoeadSPSRandom]

from moead_framework.problem.combinatorial import Rmnk, Mubqp, KnapsackProblem
rmnk = Rmnk(instance_file="moead_framework/test/data/instances/rmnk_0_2_100_1_0.dat")
comb_problems = [rmnk]

number_of_weight = 10
number_of_weight_neighborhood = 2
number_of_evaluations = 100
weight_file = "moead_framework/test/data/weights/SOBOL-" + str(rmnk.number_of_objective) + "objs-" + str(number_of_weight) + "wei.ws"

from itertools import product
all_combinations = list(product(
    comb_algorithms,
    comb_problems,
    aggregations,
    genetic_operators,
    selectors,
    parent_selectors,
    sps_strategies
))

num_success = 0
for i, (algorithm, problem, aggregation, genetic_operator, selector, parent_selector, sps_strategy) in enumerate(all_combinations):
    try:
        moead = algorithm(
            problem=problem,
            max_evaluation=number_of_evaluations,
            number_of_weight_neighborhood=number_of_weight_neighborhood,
            weight_file=weight_file,
            aggregation_function=aggregation,
            genetic_operator=genetic_operator,
            parent_selector=parent_selector,
            sps_strategy=sps_strategy,
            mating_pool_selector=selector,
        )
        moead.run()
    except Exception as e:
        print(f'{i:03} / {len(all_combinations)} failed: {e}')
        continue

    # print(f'{i} / {len(all_combinations)} succeeded')
    num_success += 1

print(f'\n\n{num_success} / {len(all_combinations)} succeeded')

Related: openjournals/joss-reviews#2974

The MOEAD algorithms require the number of objectives. Problems also require the number of objectives. However, MOEAD require a problem. Why is that?

Related: openjournals/joss-reviews#2974

With the documentation being built on a separate gh-pages branch, does that mean the docs/html folder in master is no longer needed?
If so, the repository can be cleaned up by removing it.

Related: openjournals/joss-reviews#2974

The examples listed in the README and the documentation result in errors when I try to run them, both for source and PyPI installs.

Also, I think a quick statement outside of the code-block about the two files that are needed would make it easier for people to get started.

Reproduction

Tested on Windows 10 with Python 3.6.3

Preparation

$ mkdir  ~/src/moead-test && cd ~/src/moead-test
# note that the direct files are available without the github interface at .../raw/... instead of .../blob/...
$ wget https://github.com/moead-framework/data/raw/master/problem/RMNK/Instances/rmnk_0_2_100_1_0.dat
$ wget https://github.com/moead-framework/data/raw/master/weights/SOBOL-2objs-10wei.ws
$ vim docs_example.py  # Copied example from https://moead-framework.github.io/framework/html/examples.html
$ vim readme_example.py  # Copied example from README
$ ls
docs_example.py  readme_example.py  rmnk_0_2_100_1_0.dat  SOBOL-2objs-10wei.ws
$ python3 --version
Python 3.6.3

PyPI installation

$ python3 -m venv pypi-venv

$ source pypi-venv/Scripts/activate
(pypi-venv) $ python3 -m pip install --upgrade pip moead-framework
<snipped for brevity>
Successfully installed moead-framework-0.5.9.1 numpy-1.19.5 pip-21.0.1

(pypi-venv) $ python3 docs_example.py
Traceback (most recent call last):
  File "docs_example.py", line 24, in <module>
    weight_file = "SOBOL-" + str(rmnk.function_numbers) + "objs-" + str(number_of_weight) + "wei.ws"
AttributeError: 'Rmnk' object has no attribute 'function_numbers'

(pypi-venv) $ python3 readme_example.py
Traceback (most recent call last):
  File "readme_example.py", line 49, in <module>
    + ".txt"
NameError: name 'number_of_crossover_points' is not defined

(pypi-venv) $ deactivate

Source installation

$ python3 -m venv src-venv

$ source src-venv/Scripts/activate

(src-venv) $ python3 -m pip install --upgrade pip
<snipped for brevity>
Successfully installed pip-21.0.1

(src-venv) $ cd ../moead-framework/ && python3 -m pip install . && cd ../moead-test/
Processing d:\src\moead-framework
Collecting numpy
  Using cached numpy-1.19.5-cp36-cp36m-win_amd64.whl (13.2 MB)
Using legacy 'setup.py install' for moead-framework, since package 'wheel' is not installed.
Installing collected packages: numpy, moead-framework
    Running setup.py install for moead-framework ... done
Successfully installed moead-framework-0.5.9.1 numpy-1.19.5

(src-venv) $ python3 docs_example.py
Traceback (most recent call last):
  File "docs_example.py", line 24, in <module>
    weight_file = "SOBOL-" + str(rmnk.function_numbers) + "objs-" + str(number_of_weight) + "wei.ws"
AttributeError: 'Rmnk' object has no attribute 'function_numbers'

(src-venv) $ python3 readme_example.py
Traceback (most recent call last):
  File "readme_example.py", line 49, in <module>
    + ".txt"
NameError: name 'number_of_crossover_points' is not defined

(src-venv) $ deactivate

Related: openjournals/joss-reviews#2974

Functionality: Have the functional claims of the software been confirmed?
The authors in the paper state that:

With the moead-framework package, we aim at (1) bringing the modularity of the MOEADr package by using the flexibility of Python in order to allow the user to update the behavior of MOEA/D components in their research works, and (2) to propose new variants without being limited by the software.

The authors should provide detailed code examples of (1) and (2). Unless I am missing something.

Related: openjournals/joss-reviews#2974

This issues concerns two points:

• The main page of the documentation instructs readers to raise an issue, while the CONTRIBUTING.md suggests to open a discussion.
• The README does not give any direction when searching for terms like 'support' or 'question'.

Please make the instruction on where to ask questions consistent add a (sub)section 'Support' in the README directing users there if they have any questions.

Related: openjournals/joss-reviews#2974

I'll use this issue to gather all spelling/grammar issues I find in the paper. Additional remarks will be added as content-related issues are fixed (e.g. #24)

6: either capitalize all or none of the relevant words '[M]ulti-[O]bjective [E]volutionary [A]lgorithm based on [D]ecomposition'
10: state-of-[the]-art
19: These software [implementations]
20: state-of-[the]-art ... abstraction[,] but ...
21: do not enable [detailed testing and analysis of the various algorithm's components' behavior.]
23: allows [the definition of] different
37&38: his/her [their] (suggestion, not required)

Related: openjournals/joss-reviews#2974

Something feels strange here. There is a package that is called solution that contains a module which is called solution that contains a class which is called solution that requires a parameter that is called solution.

Related: openjournals/joss-reviews#2974

As I can see type annotations are not used. For some code there are type annotations though. Here are some of the errors found.

moead_framework/solution/one_dimension_solution.py:9: error: Need type annotation for 'solution' (hint: "solution: List[<type>] = ...")
moead_framework/solution/one_dimension_solution.py:10: error: Need type annotation for 'F' (hint: "F: List[<type>] = ...")
moead_framework/tool/result.py:67: error: Bracketed expression "[...]" is not valid as a type
moead_framework/tool/result.py:67: note: Did you mean "List[...]"?
moead_framework/tool/result.py:69: error: Incompatible types in assignment (expression has type "List[<nothing>]", variable has type "MultiList")
moead_framework/tool/result.py:71: error: Function "numpy.array" is not valid as a type
moead_framework/tool/result.py:71: note: Perhaps you need "Callable[...]" or a callback protocol?
moead_framework/tool/result.py:86: error: np.array? has no attribute "__iter__" (not iterable)

Related: openjournals/joss-reviews#2974

(1) Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).

There is an example script. Although, I would expect to see several examples that will demonstrate the major components of the software.

(2) Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

I think that a user guide is missing. User tutorials have been added but I think that they need polishing.

(3) Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

The code coverage is great 95%. Although, as I can see the tests cover only the happy path. For instance if you pass not expected types the user should guess what is happening.

(4) Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

I cannot see (2) and (3)

Related: openjournals/joss-reviews#2974

State of the field: Do the authors describe how this software compares to other commonly-used packages?

In the submitted paper the authors write that "These software offer many state-of-art algorithms, visualization tools or parallelization abstraction but they do not enable to test and analyze in detail the behavior of components implemented in each algorithm"

I cannot understand this statement. @geoffreyp could you elaborate more on this?

cc @sjvrijn I think that is related to openjournals/joss-reviews#2974 (comment)

Related: openjournals/joss-reviews#2974

Moead and Problem classes require files, but I cannot see their definition somewhere documented.

A better option could be to work directly with data structures. (arrays, lists, dicts)
Helper functions could be created in order to avoid the clutter of loading these.

Related: openjournals/joss-reviews#2974

Why the problem classes are coupled with files and not with other data structures?

Now :

• number of evaluation

Next :

• manage the Termination criteria with a function / class to add easily new criteria

Related: openjournals/joss-reviews#2974

The included (example) problems are currently listed under 'main components' in the library reference section of the documentation.
While a problem is a required argument for creating a MOEAD algorithm object, I do not feel that a Problem is a 'component' of the algorithm in the true sense of that word.

Related: openjournals/joss-reviews#2974

If I am not mistaken, the following code blocks are exactly the same. Consider a refactoring.

numerical/moead.py (48–91)
combinatorial/moead.py (52–95)

Related: openjournals/joss-reviews#2974

Is there a need for this empty initializer?

Related: openjournals/joss-reviews#2974

I cannot see a way of controlling the seed on a class level. I suppose that the intention is to control the seed globally via the built-in or the numpy random modules since you use both of them in the package. Although, this is not documented in the docstrings or/and in the documentation.

Some things to consider:

1. Control the seed on class level by passing a seed number or a random state generator object. Document it in the docstrings.
2. Control the seed globally using a single entry point. Make a note in the documentation.

The example in the README starts with the following long imports:

from moead_framework.aggregation.tchebycheff import Tchebycheff
from moead_framework.algorithm.combinatorial.moead import Moead
from moead_framework.problem.combinatorial.rmnk import Rmnk
from moead_framework.tool.result import save_population

I would at least simplify them to remove the 'double' specifications, using your subfolders' __init__.py files
For example simplifying

from moead_framework.aggregation.tchebycheff import Tchebycheff

to

from moead_framework.aggregation import Tchebycheff

by adding the line

from tchebycheff import Tchebycheff

to your aggregation/__init__.py.

You could also consider further simplifying problem.combinatorial to just problem, etc... But that is more up to personal preference.

Related: openjournals/joss-reviews#2974

All functions/methods should be documented including example inputs and outputs and their expected types. This may help you.

Related: openjournals/joss-reviews#2974

As demonstrated by the provided examples, the moead.run() method returns the found pareto-set of solution evaluations, i.e. the output of f(function_id, solution).
I can't however find which (input) solutions correspond to the returned outputs.

How should these solutions be accessed?

It would be helpful if the badges in the README link to useful targets such as

• the PyPI page of the package for the PyPI version badge
• a CI build overview page for the 'Python Application - Passing'
• etc.

Related: openjournals/joss-reviews#2974

Is there a reason for having a __init__.py file in the root of the repository?

Now :

• use a file with weight vectors

Next :

• manage the decomposition component in the code (with files and function / class)

Now :

• the Update strategy is fixed with the classic strategy of MOEA/D

Next :

• Manage this component in a function or in a class

Related: openjournals/joss-reviews#2974

Ztd1 raises ZeroDivisionError. It should reaise a more user friendly message with the correct type of the exception.

from moead_framework.problem.numerical import Zdt1
problem = Zdt1(1)
problem.generate_random_solution()

Related: openjournals/joss-reviews#2974

Currently the documentation seems like a single page. Please consider adding a user guide. See here for instance.