hydraplatform / hydra-base Goto Github PK

View Code? Open in Web Editor NEW

8.0 8.0 7.0 21.55 MB

License: GNU Lesser General Public License v3.0

Python 99.97% Mako 0.03%

hydra-base's People

Contributors

Stargazers

Watchers

Forkers

jetuk gbasolu openagua jojo0094 salespaulo pmslavin naititina

hydra-base's Issues

winpaths.get_common_documents() is broken

The winpaths module looks to be old and unsupported. It is used reading in config data on windows. Suggest to try and make this code not OS dependent?

winpaths traceback:

hydra_base\__init__.py:27: in <module>
    config.load_config()
hydra_base\config.py:83: in load_config
    sysfile = os.path.join(winpaths.get_common_documents(), 'Hydra','hydra.ini')
C:\Users\mbgnqt2\Miniconda3\envs\pywr\envs\hydra\lib\site-packages\winpaths-0.2-py3.6.egg\winpaths.py:124: in get_common_documents
    return _get_path_buf(PathConstants.CSIDL_COMMON_DOCUMENTS)
C:\Users\mbgnqt2\Miniconda3\envs\pywr\envs\hydra\lib\site-packages\winpaths-0.2-py3.6.egg\winpaths.py:95: in _get_path_buf
    path_buf = wintypes.create_unicode_buffer(wintypes.MAX_PATH)
E   AttributeError: module 'ctypes.wintypes' has no attribute 'create_unicode_buffer'

Add pytest fixutres to `hydra_base.testing.fixtures`

We're starting to reuse the pytest session fixture for hydra_base in some of the dependences (e.g. apps, etc.). It would be useful if these fixtures could be imported from hydra_base directly. This would lower the future support burden by allowing all projects to reuse the same fixtures.

Suggestion is to make a hydra_base.testing subpackage containing the fixtures.

Optional xml element must be present

In import_template_xml in template.py template_description has to be in the xml file, even though in the .xsd it is an optional element - so it should be allowed to be left out of the xml.

def import_template_xml(template_xml, allow_update=True, **kwargs):

....
    template_description = xml_tree.find('template_description').text

no test for get-project-by-network-id

Every new endpoint needs a test. Please add a test for this endpoint.

[ Update attribute ]

Scenario:

The user is updating an attribute, using duplicate data, calling the "update_attr" endpoint

Problem:

The server returns this generic error:

{"message":"An unexpected error has occurred: (sqlite3.IntegrityError) UNIQUE constraint failed: tAttr.attr_name, tAttr.attr_dimen"}

To offer a good UI managing the client should interpret the returned error, that is neither easy nor mantainable.

Solution:

The returned error should be more meaningful and possibly be identified using a "code" (unique for any error codified in the server) so that the client could recognize it very easily and manage the UI accordingly

Python 3 type comparisons

I've done some more work on Python 3 compatibility (see https://github.com/hydraplatform/hydra-base/tree/py3). This is now mostly working for the same tests as Python 2. One exception is that Python 3 no longer supports comparisons of different types. Rather it raises a TypeError.

This causes problems in the dataset_util.validate_XXX functions. See travis error at the link below.

https://travis-ci.org/hydraplatform/hydra-base/jobs/344703393#L647

I think the correct answer here is to try the comparison and catch the exception? I.e.

def validate_LESSTHAN(in_value, restriction):
    """
        Test to ensure that a value is less than a prescribed value.
        Parameter: Two values, which will be compared for the difference..
    """
    #Sometimes restriction values can accidentally be put in the template <item>100</items>,
    #Making them a list, not a number. Rather than blowing up, just get value 1 from the list.
    if type(restriction) is list:
        restriction = restriction[0]

    value = _get_val(in_value)
    if type(value) is list:
        for subval in value:
            if type(subval) is tuple:
                subval = subval[1]
            validate_LESSTHAN(subval, restriction)
    else:
        try:
            if value >= restriction:
                raise ValidationError("LESSTHAN: %s"%(restriction))
        except TypeError:
            # Incompatible types for comparison.
            raise ValidationError("LESSTHAN: Incompatible types %s"%(restriction))

Before doing this for all the validate functions I would like some feedback.

Attribute groups

We discussed the idea of having some sort of grouping ability on Attr objects in the database. This would be useful for organising different types of attribute in both the frontend and backend. There might be two uses cases I guess. Frontend grouping for display, and backend grouping for processing (effectively hidden from the user). Would they be the same field?

Add a dataframe data type.

This is a specific data type to support pandas.DataFrame objects. Clients will have to read and support the pandas JSON formats (even non-Python ones).

Some implementation questions here:

When submitting a new dataframe type. How is this stored?

As sent but with metadata. I.e. Hydra doesn't touch what you sent.
Converted to a pandas.DataFrame. Saved in internally in a agreed format. (e.g. orient="tables")

When requesting a dataframe type dataset. You should be able to request in a different format. E.g. orient="columns" even if it was originally. sent in a different format. How should this type be specified?

In the cookie for the whole session?
As arguments to the GET request?

test gitlab integration

test

Adding template as JSON to SQLite

I'm trying to use add_template with a JSON / Python dict formatted template, complete with typeattrs. I have no problem with MySQL, but when using SQLite on my dev setup, I get back {'faultcode': 'Server', 'faultstring': '(sqlite3.IntegrityError) UNIQUE constraint failed: tAttr.attr_name, tAttr.attr_dimen'}. I'm using the old HydraServer (I haven't yet switched over), but I assume I'd have the same issue here in hydra-base, so am reporting this here.

One hypothesis is the case of attr_name. I haven't checked, but could it be related to SQLite's handling of existing attr_name and attr_dimen with a different case (e.g., Outflow vs outflow)? I'll look into this further.

Improve Validation & Formalisation of Hydra objects

the JSONObject currently returns 'None' when acessing an attribute that doesn't exist. If we change this to throw an AttributeError (correct) then we need a way to formalise what attributes are allowed on an incoming object.
One solution could be:

#PARENT SCHEMA
class HydraSchema():


    def validate(self, k, v):
        pass

    def __init__(self, parent=None):
        super(Project, self).__init__()

        if parent is None:
            return

        for k, v in parent.items():
            #validate k versus v
            validate(k, v)
            setattr(self, k, v)

#HYDRA SCHEMA
class HydraProject(JSONObject):
    _properties = {
        ('id',          Integer(default=None)),
        ('name',        Unicode(default=None)),
        ('description', Unicode(default=None)),
        ('status',      Unicode(default='A', pattern="[AX]")),
        ('cr_date',     Unicode(default=None)),
        ('created_by',  Integer(default=None)),
        ('attributes',  SpyneArray(ResourceAttr)),
        ('attribute_data', SpyneArray(ResourceScenario)),

    }


SpyneProject = make_spyne_resource(HydraProject)

#SPYNE SCHEMA
class SpyneProject(Resource):
    """
   - **id**          Integer(default=None)
   - **name**        Unicode(default=None)
   - **description** Unicode(default=None)
   - **status**      Unicode(default='A')
   - **cr_date**     Unicode(default=None)
   - **created_by**  Integer(default=None)
   - **attributes**  SpyneArray(ResourceAttr)
   - **attribute_data** SpyneArray(ResourceScenario)
    """
    _type_info = [
        ('id',          Integer(default=None)),
        ('name',        Unicode(default=None)),
        ('description', Unicode(default=None)),
        ('status',      Unicode(default='A', pattern="[AX]")),
        ('cr_date',     Unicode(default=None)),
        ('created_by',  Integer(default=None)),
        ('attributes',  SpyneArray(ResourceAttr)),
        ('attribute_data', SpyneArray(ResourceScenario)),
    ]

Add update_links function

Analogous to the update_nodes function.

Drop support for heterogenous dataframes?

The dataframe parsing code can support dataframes with nested dicts/lists as a value. Internally these must be stored as objects in the underlying numpy array. However it means that in general it is not guaranteed that a dataset of the type "DATAFRAME" is not a 2D array of data (i.e. it could contain nested data structures).

Consider dropping support for this.

Ref #64 for a marshmallow schema here?

login_user fails to locate username (Unicode issue?)

I've been trying to resolve my problem logging in using RemoteJSONConnection(hydraplatform/hydra-client-python#10)

and I think I've found the problem.

In hdb.py the following function:

def login_user(username, password):
    try:
        user_i = db.DBSession.query(User).filter( User.username==username ).one()
    except NoResultFound:
        raise HydraError(username)

raises an exception because the filter condition is not met (the db is an empty SQLite DB created by Hydra Server, with a "root" user and should find a match ['root']). Incidentally the message is not very helpful, a "user not found" message would be good to have.

If I change it to:

def login_user(username, password):
    try:
        username=username.decode("utf-8", "ignore")
        user_i = db.DBSession.query(User).filter( User.username==username ).one()
    except NoResultFound:
        raise HydraError(username)

then it works, the RemoteJSONConnection example now connects.

This is on Windows 10. I am using SQLAlchemy 1.36. I have checked the SQLite .db which says the encoding is UTF-8.

The above modification fixes this particular problem, I'd imagine it's going to crop up elsewhere, so I'm not sure what the best thing to do is.

Looking at the call from remote_json_connection.py:

r = requests.post(self.url, data=json.dumps(call), headers=headers, cookies=cookie)

I suspect the problem is that the json.dumps call is encoding the string to UTF-8 and SQLAlchemy doesn't like it?

Add coverage to travis.

Should be easy enough with pytest-cov.

Improve the log format

Situation

In this moment the log message is composed in this way:

pid
datetime
message

In this way the user has no ideas from where the log message has been generated, having to look inside the code to search for the message and then find an eventual error

Improvement

To be able to get the problem immediately, I suggest to improve the message adding the following data:

file name
class name
method name
line number

Plugin XML schema is out of date.

Migrate to pytest.

We've started converting the existing unittests to pytest with fixtures. I've opened this issue so we can track which modules have been done and which are left to complete.

Units for complex types

The current unit support is at the attribute / dataset level. This is fine for scalar / singular valued datasets, but problematic for complex dataset structures. For example, a dataframe where each column is a different unit - e.g. specifying the bathymetry for a reservoir as a table of volume, level and area values.

Related to #61

Lib plugins is old

lib/plugins.py contains out of date code including references to "trunk" and "svn". It also doesn't work with the XML schema. This might be related to #26. I'm not sure which is right or if they are both out of date.

Dropping support for Python 2.7

Python 2.7 is going to reach EOL in 2020:

The End Of Life date (EOL, sunset date) for Python 2.7 has been moved five years into the future, to 2020.

Numpy 1.14 has also just be release with the following statement:

A major decision affecting future development concerns the schedule for dropping Python 2.7 support in the runup to 2020. The decision has been made to support 2.7 for all releases made in 2018, with the last release being designated a long term release with support for bug fixes extending through 2019. In 2019 support for 2.7 will be dropped in all new releases. More details can be found in the relevant NEP_.

I suggest we follow the numpy approach and have a final 2.7 supported release at the end of 2018.

Any thoughts?

Allow decorator-based permissions enforcement

At present, determining the authorisation status of a user to call a particular function requires an explicit call to check_perm (defined in util/permissions.py) for each permission a user must possess in order to run the routine.

The invocation of check_perm takes the same form in every function, and this call may therefore be replaced with a decorator which modifies the called function to automatically verify that the current user possesses the required permission - akin to a generalisation of the Flask @login_required decorator.

This issue relates to the development of a permissions-enforcement decorator, and any modifications to the existing access-control framework required to support this new technique.

Add a profile data type.

For example JSON frequency of a daily profile.

{
"type": "profile",
"value": {
    "frequency": "daily",
    "values": [1, 2, 3, ..., 366]
}

Frequencies to support.

Daily
Weekly
Monthly

Alembic upgrade

During the upgrade I had some errors I will report after here. I think that before creating an index should be appropriate trying to drop (with IF EXISTS) it to avoid "already existing" errors meanwhile having the possibility to modify them in the upgrade script.

alembic upgrade head
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade dd1110144e53 -> f579b06e9cc2, attr_id
ERROR [f579b06e9cc2_attr_id_py] (sqlite3.OperationalError) index ix_tMetadata_new_dataset_id already exists [SQL: u'CREATE INDEX "ix_tMetadata_new_dataset_id" ON "tMetadata_new" (dataset_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/f579b06e9cc2_attr_id.py", line 74, in upgrade
    sa.Column('value', sa.Text(),  nullable=False),
  File "<string>", line 8, in create_table
  File "<string>", line 3, in create_table
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1120, in create_table
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 101, in create_table
    operations.impl.create_table(table)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 200, in create_table
    self._exec(schema.CreateIndex(index))
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
    compiled
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) index ix_tMetadata_new_dataset_id already exists [SQL: u'CREATE INDEX "ix_tMetadata_new_dataset_id" ON "tMetadata_new" (dataset_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
INFO  [alembic.runtime.migration] Running upgrade f579b06e9cc2 -> a7896842f484, template
ERROR [a7896842f484_template_py] (sqlite3.OperationalError) no such column: template_id [SQL: u'insert into tTemplate_new (id, name, cr_date, layout) select template_id, template_name, cr_date, layout from tTemplate'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/a7896842f484_template.py", line 59, in upgrade
    op.execute("insert into tTemplate_new (id, name, cr_date, layout) select template_id, template_name, cr_date, layout from tTemplate")
  File "<string>", line 8, in execute
  File "<string>", line 3, in execute
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1856, in execute
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 161, in execute_sql
    execution_options=operation.execution_options
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 121, in execute
    self._exec(sql, execution_options)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) no such column: template_id [SQL: u'insert into tTemplate_new (id, name, cr_date, layout) select template_id, template_name, cr_date, layout from tTemplate'] (Background on this error at: http://sqlalche.me/e/e3q8)
ERROR [a7896842f484_template_py] (sqlite3.OperationalError) no such column: type_name [SQL: u'insert into tTemplateType_new (name, id, template_id, resource_type, alias, layout, cr_date) select type_name, type_id, template_id, resource_type, alias, layout, cr_date from tTemplateType'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/a7896842f484_template.py", line 87, in upgrade
    op.execute("insert into tTemplateType_new (name, id, template_id, resource_type, alias, layout, cr_date) select type_name, type_id, template_id, resource_type, alias, layout, cr_date from tTemplateType")
  File "<string>", line 8, in execute
  File "<string>", line 3, in execute
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1856, in execute
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 161, in execute_sql
    execution_options=operation.execution_options
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 121, in execute
    self._exec(sql, execution_options)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) no such column: type_name [SQL: u'insert into tTemplateType_new (name, id, template_id, resource_type, alias, layout, cr_date) select type_name, type_id, template_id, resource_type, alias, layout, cr_date from tTemplateType'] (Background on this error at: http://sqlalche.me/e/e3q8)
INFO  [alembic.runtime.migration] Running upgrade a7896842f484 -> 08f0ebe40290, resource_attr_id
ERROR [08f0ebe40290_resource_attr_id_py] (sqlite3.OperationalError) index ix_tResourceAttr_new_link_id already exists [SQL: u'CREATE INDEX "ix_tResourceAttr_new_link_id" ON "tResourceAttr_new" (link_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/08f0ebe40290_resource_attr_id.py", line 57, in upgrade
    sa.UniqueConstraint('group_id',   'attr_id', name = 'group_attr_1'),
  File "<string>", line 8, in create_table
  File "<string>", line 3, in create_table
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1120, in create_table
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 101, in create_table
    operations.impl.create_table(table)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 200, in create_table
    self._exec(schema.CreateIndex(index))
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
    compiled
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) index ix_tResourceAttr_new_link_id already exists [SQL: u'CREATE INDEX "ix_tResourceAttr_new_link_id" ON "tResourceAttr_new" (link_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
INFO  [alembic.runtime.migration] Running upgrade 08f0ebe40290 -> a48ad41579de, dataset_collection
ERROR [a48ad41579de_dataset_collection_py] (sqlite3.OperationalError) no such column: collection_id [SQL: u'insert into tDatasetCollection_new (id, name, cr_date) select collection_id, collection_name, cr_date from tDatasetCollection'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/a48ad41579de_dataset_collection.py", line 49, in upgrade
    op.execute("insert into tDatasetCollection_new (id, name, cr_date) select collection_id, collection_name, cr_date from tDatasetCollection")
  File "<string>", line 8, in execute
  File "<string>", line 3, in execute
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1856, in execute
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 161, in execute_sql
    execution_options=operation.execution_options
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 121, in execute
    self._exec(sql, execution_options)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) no such column: collection_id [SQL: u'insert into tDatasetCollection_new (id, name, cr_date) select collection_id, collection_name, cr_date from tDatasetCollection'] (Background on this error at: http://sqlalche.me/e/e3q8)
INFO  [alembic.runtime.migration] Running upgrade a48ad41579de -> f52ccd2b7ffb, resource_type_id
ERROR [f52ccd2b7ffb_resource_type_id_py] (sqlite3.OperationalError) no such column: resource_type_id [SQL: u'insert into tResourceType_new (id, type_id, ref_key, network_id, node_id, link_id, group_id, cr_date) select resource_type_id, type_id, ref_key, network_id, node_id, link_id, group_id, cr_date from tResourceType'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/f52ccd2b7ffb_resource_type_id.py", line 58, in upgrade
    op.execute("insert into tResourceType_new (id, type_id, ref_key, network_id, node_id, link_id, group_id, cr_date) select resource_type_id, type_id, ref_key, network_id, node_id, link_id, group_id, cr_date from tResourceType")
  File "<string>", line 8, in execute
  File "<string>", line 3, in execute
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1856, in execute
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 161, in execute_sql
    execution_options=operation.execution_options
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 121, in execute
    self._exec(sql, execution_options)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) no such column: resource_type_id [SQL: u'insert into tResourceType_new (id, type_id, ref_key, network_id, node_id, link_id, group_id, cr_date) select resource_type_id, type_id, ref_key, network_id, node_id, link_id, group_id, cr_date from tResourceType'] (Background on this error at: http://sqlalche.me/e/e3q8)
INFO  [alembic.runtime.migration] Running upgrade f52ccd2b7ffb -> 6b24d6443e31, resourcegroupitem
ERROR [6b24d6443e31_resourcegroupitem_py] (sqlite3.OperationalError) index ix_tResourceGroupItem_new_scenario_id already exists [SQL: u'CREATE INDEX "ix_tResourceGroupItem_new_scenario_id" ON "tResourceGroupItem_new" (scenario_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/6b24d6443e31_resourcegroupitem.py", line 53, in upgrade
    sa.UniqueConstraint('group_id', 'subgroup_id', 'scenario_id', name = 'subgroup_group_1'),
  File "<string>", line 8, in create_table
  File "<string>", line 3, in create_table
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1120, in create_table
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 101, in create_table
    operations.impl.create_table(table)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 200, in create_table
    self._exec(schema.CreateIndex(index))
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddproject/2project/2l(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
    compiled
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/sitproject/2e-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) index ix_tResourceGroupItem_new_scenario_id already exists [SQL: u'CREATE INDEX "ix_tResourceGroupItem_new_scenario_id" ON "tResourceGroupItem_new" (scenario_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
INFO  [alembic.runtime.migration] Running upgrade 6b24d6443e31 -> 56ea2e814437, rule
ERROR [56ea2e814437_rule_py] (sqlite3.OperationalError) index ix_tRule_new_group_id already exists [SQL: u'CREATE INDEX "ix_tRule_new_group_id" ON "tRule_new" (group_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/56ea2e814437_rule.py", line 59, in upgrade
    sa.UniqueConstraint('scenario_id', 'name', name="unique rule name")
  File "<string>", line 8, in create_table
  File "<string>", line 3, in create_tableproject/2project/2project/2
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1120, in create_tableproject/2project/2project/2
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 101, in create_table
    operations.impl.create_table(table)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 200, in create_table
    self._exec(schema.CreateIndex(index))
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddlproject/2project/2project/2
    compiled
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) index ix_tRule_new_group_id already exists [SQL: u'CREATE INDEX "ix_tRule_new_group_id" ON "tRule_new" (group_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
INFO  [alembic.runtime.migration] Running upgrade 56ea2e814437 -> d0c45fdc4fac, note
ERROR [d0c45fdc4fac_note_py] (sqlite3.OperationalError) index ix_tNote_new_link_id already exists [SQL: u'CREATE INDEX "ix_tNote_new_link_id" ON "tNote_new" (link_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/d0c45fdc4fac_note.py", line 55, in upgrade
    sa.Column('group_id',sa.Integer(),  sa.ForeignKey('tResourceGroup.id'), index=True, nullable=True),
  File "<string>", line 8, in create_table
  File "<string>", line 3, in create_table
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1120, in create_table
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 101, in create_table
    operations.impl.create_table(table)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 200, in create_table
    self._exec(schema.CreateIndex(index))
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/sitproject/2e-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
    compiled
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) index ix_tNote_new_link_id already exists [SQL: u'CREATE INDEX "ix_tNote_new_link_id" ON "tNote_new" (link_id)'] (Background on this error at: http://sqlalche.me/e/e3q8)
INFO  [alembic.runtime.migration] Running upgrade d0c45fdc4fac -> 21e8d704c020, roles_perms
ERROR [21e8d704c020_roles_perms_py] (sqlite3.OperationalError) no such column: perm_id [SQL: u'insert into tPerm_new (id, name, code, cr_date) select perm_id, perm_name, perm_code, cr_date from tPerm'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/home/giovanni/UoM/hydra-base/hydra_base/db/alembic/versions/21e8d704c020_roles_perms.py", line 57, in upgrade
    op.execute("insert into tPerm_new (id, name, code, cr_date) select perm_id, perm_name, perm_code, cr_date from tPerm")
  File "<string>", line 8, in execute
  File "<string>", line 3, in execute
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1856, in execute
    return operations.invoke(op)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 161, in execute_sql
    execution_options=operation.execution_options
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 121, in execute
    self._exec(sql, execution_options)
  File "/home/giovanni/.local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in executeproject/2
    return meth(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/giovanni/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) no such column: perm_id [SQL: u'insert into tPerm_new (id, name, code, cr_date) select perm_id, perm_name, perm_code, cr_date from tPerm'] (Background on this error at: http://sqlalche.me/e/e3q8)

Use marshmallow for serialization?

I was shown marshmallow recently. It might be useful for Hydra to use for serialization? Not sure.

marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, to and from native Python datatypes.

https://marshmallow.readthedocs.io/en/latest/

There's a sqlalchemy integration library.

https://marshmallow-sqlalchemy.readthedocs.io/en/latest/

XML is rubbish!

The plugins are defined plugin.xml. The XML schema is a bit too restrictive. It requires ordered inputs for data that has no implicit order requirement, for example. Might better to consider a YAML or JSON style format here.

Zope.sqlalchemy version 1.2 renames ZopeTransactionExtension class

Version 1.2 of Zope Sqlalchemy has renamed the ZopeTransactionExtension class to ZopeTransactionEvents.

This is in use at two locations in hydra_base/db/__init__.py:

hydra-base/hydra_base/db/__init__.py

Line 23 in 36c01a7

from zope.sqlalchemy import ZopeTransactionExtension

...and...

hydra-base/hydra_base/db/__init__.py

Line 103 in 36c01a7

extension=ZopeTransactionExtension())

The former name has been retired so L23 results in an ImportError.

Default admin user has no permision to add networks

The following must be added to the DB.

insert into tRoleUser (user_id, role_id) values (2, 1);

Support creation of multi-template networks

Hydra base allows the existence of networks associated with multiple templates, but does not provide a method of creating these.

Following updates to hwi, the arguments to _bulk_add_resource_attrs now include a list of (template) resource types where the ref_key is NETWORK, leading to a TypeError when this list is used as a key to the type_dict.

This issue is to extend the network creation routine to support the updated argument format originating from the frontend.

Make all 'name' columns bigger

We've been dealing with issues of late whereby the name column on nodes and links is too short.
We should increase these to 200 across the board.

The 'warn' function is deprecated, use 'warning' instead

I have spotted these warnings about warnings.

hydra-base/hydra_base/lib/network.py:208: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
  logging.warn("No attributes on any %s....", ref_key.lower())

DataType uses deprecated @abstractproperty

@abstractproperty is deprecated since Python 3.3. The proper syntax is to decorate an @abstractmethod with the normal @property decorator.

Not sure about 2.7 support.

Tracking Scenarios Parenting

I suggest to add a field inside the tScenario table called "parent_id" to track the cloning history from a Scenario to another.

Attributes should be allows to be scoped to a project

If there are multipel unrelated projects in Hydra, finding an attribute can become difficult when there are many attributes unrelated to the user's current project. There should be a mechanism by which attributes can be scoped to a project, such that requests for attributes only return a list of attributes relevant to that project, and not globally.

SqlAlchemy Warning during the startup

Problem

During the startup the HWI always shows this warning:

~/.local/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py:794: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
<module 'hwi.wrmp' from '~/UoM/hwi/hwi/wrmp.pyc'>

update and migreate test_attributes.py

Update test_attributes to use hydra-base instead of hydra-server and JSONObjects instead of SOAP requests.
Move test_attributes.py to 'tests' so they work with pytest and CI etc.

Changes to lib and util routines relating to restrictions and validation

This issue contains the hydra-base changes that are made during resolution of template-manager issue 1 over at GitLab.

Add user groups?

Having multiple distinct user groups on the same system can become both a security issue and a usability problem. Should we support user groups natively in hydra, rather relying on an app to manage that stuff?

What behaviour should this exhibit? Have global users and non-global users

Add generic Bool restriction type

Currently hydra_base/util/dataset_util.py defines the BOOL10 and BOOLYN restriction types, each of which requires particular values for its "true-role" and "false-role" values.

The new frontend for specifying restrictions introduced to the template manager enables custom true and false-role values to be defined.

As such, it is necessary to extend the hydra-base dataset utilities to support the new restriction type - a generic BOOL in which no assumptions are made about the type or naming of the restriction's pair of values.

Migrate all tests

Problem

We need to migrate all the tests from the "unittests" folder to the "tests" one.

Make timeseries data type more strict.

Agreed that this type will only be absolute times. No relative timeseries allowed. Will allow for more strict checking of the formats, etc.

DB Encoding

Problem

The Db/Table/Fields encoding seems probably to follow the machine settings instead of being forced to an unified "utf8_general_ci" or "utf8_unicode_ci" to support any kind of data.
In this way there could be some problems in saving data that are not compliant with the local server settings, because most of the hydra code can manage utf8 while the DB in that case cannot.

Solution

We have to use specific instructions in creating connections as the following one:

create_engine('mysql+mysqldb://USER:@SERVER:PORT/DB?charset=utf8', encoding='utf-8')

and inside the model code, when creating table text fields, we can also use the Unicode type with encoding utf8, for example:

name = Column(Unicode(60, collation='utf8_bin'),  nullable=False)

Upgrade Rules

The rules system is not flexible or expressive enough for some of our current tasks, so rules should be updated to do the following:

In the following a 'resource' is a node, link, group or network instance.

Remove dependency on scenario_ids. A rule should be associated to a whole network or to a scenario within a network.
Allow different types of rules to be defined. This is separate from a template, and is simply a global categorization of rules to help sort them.
Allow rules to be searched by name, type, description etc??

Unique names across all physical resources?

Should resource names be unique between all resources, regardless of type (node or link)? I am converting some generic nodes to Pywr Links, but this causes an issue in Pywr if a Hydra link happens to share a name with one of those generic nodes. I can of course easily enforce this on my end, just wondering if this is an issue generally that should be addressed at the Hydra Base level.

It is possible to manipulate deleted networks.

I think HB marks networks as deleted rather than removing them from the DB. It is possible to still manipulate / change these models via the API.

Should this be possible? I was operating as an admin. I've not checked if a normal user can do this.

QC unit conversions

Somewhat related to #61, at some point the unit conversion factors should be double-checked. The lf value for ac-ft in particular appears to be too low by a factor of 1000 (unless my interpretation of lf is incorrect). There could be others.

add layout to Projects

We should add a layout column to tProject

add layout to resource group

we should have a layout column on tResourceGroup

Handle large datasets

Some datasets are larger than the DB column allows.
THe prior approach to this was to ZIP large values, but this does not work well with UTF-8 encoded databases, so this has been removed.

The consensus therefore is to store datasets larger than a configurable threshold (already there with the compression_threshold config item) to an external location on the filesystem, and to reference this dataset in the DB column. Something like: file://path/to/external/datsets/HASH

Move units & dimensions to the DB

Rather than using an xml file, we need to add units and dimensions to the DB.
The following steps are:
1: Migrate the tests from 'unittests' to 'tests' folder and ensure all tests pass on travis.
2: Create 2 new database tables: tUnit and tDimension: (add two classes to models.py)

tDimension:
-- ID
-- name
-- description
-- project_id (OPTIONAL)

tUnit
-- dimension_id
-- name
-- abbreviation
-- lf (Linear conversion factor)
-- cf (constant conversion factor)
-- description
-- project_id (OPTIONAL)

An empty project ID entry means the unit / dimension is global. An entry in the project_id means that this is only visibile within a specified project. To make it available in another project, a new row must be created or it must be made global.

WHen adding a unit:
If the unit project_id is EMPTY, the project ID of the dimension must also be empty
if the unit project ID not empty, the project ID of the dimension may be empty or not empty.

Conver the unit_definitions.xml file to JSON and import these units and dimensions into the DB on creation, when the users are being created etc.
Create an alembic DB upgrade script. (ASK STEVE!)
Rewrite the units.py end points / objects etc to use the DB instead of the XML file.
Re-run the tests and ensure they still all work!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.