gmlc-tdc / pyhelics Goto Github PK

A set of domain specific standard/features to be followed along with a base class that can be extended by the user. The domain standard and the base class -- together called the common interface API -- will allow the user to write adapters to integrate a simulator with HELICS. The goal of the common interface API is to allow the user to write domain specific federates instead of simulator specific federates.

helicsFederateDisconnect() is the API replacing helicsFederateFinalize() but it doesn't appear supported in pyhelics 2.7.1.

  File "Controller.py", line 38, in destroy_federate
    status = h.helicsFederateDisconnect(fed)
AttributeError: module 'helics' has no attribute 'helicsFederateDisconnect'

Tried h.helicsFederateSetLogFile(fed, helics.log) as well as setting logfile : "helics.log" in the config.json file. How do I get the helics federate to log to a file similar to the helics broker. Using version 3.3.0post0

I'm running a federation with a bug in one of the federates, which causes that federate to fail. The other federates do not have any timeout check, so they get hung waiting to receive communications from the failed federate. This causes the whole federation to get hung, rather than fail. This seems like a bug, but maybe it's a feature: When 1 federate within a federation fails, should the whole federation fail?

It seems the behavior could be changed if you use process.poll() with a timeout rather than process.wait() here:

I haven't made a minimal working example yet, but I can make one if you think it's necessary.

Environment

• Operating System: Linux (HPC)
• Installation: `pip install helics[cli]
• helics and pyhelics version:

helics, version v3.3.2

Python HELICS version v3.3.2

HELICS Library version 3.3.2 (2022-12-02)

{
    "buildflags": " -static-libstdc++ -static-libgcc -O3 -DNDEBUG -static-libstdc++ -static-libgcc  $<$<COMPILE_LANGUAGE:CXX>:-std=c++17>",
    "compiler": "Unix Makefiles  Linux-5.15.0-1023-azure:GNU-8.3.1",
    "cores": [
        "zmq",
        "zmqss",
        "tcp",
        "tcpss",
        "udp",
        "ipc",
        "interprocess",
        "inproc"
    ],
    "cpu": " Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz",
    "cpucount": 24,
    "cputype": "x86_64",
    "hostname": "el3",
    "memory": "192789 MB",
    "os": "Linux  3.10.0-1062.9.1.el7.x86_64  #1 SMP Fri Dec 6 15:49:49 UTC 2019",
    "version": {
        "build": "",
        "major": 3,
        "minor": 3,
        "patch": 2,
        "string": "3.3.2 (2022-12-02)"
    },
    "zmqversion": "ZMQ v4.3.4"
}

GitHub runners fail because setup-python has dropped Python 2.7 support: actions/setup-python#672

Summary

HelicsMessage.data raises a UnicodeDecodeError for messages containing long strings. I've tried this both by creating a HelicsMessage object and assigning the data property to a string, and by passing a string to HelicsEndpoint.send_data(). For short strings the data is decoded successfully, but for longer strings an error is raised. I have also tried helicsMessageSetString() and helicsMessageGetString() directly with the same result.

In some cases a UnicodeDecodeError is not raised, but I have seen extra (apparently random) characters appended to the end of the decoded string. I suspect the root cause of both errors is the same, I just randomly got lucky with data that was incorrect, but could still be decoded as a valid unicode string.

Example errors

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcd in position 99: unexpected end of data
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 251: invalid start byte

Versions

• python version: 3.9.7
• helics version: 3.0.1.post7

Examples

These examples consistently raise a UnicodeDecodeError.

Using HelicsEndpoint.send_data()

# sending
message = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
          'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
          'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
          'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
          'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
          'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
          'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
          'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
endpoint.send_data(message, destination="foo/ep")

# receiving in federate "foo"
endpoint = federate.register_endpoint("ep")
# ...
if endpoint.has_message():
    message = endpoint.get_message().data

Sending a HelicsMessage object

message = endpoint.create_message()
message.destination = "foo/ep"
message.data = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
               'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
               'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
               'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
               'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
               'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
               'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
               'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
endpoint.send_data(message)

# receiving in federate "foo" (same as previous example)
endpoint = federate.register_endpoint("ep")
# ...
if endpoint.has_message():
    message = endpoint.get_message().data

I ran the following:

import subprocess
import shlex

names = subprocess.check_output(shlex.split("nm helics/install/lib/libhelics.3.2.1.dylib")).splitlines()

names = {str(name).strip().split()[-1].split(".")[0].strip("_").strip("'") for name in names if name.strip().split()[-1].startswith(b"_helics")}

import helics as h

print("These functions are missing in pyhelics {} with HELICS {}".format(h.__version__, h.helicsGetVersion()))

for name in sorted([n for n in names]):
    try:
        getattr(h, name)
    except:
        print(name)

to get this:

These functions are missing in pyhelics v3.2.1 with HELICS 3.2.1 (2022-06-16)
helicsBooleanToBytes
helicsComplexObjectToBytes
helicsComplexVectorToBytes
helicsDataBufferClone
helicsDataBufferConvertToType
helicsDataBufferIsValid
helicsDataBufferStringSize
helicsDataBufferToChar
helicsDataBufferToComplexObject
helicsDataBufferToComplexVector
helicsDataBufferToInteger
helicsDataBufferToRawString
helicsDataBufferToString
helicsDataBufferToTime
helicsDataBufferToVector
helicsDataBufferVectorSize
helicsEndpointSendMessageZeroCopy
helicsIntegerToBytes
helicsNamedPointToBytes
helicsRawStringToBytes

These functions need to be implemented.

cc @phlptp @kdheepak

Versions etc.

• pyhelics v2.7.1.post1
• Python 3.8
• Windows

Reproducing:

Install helics and pytest into a Python virtual environment. Save the following code at test_invalid_state.py. Everything is test setup except for the test_broker_disconnect_disrupts_co_simulation function, which is the actual test. Pay particular attention to the with pytest.raises block: this demonstrates that the underlying C-library reports a HelicsFederateState of 10, but the enumeration only goes up to 9.

import shutil
import time

import helics as h
import pytest

def write_config(config) -> Tuple[str, str]:
    """No cleanup is performed."""
    # Reading an open NamedTemporaryFile doesn't work on Windows. Yay.
    temp_dir = tempfile.mkdtemp()
    config_path = os.path.join(temp_dir, "config.json")
    with open(config_path, "w") as f:
        json.dump(config, f)

    return temp_dir, config_path


@pytest.fixture(autouse=True)
def clean():
    """Avoid state corruption between tests."""
    h.helicsCleanupLibrary()


@pytest.fixture(scope='function')
def broker() -> h.HelicsBroker:
    """Simple in process broker awaiting a single federate."""
    broker = h.helicsCreateBroker(
        type='inproc', name='test_broker', init_string='--federates 1'
    )
    yield broker
    # Clean up. This seems to be necessary even with an automatic call
    # of h.helicsCleanupLibrary() after each test function...
    broker.disconnect()
    del broker


@pytest.fixture(scope='function')
def federate(broker) -> h.HelicsCombinationFederate:
    """No-frills in-process combination federate."""
    config = {
        "name": "external_federate",
        "period": 60.0,
        "coretype": 'inproc',
        "corename": "external_federate",
        "broker": broker.name,
    }
    temp_dir, config_path = write_config(config)
    federate = h.helicsCreateCombinationFederateFromConfig(config_path)
    yield federate
    shutil.rmtree(temp_dir)
    federate.disconnect()
    del federate

def test_broker_disconnect_disrupts_co_simulation(broker, federate):
    """Disconnecting the broker mid simulation causes federates to
    receive HELICS_TIME_MAXTIME when requesting the next time step.
    Also, there seems to be a bug in HELICS related where a bad
    state is reported.
    """
    # Start co-simulation, and request a time step.
    federate.enter_executing_mode()
    federate.request_next_step()

    # Broker and federate should be running.
    assert broker.is_connected()
    assert federate.state is h.HELICS_STATE_EXECUTION

    # Now, forcibly disconnect the broker.
    broker.disconnect()
    assert not broker.is_connected()
    # I would have expected an error state...
    # assert federate.state is h.HELICS_STATE_ERROR
    # But the federate still reports that it's executing...
    assert federate.state is h.HELICS_STATE_EXECUTION

    # Apparently requesting the next time in this situation gives back
    # the maximum time?
    current_time = federate.request_next_step()
    assert current_time == h.HELICS_TIME_MAXTIME

    # Bug in HELICS/pyhelics:
    with pytest.raises(
            ValueError, match=r'10 is not a valid HelicsFederateState'):
        assert federate.state is h.HELICS_STATE_EXECUTION

    # And again.
    current_time = federate.request_next_step()
    assert current_time == h.HELICS_TIME_MAXTIME
    # assert federate.state is h.HELICS_STATE_EXECUTION

    # Broker should still be disconnected.
    assert not broker.is_connected()

Run the test (in your activated virtual environment) via python -m pytest test_invalid_state.py.

The deprecated functions, helicsEndpointSendMessageObject and helicsEndpointSendMessageObjectZeroCopy incorrectly call helicsEndpointSendMessage as they both fail to pass the message argument to that function call resulting in an missing argument error.

Receiving serialized python objects over via helicsInputGetBytes does not work properly. It seems to send the bytes array properrly, helicsInputGetByteCount returns the right number of bytes. But the byte array returned is truncated by the first '\0' There are of course workarounds (encoding the serialized object using codecs) but that adds extra processing, and doubles the size of the data to be transfered... Which is not ideal.

Platform/Versions

OS

OS: Windows 10 Enterprise
Version: 20H2
Build: 19042.1165
Experience: Windows Feature Experience Pack 120.2212.3530.0

OS: Ubuntu
Version: 20.04.2 LTS

Software

python: 3.8.10
pyhelics: 3.0.0.0

Code

from threading import Thread
import helics as h
import time
import pickle

def the_pickle():
    deltat = 0.01
    fedinitstring = "--federates=1"
    fedinfo = h.helicsCreateFederateInfo()
    h.helicsFederateInfoSetCoreName(fedinfo, "Pickle Federate")
    h.helicsFederateInfoSetCoreTypeFromString(fedinfo, "zmq")
    h.helicsFederateInfoSetCoreInitString(fedinfo, fedinitstring)
    h.helicsFederateInfoSetTimeProperty(fedinfo, h.helics_property_time_delta, deltat)
    vfed = h.helicsCreateValueFederate("Pickle Federate", fedinfo)
    pub = h.helicsFederateRegisterGlobalTypePublication(vfed, "Pickle", "double", "")
    h.helicsFederateEnterExecutingMode(vfed)
    simpleDict = { 'alpha' : 3, 'beta' : 0, 'zeta' : 0}
    serializedPickle = pickle.dumps(simpleDict)
    print("This is my pickle")
    print(serializedPickle)
    print("Size: {}".format(len(serializedPickle)))
    for t in range(0, 10):
        currentTime = h.helicsFederateRequestTime(vfed, t+1)
        h.helicsPublicationPublishBytes(pub, serializedPickle)
        time.sleep(1)

if __name__ == '__main__':

    initstring = "-f 2"
    broker = h.helicsCreateBroker("zmq", "", initstring)
    isconnected = h.helicsBrokerIsConnected(broker)
    if (isconnected == 1):
        print("Broker Connected")


    print("Making Pickle Process")
    pickle_process = Thread(
        target=the_pickle,
        name="Pickle Process",
        args=(),
    )
    print("Starting Pickle Process")
    pickle_process.start()

    fedinitstring = "--federates=1"
    deltat = 0.01
    fedinfo = h.helicsCreateFederateInfo()
    h.helicsFederateInfoSetCoreName(fedinfo, fedinitstring)
    h.helicsFederateInfoSetTimeProperty(fedinfo, h.helics_property_time_delta, deltat)
    vfed = h.helicsCreateValueFederate("Reciever Federate", fedinfo)
    golden_sub = h.helicsFederateRegisterSubscription(vfed, "Pickle", "")
    h.helicsFederateEnterExecutingMode(vfed)

    currentTime = -1
    last_golden_value = 0
    while currentTime <= 100:
        currentTime = h.helicsFederateRequestTime(vfed, 100)
        byteCount = h.helicsInputGetByteCount(golden_sub)
        print("Number of Bytes: {}", byteCount) # This works...
        todeserialize = h.helicsInputGetBytes(golden_sub)
        print(todeserialize) # Why doesn't this???
        whatToSay = pickle.loads(todeserialize)
        print(whatToSay['alpha'])

    print("Finished")
    h.helicsFederateDisconnect(vfed)
    h.helicsFederateFree(vfed)
    h.helicsCloseLibrary()

Output from the above

Excluding the error caused by truncated pickle data.

This is my pickle
b'\x80\x04\x95!\x00\x00\x00\x00\x00\x00\x00}\x94(\x8c\x05alpha\x94K\x03\x8c\x04beta\x94K\x00\x8c\x04zeta\x94K\x00u.'
Size: 44
Size of thing I receive: 44
This is the pickle I receive
b'\x80\x04\x95!'

As you can see, the number of bytes received by helicsInputGetByteCount is 44. But the number of bytes received by helicsInputGetBytes is only the first 4 bytes.

Proposed Solution:

Change this line of code

What is the relationship between the standard logging Python library and the logging that is set up in HELICS? Has pyhelics attempted to map them to each other? Does the HELICS logging go to a separate channel and ends up in a separate logging file?

Describe the bug

Using helics server (web interface) to launch a co-simulation fails due to where the web interface chooses to look for federates called out in the JSON runner file.

To Reproduce
Steps to reproduce the behavior:

1. Clone in the HELICS-Examples repo
2. Go to the Fundamental Default User Guide example
3. Launch the web interface with helics server
4. Go to the "Runner" tab
5. Drop in the fundamental_default_runner.json
6. Click "Run".

Environment (please complete the following information):

• Operating System: macOS 12.6.2
• Installation: pip
• helics and pyhelics version:

helics --version
HELICS Library version 3.4.0-config_with_spaces-g9d601c3b4-dirty (2023-05-09)

$ python -c "import helics as h; import json; print(json.dumps(h.helicsGetSystemInfo(), indent=4, sort_keys=True))"
{
    "buildflags": " -O3 -DNDEBUG  $<$<COMPILE_LANGUAGE:CXX>:-std=c++17>",
    "compiler": "Unix Makefiles  Darwin-21.6.0:AppleClang-14.0.0.14000029",
    "cores": [
        "zmq",
        "zmqss",
        "tcp",
        "tcpss",
        "udp",
        "ipc",
        "interprocess",
        "inproc"
    ],
    "cpu": "Apple M1 Max",
    "cpucount": 10,
    "cputype": "arm64",
    "hostname": <redacted>
    "memory": "32768 MB",
    "os": "Darwin  21.6.0  Darwin Kernel Version 21.6.0: Sun Nov  6 23:31:13 PST 2022; root:xnu-8020.240.14~1/RELEASE_ARM64_T6000",
    "version": {
        "build": "",
        "major": 3,
        "minor": 4,
        "patch": 0,
        "string": "3.4.0-config_with_spaces-g9d601c3b4-dirty (2023-05-09)"
    },
    "zmqversion": "ZMQ v4.3.4"
}

Additional context and information

Screenshot of what I'm seeing after I drop in the runner file is attached. It appears the server is intended to copy over the federate and config files to its own __helics_server folder but fails to do so. Just looking at the paths it writes in for itself makes it seem pretty clear the co-simulation will fail to launch.

pyhelics development hasn't used git flow in the past, and using a develop branch is not required.

Right now there are 3 commits that are on develop that are not a main.

Let me know if you want me to fix that and cherry pick those commits to main. After that the develop branch can probably be deleted to prevent any confusion.

Helics-apps is at version 3.3.1, while helics[cli] is at version 3.3.0.post0. This is throwing the error
[warn] helics and helics-apps versions don't match. You may want to run pip install helics helics-apps --upgrade.

To Reproduce

• Reproducible using test suite located here: https://github.com/GMLC-TDC/pyhelics/blob/main/tests/test_python_api.py

Steps to reproduce the behavior:

Environment (please complete the following information):

• Operating System:
• Installation:
• helics and pyhelics version:

helics --version

Share the output of the following:

$ python -c "import helics as h; import json; print(json.dumps(h.helicsGetSystemInfo(), indent=4, sort_keys=True))"

Additional context and information

HELICS 3.0.1 (2021-08-26)

Python HELICS bindings
Version: 3.0.1.post5

import helics as h
<...>
fed = create_federate()
fedName = h.helicsFederateGetName(fed)
<...>
swStatusEpName = fedName + '/sw_status'
swStatusEp = h.helicsFederateGetEndpoint(fed, swStatusEpName)
<...>
h.helicsEndpointSendBytes(swStatusEp, 'OPEN'.encode())

raises

Traceback (most recent call last):
  File "/mnt/pcs-test/CoSim/loadshed/helicshed.py", line 75, in <module>
    main()
  File "/mnt/pcs-test/CoSim/loadshed/helicshed.py", line 67, in main
    h.helicsEndpointSendBytes(swStatusEp, 'OPEN'.encode())
AttributeError: module 'helics' has no attribute 'helicsEndpointSendBytes'

Changed to

h.helicsEndpointSendBytesTo(swStatusEp, 'OPEN'.encode(), h.helicsEndpointGetDefaultDestination(swStatusEp))

and everything works.

However, according to the documentation, there should be a helicsEndpointSendBytes attribute. But it is not found in capi.py.

cosim-user@050bd6a38d0a:/mnt/pcs-test/CoSim/runSimulation$ cat /home/cosim-user/.local/lib/python3.9/site-packages/helics/capi.py | grep "helicsEndpointSendBytes"

returns

            helicsEndpointSendBytesTo(self, data, destination)
            helicsEndpointSendBytesToAt(self, data, destination, time)
def helicsEndpointSendBytesTo(endpoint: HelicsEndpoint, data: bytes, destination: str):
        f = loadSym("helicsEndpointSendBytesTo")
    Use `helicsEndpointSendBytesTo` instead
    warnings.warn("This function is deprecated. Use `helicsEndpointSendBytesTo` instead.")
    helicsEndpointSendBytesTo(endpoint, data, destination)
def helicsEndpointSendBytesToAt(endpoint: HelicsEndpoint, data: bytes, destination: str, time: HelicsTime):
        f = loadSym("helicsEndpointSendBytesToAt")
    Use `helicsEndpointSendBytesToAt` instead.
    warnings.warn("This function is deprecated. Use `helicsEndpointSendBytesToAt` instead.")
    helicsEndpointSendBytesToAt(endpoint, data, destination, time)

PyPI added a new feature that allows using GitHub deployment environments -- this is more secure than the current use of a token.

The current HELICS 3.x installers require macos 10.14, so we should update the platform tag in compiled wheels to reflect that (instead of 10.9).

Another upcoming change will be when we start releasing universal macOS binaries, the name of the macOS binary downloaded (and the platform tag) will need updating again. The tags used by the CMake python package are macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2, so we should consider a similar set along the lines of macosx_10_14_universal2.macosx_10_14_x86_64.macosx_11_0_arm64.macosx_11_0_universal2

Currently, any string can be used as the destination in the helicsEndpointSendBytesTo() and being inattentive to the parameter order could lead to accidentally specifying an invalid destination endpoint. There is no check in pyhelics for the validity of the endpoint specified in the API call and thus users are able to send out a message to an invalid destination.

Though there are error messages generated in the broker log, they are generic ("no route for message") and more specific error messages stating that the defined endpoint is not valid would be helpful.

I'll leave it up the the developers to determine if this should be implemented in the language bindings or if an update to the core library is needed.

The pythonic API for HelicsInput should allow a general .value property that converts to the type given by .type.

On Mac OS 10.14 using Python 3.8.5, HELICS 3.0.0-alpha.2, and pyhelics 3.0.0-alpha.2

Doesn't work:

    fake_max_time = h.HELICS_TIME_MAXTIME
    starttime = fake_max_time
    grantedtime = h.helicsFederateRequestTime (fed, starttime)

Works:

    fake_max_time = int(h.HELICS_TIME_MAXTIME)
    starttime = fake_max_time
    grantedtime = h.helicsFederateRequestTime (fed, starttime)

Describe the bug
pip install helics fails in python 3.10 official image

root@239d5ba4c422:/proj# pip install helics
Collecting helics
  Downloading helics-3.3.0.tar.gz (289 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 289.2/289.2 kB 3.4 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      running egg_info
      running jsdeps
      `npm` unavailable.  If you're running this command using sudo, make sure `npm` is available to sudo
      rebuilding js and css failed
      missing files: ['index.html']
      error: [Errno 2] No such file or directory: '/tmp/pip-install-piqblljm/helics_2570b8aac49544ba90e0b22040b7c664/client'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

To Reproduce

Steps to reproduce the behavior:
In Dockerfile:

FROM python:3.10

RUN pip install helics

Environment (please complete the following information):

• Operating System: Docker version 20.10.17, build 100c701
• helics and pyhelics version: latest

Share the output of the following:

$ python -c "import helics as h; import json; print(json.dumps(h.helicsGetSystemInfo(), indent=4, sort_keys=True))"
ModuleNotFoundError: No module named 'helics'
root@239d5ba4c422:/proj# pip install helics

Additional context and information

I've discovered a weird inexplicable conflict between helics and matplotlib.

The bug is if matplotlib is imported before helics then helics.helicsCreateCombinationFederateFromConfig() raises an exception when trying to connect to a remote broker (in a separate docker container).

The exception I get with the latest versions of helics is:

helics.capi.HelicsException: [-3] --brokerport: Value  not in range 0.000000 to 179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368.000000

Here's a gist with a a minimal docker-compose.yml, Dockerfile image and a simple reproduction.py script which triggers this.

https://gist.github.com/devanubis/ec452950c317d3684016dd3b609ebca3

This can be executed with just docker-compose up.

I've found the following:

• Moving the import matplotlib to after import helics does not result in the same exception being triggered
  • Which is very strange
• The exception does trigger if running the federate and broker on the same host
  • Without the broker_address but with the broker_port
  • It was just easiest to do the reproduction with docker

I don't know enough about either library to dig much deeper.

I can also report this to https://github.com/matplotlib/matplotlib if you want, but thought I'd start with pyhelics since the exception is from here.

Fortunately I've been able to remove matplotlib from our affected code, so this isn't urgent for us, but it was painful to track down what was causing this weirdness.

Under the v3beta, two of the User Guide examples that are very similar have different logging behaviors:

• Fundamental default example- only one logging message per logging event; this is the desired behavior

Entered HELICS execution mode
Requesting time 60
Granted time 60.0
Battery 1 time 60.0

• Fundamental endpoint example is similar but uses messages instead of pub/sub and produces two logging messages per logging event; this is not the desired behavior

Entered HELICS execution mode
INFO:__main__:Entered HELICS execution mode
Requesting time 70
DEBUG:__main__:Requesting time 70
Granted time 70.0
DEBUG:__main__:Granted time 70.0
Battery 1 time 70.0
DEBUG:__main__:Battery 1 time 70.0

I'm assuming this is a PyHELICS thing but I'm not sure. I've looked at the two config files and the runner config and I'm not seeing any differences that would explain what's happening.

@phlptp has confirmed that the HELICS library generates error messages back to the federate when sending messages to an invalid endpoint (as was being done in this older version of this example). This error message is not making it all the way to the federate log when using pyhelics.

Its not clear to me if the new HELICS library logging levels are being appropriately supported in pyhelics 2.7.1. It seems like I'm able to get away with not using a valid logging level (e.g. "1") when I thought that was deprecated in HELICS v3.

I don't 100% understand the cause of this issue, but I do have a solution. So, rather than trying to verbosely describe what's going on, I'm going to let the code do the talking.

Platform/Versions

OS

OS: Windows 10 Pro
Version: 20H2
Build: 19042.1081
Experience: Windows Feature Experience Pack 120.2212.3530.0

Software

• Python: 3.8.8
• pytest: 6.2.4
• pyhelics (simply helics on PyPI): v.2.7.1.post1
• helics (plus apps): 2.7.1

Reproducing

After setting up a virtual environment (outside the scope of this ticket), put the following code in test_bug.py:

import helics as h
import pytest


class ComboFed:
    """Simple wrapper around a HelicsCombinationFederate."""

    def __init__(self, timeout, core_type, broker_name):
        fed_info = h.helicsCreateFederateInfo()

        # Configure and create the HELICS federate.
        # Basic properties.
        fed_info.core_name = 'test_fed'
        fed_info.core_init = (
            f'--federates=1 '
            f'--network_timeout={timeout}s'
        )
        fed_info.core_type = core_type
        fed_info.broker = broker_name

        # Hard-code the period (no "Pythonic" property available yet)
        h.helicsFederateInfoSetTimeProperty(
            fi=fed_info,
            time_property=h.helics_property_time_period,
            value=60.0
        )

        # Create the federate.
        self.federate = h.helicsCreateCombinationFederate(
            fed_name='test_fed', fi=fed_info
        )


def test_cannot_create_combo_fed_without_broker():
    # Note that this test is both listed first and comes alphabetically
    # before "test_create_combo_fed_with_broker" so should be executed
    # first.
    with pytest.raises(h.HelicsException, match='Unable to connect to broker'):
        ComboFed(timeout=0.5, core_type='inproc', broker_name='test_broker')


def test_create_combo_fed_with_broker():
    broker = h.helicsCreateBroker(
        type='inproc',
        name="test_broker",
        init_string=f"--federates 1"
    )
    fed = ComboFed(timeout=0.5, core_type='inproc', broker_name='test_broker')
    assert isinstance(fed.federate, h.HelicsCombinationFederate)

Now, run the tests (in your activated virtual environment): pytest test_bug.py. Expected output:

collected 2 items

test_cleanup_bug.py .F                                                                                                                                                                                                                [100%]

================================================================================================================= FAILURES =================================================================================================================
____________________________________________________________________________________________________ test_create_combo_fed_with_broker _____________________________________________________________________________________________________

    def test_create_combo_fed_with_broker():
        broker = h.helicsCreateBroker(
            type='inproc',
            name="test_broker",
            init_string=f"--federates 1"
        )
>       fed = ComboFed(timeout=0.5, core_type='inproc', broker_name='test_broker')

test_cleanup_bug.py:48:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test_cleanup_bug.py:29: in __init__
    self.federate = h.helicsCreateCombinationFederate(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

fed_name = 'test_fed', fi = <helics.HelicsFederateInfo() at 0x21290a48730>

    def helicsCreateCombinationFederate(fed_name: str, fi: HelicsFederateInfo = None) -> HelicsCombinationFederate:
        """
        Create a combination federate from `helics.HelicsFederateInfo`.
        Combination federates are both value federates and message federates, objects can be used in all functions
        that take a `helics.HelicsFederate` object as an argument.

        **Parameters**

        - **`fed_name`** - A string with the name of the federate, can be NULL or an empty string to pull the default name from fi.
        - **`fi`** - The federate info object that contains details on the federate.

        **Returns**: `helics.HelicsCombinationFederate`.
        """
        f = loadSym("helicsCreateCombinationFederate")
        err = helicsErrorInitialize()
        if fi is None:
            fi = helicsCreateFederateInfo()
        result = f(cstring(fed_name), fi.handle, err)
        if err.error_code != 0:
>           raise HelicsException("[" + str(err.error_code) + "] " + ffi.string(err.message).decode())
E           helics.capi.HelicsException: [-1] Unable to connect to broker->unable to register federate

..\venv\lib\site-packages\helics\capi.py:3267: HelicsException
========================================================================================================= short test summary info ==========================================================================================================
FAILED test_cleanup_bug.py::test_create_combo_fed_with_broker - helics.capi.HelicsException: [-1] Unable to connect to broker->unable to register federate
======================================================================================================= 1 failed, 1 passed in 1.00s ========================================================================================================

Notice that test_create_combo_fed_with_broker failed unexpectedly.

Next, comment out the entirety of test_cannot_create_combo_fed_without_broker and execute the tests (well, just one test this time) again. Expected output:

collected 1 item

test_cleanup_bug.py .                                                                                                                                                                                                                 [100%]

============================================================================================================ 1 passed in 0.34s =============================================================================================================

These two tests are completely isolated, yet test_cannot_create_combo_fed_without_broker is interfering with test_create_combo_fed_with_broker.

Next, uncomment test_cannot_create_combo_fed_without_broker to restore the test module to its original state. Now, add the following fixture before test_cannot_create_combo_fed_without_broker:

@pytest.fixture(scope='class', autouse=True)
def clean_helics():
    h.helicsCleanupLibrary()

The autouse=True flag means this fixture will be run before every test.

Expected output after re-running the tests:

collected 2 items

test_cleanup_bug.py ..                                                                                                                                                                                                                [100%]

============================================================================================================ 2 passed in 1.01s =============================================================================================================

Note that I cannot provide any guarantee that helicsCleanupLibrary is actually solving the problem by whatever cleanup actions it is taking. If you add an import time line to the top of the module and replace h.helicsCleanupLibrary() with time.sleep(1) the tests also pass.

When a HelicsMessageFederate (or a HelicsCombinationFederate) is created with helicsCreateMessageFederateFromConfig(), the endpoints field does not include any of the endpoints from the config file. For example:

{
  "name": "foo",
  "endpoints": [
    {
        "name": "ep"
    }
  ]
}

from helics import helicsCreateMessageFederateFromConfig
fed = helicsCreateMessageFederateFromConfig("foo.json")
# this raises a KeyError
fed.endpoints["ep"]
# this returns the endpoint
fed.get_endpoint_by_name("ep")

It would be nice to have uniform handling of endpoints registered via fed.register_endpoint() and endpoints registered by the config file, the same as publications and inputs.

The call to helicsInputGetComplex() issues a UserWarning with the message "This function will return a complex number in the next major release". Since the "pythonic" API is already converting the output of this function to a complex number, this warning is misleading/surprising here.

I think the warnings.catch_warnings context manager would be useful here to filter this warning just from this call, and restore the previous warnings filter/configuration after returning.

Describe the bug

The federate class method "get_subscription_by_name" doesn't work for subscriptions since, by definition, they are unnamed inputs.

To Reproduce

Steps to reproduce the behavior:

Configure a federate using a JSON with the "subscription" object and try using the above method to get the subscription object. It will fail with a HELICS error saying there is no subscription by that name.

Environment (please complete the following information):

• Operating System: Mac 10.15.7
• Installation: pip install helics
• helics and pyhelics version: 3.3.0 for both

Share the output of the following:

$ python -c "import helics as h; import json; print(json.dumps(h.helicsGetSystemInfo(), indent=4, sort_keys=True))"


{
    "buildflags": " -O3 -DNDEBUG  $<$<COMPILE_LANGUAGE:CXX>:-std=c++17>",
    "compiler": "Unix Makefiles  Darwin-19.6.0:AppleClang-12.0.0.12000032",
    "cores": [
        "zmq",
        "zmqss",
        "tcp",
        "tcpss",
        "udp",
        "ipc",
        "interprocess",
        "inproc"
    ],
    "cpu": "Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz",
    "cpucount": 8,
    "cputype": "x86_64",
    "hostname": "WE34262",
    "memory": "16384 MB",
    "os": "Darwin  19.6.0  Darwin Kernel Version 19.6.0: Tue Jun 21 21:18:39 PDT 2022; root:xnu-6153.141.66~1/RELEASE_X86_64",
    "version": {
        "build": "",
        "major": 3,
        "minor": 3,
        "patch": 0,
        "string": "3.3.0 (2022-09-15)"
    },
    "zmqversion": "ZMQ v4.3.4"
}

Additional context and information
Here's the logging output we're seeing with the error message:

Created federate transmission
      Number of subscriptions: 1
      Number of publications: 11
      Registered subscription---> distribution5/pcc.11.pq
Traceback (most recent call last):
  File "/home/ananth/code/NRECA/HELICS/multisite-demo/distribution5/transmission_dummy.py", line 62, in <module>
    subid[k] = fed.get_subscription_by_name(k)
  File "/home/ananth/.local/lib/python3.10/site-packages/helics/capi.py", line 2713, in get_subscription_by_name
    return helicsFederateGetInput(self, name)
  File "/home/ananth/.local/lib/python3.10/site-packages/helics/capi.py", line 7768, in helicsFederateGetInput
    raise HelicsException("[" + str(err.error_code) + "] " + ffi.string(err.message).decode())
helics.capi.HelicsException: [-4] the specified input name is a not a recognized input

capi.py for pyhelics adds the subscription objects to the federates "subscriptions" attribute (a dictionary keyed to the subscription target) by referencing the subscription objects by index; both inputs and subscriptions are put into the same dictionary and then the "inputs" attribute is equated to the "subscription" attribute. The above method takes a string (the subscription name) as an input and calls "helicsFederateGetInput()". This will always fail for subscriptions since they are, by definition, unnamed.

The best improvement we could make here is:

• Correctly populating inputs and subscriptions in the federate attributes so that they are unique dictionaries
• Adding a "n_inputs" attribute for the HelicsFederateClass
• Changing the name of "get_subscriptions_by_name" to "get_input_by_name"
• Probably other things I haven't thought of yet.

If HELICS v3.0.0-alpha.2 was built and installed with the C++ library active, the logic in _build.py responsible for library loading will accidentally bind and load the C++ shared library instead of the C shared library, breaking some bindings.
The responsible line appears to be if "libhelics" in file: on MacOS (other operating systems may share similar bug after shared library renames).
Modifying the line as follows resolved the issue on MacOS, however this may not be the desired solution: if "libhelics" in file and not "cpp" in file:

I have the following code snippet that works with h.helicsFederateRegisterGlobalTypePublication but raises the following error when h.helicsFederateRegisterPublication is used instead

File "c:\users\alatif\documents\github\cymepy\cymepy\cli\run.py", line 53, in run
instance = cymeInstance(settings)
File "c:\users\alatif\documents\github\cymepy\cymepy\cymepy.py", line 93, in init
self.HI = HELICS(SettingsDict, self.cympy, self.simObj, self.__Logger)
File "c:\users\alatif\documents\github\cymepy\cymepy\helics_interface.py", line 48, in init
self.register_standard_publications()
File "c:\users\alatif\documents\github\cymepy\cymepy\helics_interface.py", line 180, in register_standard_publications
self.create_propery_publication(device, cName, pubInfo)
File "c:\users\alatif\documents\github\cymepy\cymepy\helics_interface.py", line 191, in create_propery_publication
self.create_publication_object( device, cName, ppty_name, pubInfo)
File "c:\users\alatif\documents\github\cymepy\cymepy\helics_interface.py", line 200, in create_publication_object
self.update_publication_object(hmap, pubInfo)
File "c:\users\alatif\documents\github\cymepy\cymepy\helics_interface.py", line 211, in update_publication_object
hmap.units
File "C:\Users\alatif\Anaconda3\envs\cymepy\lib\site-packages\helics\capi.py", line 6778, in helicsFederateRegisterPublication
result = f(fed.handle, cstring(name), HelicsDataType(type), cstring(units), err)
File "C:\Users\alatif\Anaconda3\envs\cymepy\lib\enum.py", line 307, in call
return cls.new(cls, value)
File "C:\Users\alatif\Anaconda3\envs\cymepy\lib\enum.py", line 555, in new
return cls.missing(value)
File "C:\Users\alatif\Anaconda3\envs\cymepy\lib\enum.py", line 568, in missing
raise ValueError("%r is not a valid %s" % (value, cls.name))
ValueError: 'COMPLEX' is not a valid HelicsDataType

The code included in this runs on Windows
it doesn't on Linux
For some reason I do not understand, the Process is unable to connect to the broker.

I believe that the issue is caused by the difference in implementation between Multiprocessing on Windows and Linux, and some interaction that PyHelics has with that difference in implementation.

However I do not know the actual cause.

Platform/Versions

OS

OS: Windows 10 Enterprise
Version: 20H2
Build: 19042.1165
Experience: Windows Feature Experience Pack 120.2212.3530.0

OS: Ubuntu
Version: 20.04.2 LTS

Software

python: 3.8.10
pyhelics: 3.0.0.0

Code

from multiprocessing import Process
import multiprocessing
import helics as h
import time

class GoldenModel():
    def __init__(self):
        deltat = 0.01
        fedinitstring = "--federates=1"
        fedinfo = h.helicsCreateFederateInfo()
        h.helicsFederateInfoSetCoreName(fedinfo, "Golden Federate")
        h.helicsFederateInfoSetCoreTypeFromString(fedinfo, "zmq")
        h.helicsFederateInfoSetCoreInitString(fedinfo, fedinitstring)
        h.helicsFederateInfoSetTimeProperty(fedinfo, h.helics_property_time_delta, deltat)
        self.vfed = h.helicsCreateValueFederate("Golden Federate", fedinfo)
        self.pub = h.helicsFederateRegisterGlobalTypePublication(self.vfed, "Golden", "double", "")

    def __del__(self):
        h.helicsFederateDisconnect(self.vfed)
        h.helicsFederateFree(self.vfed)
        pass

    def exec(self):
        h.helicsFederateEnterExecutingMode(self.vfed)
        goldenRatio = 1.6180339887
        currentTime = 0
        for t in range(0, 10):
            val = goldenRatio + t
            currentTime = h.helicsFederateRequestTime(self.vfed, t+1)
            h.helicsPublicationPublishDouble(self.pub, val)
            time.sleep(1)
        
def golden_model():
    p = GoldenModel()
    p.exec()
    pass


if __name__ == '__main__':
    multiprocessing.freeze_support()

    initstring = "-f 2"
    broker = h.helicsCreateBroker("zmq", "", initstring)
    isconnected = h.helicsBrokerIsConnected(broker)
    if (isconnected == 1):
        print("Broker Connected")


    print("Making Golden Process")
    golden_process = Process(
        target=golden_model,
        name="GoldenModel",
        args=(),
    )
    print("Starting Golden Process")
    golden_process.start()

    fedinitstring = "--federates=1"
    deltat = 0.01
    fedinfo = h.helicsCreateFederateInfo()
    h.helicsFederateInfoSetCoreName(fedinfo, fedinitstring)
    h.helicsFederateInfoSetTimeProperty(fedinfo, h.helics_property_time_delta, deltat)
    vfed = h.helicsCreateValueFederate("Reciever Federate", fedinfo)
    golden_sub = h.helicsFederateRegisterSubscription(vfed, "Golden", "")
    h.helicsFederateEnterExecutingMode(vfed)

    currentTime = -1
    last_golden_value = 0
    while currentTime <= 100:
        currentTime = h.helicsFederateRequestTime(vfed, 100)
        temp = h.helicsInputGetDouble(golden_sub)
        if (temp != last_golden_value):
            last_golden_value = temp
            print("Recieved Value {}".format(temp))

    print("Finished")
    h.helicsFederateDisconnect(vfed)
    h.helicsFederateFree(vfed)
    h.helicsCloseLibrary()

Error that occurs running on a Linux machine.

Broker Connected
Making Golden Process
Starting Golden Process
[2021-08-19 09:16:56.099] [console] [warning] commWarning||Golden Federate (0)[t=-98763.2]::zmq broker connection timed out, trying again (2)
[2021-08-19 09:16:56.099] [console] [warning] commWarning||Golden Federate (0)[t=-98763.2]::sending message to tcp://127.0.0.1:23405
[2021-08-19 09:17:16.122] [console] [error] commERROR||Golden Federate (0)[t=-98763.2]::zmq broker connection timed out after trying 5 times (2)
[2021-08-19 09:17:16.122] [console] [error] commERROR||Golden Federate (0)[t=-98763.2]::receiver connection failure
Process GoldenModel:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "goldenmultiprocessor.py", line 37, in golden_model
    p.exec()
  File "goldenmultiprocessor.py", line 24, in exec
    self.vfed = h.helicsCreateValueFederate("Golden Federate", fedinfo)
  File "/home/gang/.local/lib/python3.8/site-packages/helics/capi.py", line 3204, in helicsCreateValueFederate
    raise HelicsException("[" + str(err.error_code) + "] " + ffi.string(err.message).decode