opendxl / opendxl-client-python Goto Github PK

When a an error occurs and the client is unexpectedly disconnected from the broker it will automatically reconnect.
If this occurs when responding to multiple concurrent requests each thread will trigger a reconnect via the on_disconnect MQTT callback. Thus multiple on_connect and on_disconnect callback threads will be running concurrently.

It can occur that an on_disconnect callback thread is holding the ServiceManager Rlock and is attempting to acquire the _connected_lock in DxlClient (via the connected method) while at the same time an on_connect callback thread is attempting to acquire the ServiceManager lock (via ServiceManager.on_connect called from _on_connect_run)

This only occurs when many concurrent requests all trigger the broker to disconnect the client as it does when a oversized message is received. 6-10 concurrent requests will usually trigger the deadlock.

This is related to and/or the cause of #55

Noticed a lot of dependencies on python2 (e.g print statement, except, ...) - gave me some headaches trying to modify the code for python3 until I sucked it up and installed python2 to get the client to run.

Should at min document python2 required

opendxl client stop working and hangs there when concurrent responses have message size larger than messageSizeLimit( default to 1048576)
From debug message it blocked on:

{"written_at": "2020-12-22T15:38:47.859Z", "written_ts": 1608651527859996160, "type": "log", "msg": "disconnect on <dxlclient.client.DxlClient object at 0x7f141e1c7e50>, True", "logger": "dxlclient.client", "thread": "Thread-20", "level": "INFO", "module": "client", "line_no": 140, "filename": "client.py"}
{"written_at": "2020-12-22T15:38:47.860Z", "written_ts": 1608651527860232960, "type": "log", "msg": "Unexpected disconnect with result code 1", "logger": "dxlclient.client", "thread": "Thread-20", "level": "ERROR", "module": "client", "line_no": 150, "filename": "client.py"}
{"written_at": "2020-12-22T15:38:47.860Z", "written_ts": 1608651527860377088, "type": "log", "msg": "Waiting for thread pool completion...", "logger": "dxlclient.client", "thread": "Thread-20", "level": "INFO", "module": "client", "line_no": 581, "filename": "client.py"}

Once this happened, following request got 'unable to locate service for request (-2147483647)'

PySocks 1.6.8 is using deprecated features that will cause an error in 3.10. PySocks 1.7 fixes the issue

The PySocks required version should be changed.

dxlclient failed to startup when we used python:3-slim image which is in python 3.9.

Oct 22 16:14:46 localhost journal: File "/usr/local/lib/python3.9/site-packages/dxlbootstrap/app.py", line 268, in run
Oct 22 16:14:46 localhost journal: self._dxl_connect()
Oct 22 16:14:46 localhost journal: File "/usr/local/lib/python3.9/site-packages/dxlbootstrap/app.py", line 246, in _dxl_connect
Oct 22 16:14:46 localhost journal: self._dxl_client.connect()
Oct 22 16:14:46 localhost journal: File "/usr/local/lib/python3.9/site-packages/dxlclient/client.py", line 502, in connect
Oct 22 16:14:46 localhost journal: while self._thread.isAlive():
Oct 22 16:14:46 localhost journal: AttributeError: 'Thread' object has no attribute 'isAlive'

Can the opendxl project claim the name space and publish dxlclient to pypi

PyPI - the Python Package Index
https://pypi.python.org/pypi?%3Aaction=search&term=dxlclient&submit=search

Currently, the on_connect_run logic in the client attempts to resubscribe for all previously subscribed topics when a reconnect occurs. This logic happens here. If any subscriptions were to fail, the current logic just writes out an error to the console but does not retry the attempt at a later time. This could leave the client in a state where some subscriptions are unexpectedly not active until / unless the client happens to reconnect again. There should be some additional logic added to retry the failed subscriptions - perhaps by triggering the full disconnect / reconnect sequence up to the number of connect retries which have been configured.

When a service re-registering an exception in send_register_service_request() will cause the registration to fail and it will not be retried and the timer will not be restarted. As a result the service will no longer be registered.

See:

We occasionally see WaitTimeoutException occur in send_register_service_request() to trigger this problem.

This is a similar but different problem to #34

Please use only one README file.

README.md is the default for git repositories on github.

Maintaining seperate files will only lead to confusion or worse bad data.

Tried installing dxlclient-3.0.1.170 via pip on Windows for a 32-bit Python 3.5 environment. It failed with "invalid syntax" on broker.py, line 223.

I followed the instructions for the certificates, but when I try to run the examples I get this:

2016-12-12 10:43:18,717 dxlclient.client - ERROR - Failed to connect to broker {Unique id: {#### Scrubbed ####}, Host name: tie01.test.lab, IP address: 192.168.1.32, Port: 8883}:
2016-12-12 10:43:18,739 dxlclient.client - ERROR - Failed to connect to broker {Unique id: {#### Scrubbed ####}, Host name: dxl01.test.lab, IP address: 192.168.1.31, Port: 8883}:
2016-12-12 10:43:19,791 dxlclient.client - ERROR - Failed to connect to broker (host name) {Unique id: {#### Scrubbed ####}, Host name: tie01.test.lab, IP address: 192.168.1.32, Port: 8883}: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:581)
2016-12-12 10:43:19,815 dxlclient.client - ERROR - Failed to connect to broker (IP address) {Unique id: {#### Scrubbed ####}, Host name: tie01.test.lab, IP address: 192.168.1.32, Port: 8883}: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:581)
2016-12-12 10:43:19,836 dxlclient.client - ERROR - Failed to connect to broker (host name) {Unique id: {#### Scrubbed ####}, Host name: dxl01.test.lab, IP address: 192.168.1.31, Port: 8883}: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:581)

This properties should be set to correct values by broker, for now they can be manipulated by client.
Broker version: 3.0.1 (build 186) (core 1.3.5)

The following code:

event = Event(EVENT_TOPIC)
event._source_broker_id = "aaaa"
event._client_ids = "bbbb"
event._source_client_id = 'cccc'
client.send_event(event)

Results in _source_client_id and _source_broker_id substitution:

{'_source_client_id': 'cccc', '_payload': '', '_version': 0, '_client_ids': 'bbbb', '_destination_topic': '/isecg/sample/basicevent', '_broker_ids': [], '_source_broker_id': 'aaaa', '_message_id': '{37134c4c-90e5-4a3e-adc2-6e419946af55}'}

Seems the provided samples cannot run without having an instance of ePO Broker:
https://opendxl.github.io/opendxl-client-python/pydoc/epobrokerlistexport.html
Is there any procedure to generate a valid broker list without having to own and run an ePO instance? Having a complete (service + client) runnable demo would be quite helpful.
Thanks!

Hi,

Using dxlclient==3.1.0.586, we have a script that sends a batch of events (~1000) and then terminates, but sometimes the last few events are not sent or lost. Neither the brokers nor the network are under heavy load so we wouldn't expect any messages to be lost. We are using the DxlClient's send_event method and its context manager to handle the disconnection.

The issue can be reproduced by splitting the sender and listener from https://github.com/opendxl/opendxl-client-python/blob/master/examples/basic/event_example.py into separate scripts. Please help.

Damian

Hi guys
I downloaded the current package and when I run it, I get an ImportError regarding "Callable" from "Collections".

The Python version I'm using is Python 3.10.12.
I tested this on a Windows machine (Win11) as well as an Ubuntu 22.04.1 Ubuntu.

I did the initial installation using "pip installdxlclient-5.6.0.5-py2.py3-none-any.whl".
However, I also tried to install everything from the ./lib/dxlclient-5.6.0.5.zip using "python setup.py install".

Has the Collections module received a short-term update, or is my approach wrong?
Thanks for your help

Hello,

I'm trying to provision a Linux client but getting a failure message (below) immediately after generating the client.key. Any assistance is appreciated.

Python 2.7.13
'OpenSSL 1.0.2k-fips 26 Jan 2017'
dxlclient 4.1.0.185
RHEL 7.4
ePO 5.9

-bash-4.2$ python -m dxlclient provisionconfig config x.x.x.x epo-subjectname

Enter server username:

Enter server password:

INFO: Saving csr file to config/client.csr

INFO: Saving private key file to config/client.key

ERROR: Command failed. Message: Request to x.x.x.x:8443/DxlBrokerMgmt.generateOpenDXLClientProvisioningPackageCmd failed with status: Error 0. Message: For input string: "7684354456"

Ref : python/cpython#25174

Camelcase aliases have been deprecated and raise deprecation warning.

notifyAll -> notify_all
currentThread -> current_thread

opendxl-client-python/dxlclient/_request_manager.py:                    self.sync_wait_message_condition.notifyAll()
opendxl-client-python/dxlclient/client.py:                self._connect_wait_condition.notifyAll()
opendxl-client-python/dxlclient/client.py:        if threading.currentThread().name.startswith(self._message_pool_prefix):

Mcafee-DXL.zip

eggs wheels , ,exe , MSI

python setup.py build bdist_egg bdist_wininst bdsit_msi for various Open-dxl related items
(bdist_wheel when possible) on windows , bdist_rpm can be cone on linux boxes...

can add to distributions https://github.com/opendxl/****/releases if you like.

When attempting to run the "provisionconfig" CLI tool on macOS High Sierra, an "Error initializing ctypes" error is displayed and none of the provisioning artifacts - private key, certificates, or configuration - are created. Here a sample output from running this command:

› python -m dxlclient provisionconfig sample myeposerver client
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/dxlclient/__main__.py", line 9, in <module>
    from dxlclient._cli import cli_run
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/dxlclient/_cli/__init__.py", line 13, in <module>
    from dxlclient._cli._cli_subcommands import \
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/dxlclient/_cli/_cli_subcommands.py", line 18, in <module>
    from dxlclient._cli._crypto import X509Name, validate_cert_pem, \
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/dxlclient/_cli/_crypto.py", line 15, in <module>
    from oscrypto import asymmetric
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/asymmetric.py", line 15, in <module>
    from .kdf import pbkdf2, pbkdf2_iteration_calculator
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/kdf.py", line 9, in <module>
    from .util import rand_bytes
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/util.py", line 10, in <module>
    from ._osx.util import rand_bytes
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/_osx/util.py", line 208, in <module>
    from .._openssl._libcrypto import libcrypto
  File "/virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/_openssl/_libcrypto.py", line 14, in <module>
    from ._libcrypto_ctypes import (
  File "/virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/_openssl/_libcrypto_ctypes.py", line 668, in <module>
    raise FFIEngineError('Error initializing ctypes')
oscrypto._ffi.FFIEngineError: Error initializing ctypes

I saw the error both with Python 2.7.10 and 3.7.0. I was also able to reproduce the issue with two different SSL versions - LibreSSL 2.2.7 (default SSL installed with High Sierra) and a Homebrew-installed OpenSSL 1.0.2p (14 Aug 2018). In each case, I had the latest oscrypto release, 0.19.1, installed.

This was raised as an issue to the oscrypto project, wbond/oscrypto#22. The issue has been fixed but has not been delivered in a new release of the oscrypto library yet.

In each case, I was able to perform the provisionconfig command successfully by performing either (or both) of the following:

• Run pip install cffi.
• Run pip uninstall oscrypto, followed by pip install git+https://github.com/wbond/oscrypto@af778bf.

oscrypto version af778bf was the head of the master branch at the time I ran the test.

A separate PR which would include cffi as a formal project dependency, #35, has been proposed.

Dxlclient._config._client_id is generated or taken from config file at start. It can not be used in communication neither as message.source_id (for checking sender) nor event.client_ids (for limiting recipient) as these are different values, unreachable directly from client's code.

With newly build docker image, notice it's pulling msgpack-1.0.0 and it failed to register topic with follwoing error. Switch back to msgpack-0.6.2 solved the problem. Wondering if could do something in dxlclient to solve this issue. dxlbroker ran in old image.

docker.io/opendxl/opendxl-broker                                     latest              7a3343070a4a        3 months ago        261 MB

2020-02-18 20:10:20,756 gtdbapiservice.app INFO     Registering topic: /twapi/v1/cti/gtdb/domain/report
2020-02-18 20:10:20,759 dxlclient._thread_pool ERROR    Error in worker thread
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/dxlclient/_thread_pool.py", line 54, in run
    func(*args, **kargs)
  File "/usr/local/lib/python3.8/site-packages/dxlclient/client.py", line 1158, in _handle_message
    message = Message._from_bytes(payload)
  File "/usr/local/lib/python3.8/site-packages/dxlclient/message.py", line 372, in _from_bytes
    message._unpack_message(unpacker)
  File "/usr/local/lib/python3.8/site-packages/dxlclient/message.py", line 554, in _unpack_message
    super(Response, self)._unpack_message(unpacker)
  File "/usr/local/lib/python3.8/site-packages/dxlclient/message.py", line 224, in _unpack_message
    self._message_id = self._unpack_next_unicode_string(unpacker)
  File "/usr/local/lib/python3.8/site-packages/dxlclient/message.py", line 389, in _unpack_next_unicode_string
    return Message._decode_to_unicode_string(next(unpacker))
  File "/usr/local/lib/python3.8/site-packages/dxlclient/message.py", line 385, in _decode_to_unicode_string
    return None if obj is None else obj.decode('utf8')
AttributeError: 'str' object has no attribute 'decode'