Test Suite for testing protocol compliance of Aries Agents

Home Page: https://wiki.hyperledger.org/display/aries

License: Apache License 2.0

Python 98.90% Shell 1.10%

aries-protocol-test-suite's People

Contributors

Stargazers

Watchers

Forkers

dbluhm hyperledger-snapshot-8-12-2019 llorllale telegramsam burdettadam artemkaaas smithbk911 smithbk hyperledger-aries-for-p2p panickervinod hyperledgerkochi ianco tangelo1 huhn511 l3-igrant decentralised-dataexchange aviarytech pinkdiamond1

aries-protocol-test-suite's Issues

Create a MAINTAINERS.md file

Example: https://github.com/hyperledger-labs/solang/blob/main/MAINTAINERS.md Example of a pointer: https://github.com/hyperledger-labs/homebrew-solang/blob/main/MAINTAINERS.md

Loading conftest fails (likely pytest version issue)

I suspect this is a version issue as this works with pytest version 4.6.3.
stacktrace.txt

Officially deprecate and archive this project?

The last commit on this project was in February 2021. Generally speaking, there has been more interest in testing an agent against another agent using the Aries Agent Test Harness rather than testing an agent against this project. As a result, this project has seen a lack of interest or contributions for some time. As one of the original authors, I think it's time to officially deprecate and archive this project.

@ryjones @swcurran @smithbk @TelegramSam @dhh1128

publish test results in standard data structure

Would like to use a data structure like this one: https://github.com/hyperledger/indy-hipe/tree/master/text/0015-agent-test-suite-interface#interop-profile-json

This PR against the aries-rfcs repo calls for implementations to formally publish test results; I'd like this data structure to be the embodiment of what's published. That is, a hyperlink to a JSON file containing this structure should be how Aries RFC implementations report their interoperability.

It might also be nice (but is less important) for this data structure to be returned as part of the disclose message of the Discover Features protocol.

need tests for DID Exchange / Connection Protocol

All of the following tests are needed.

An overly long invitation is rejected since it can't be turned into a QR code?
Agent can be either inviter or invitee.
A public DID can be used as an implicit invitation.
A connection resp that is not signed by the key used in the invitation generates a graceful error.
A connection resp sent before a connection req generates a graceful error.
A connection resp or connection req that uses a different thread from the previous message generates a graceful error.
A connection resp or req that uses the same thread as previous, but an inconsistent state (e.g., different DID Doc or endpoint) is rejected.
DID Docs with extra fields are handled.
DID Docs lacking required fields are rejected.
DID Docs lacking public keys of type {Ed25519|Secp256k1|RSA} are either handled fully, or generate a graceful error.
A plaintext version of a connection request or connection response is gracefully rejected.

need tests for decorators

All of the following tests are needed:

The ~thread decorator is handled correctly (including thid and received_orders)
The ~timing decorator is handled correctly.
The ~please_ack decorator is handled correctly.

Unable to run the test suite - No suitable handler for message of type

Stacktrace for reference

apts    | 
apts    | Attempting to load configuration from file: /aries-protocol-test-suite/config.toml
apts    | 
apts    | ============================= test session starts ==============================
apts    | platform linux -- Python 3.6.9, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
apts    | cachedir: .pytest_cache
apts    | rootdir: /aries-protocol-test-suite, configfile: pytest.ini
apts    | plugins: asyncio-0.14.0
apts    | collecting ... collected 21 items / 11 deselected / 10 selected
apts    | 
apts    | protocol_tests/connection/test_connection.py::test_connection_started_by_tested_agent FAILED [ 10%]
apts    | 
apts    | =================================== FAILURES ===================================
apts    | ___________________ test_connection_started_by_tested_agent ____________________
apts    | 
apts    | inviter = <async_generator object _inviter at 0x7f5b0d6f0528>
apts    | 
apts    |     @pytest.mark.asyncio
apts    |     @meta(protocol='connections', version='1.0',
apts    |           role='inviter', name='can-be-inviter')
apts    |     async def test_connection_started_by_tested_agent(inviter):
apts    |         """Test a connection as started by the agent under test."""
apts    | >       await run(inviter)
apts    | 
apts    | protocol_tests/connection/test_connection.py:326: 
apts    | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apts    | protocol_tests/__init__.py:234: in run
apts    |     async for _event, *_data in generator:
apts    | protocol_tests/connection/test_connection.py:296: in _inviter
apts    |     timeout=30
apts    | /usr/local/lib/python3.6/dist-packages/aries_staticagent/static_connection.py:334: in send_and_await_reply_async
apts    |     anoncrypt=anoncrypt,
apts    | /usr/local/lib/python3.6/dist-packages/aries_staticagent/static_connection.py:314: in send_async
apts    |     partial(_error_handler, self)
apts    | /usr/local/lib/python3.6/dist-packages/aries_staticagent/utils.py:71: in http_send
apts    |     await response_handler(body)
apts    | /usr/local/lib/python3.6/dist-packages/aries_staticagent/static_connection.py:305: in _response_handler
apts    |     await self.handle(msg)
apts    | /usr/local/lib/python3.6/dist-packages/aries_staticagent/static_connection.py:266: in handle
apts    |     await self._dispatcher.dispatch(msg, self)
apts    | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apts    | 
apts    | self = <aries_staticagent.dispatcher.Dispatcher object at 0x7f5b0d65a470>
apts    | msg = {'@type': 'did:sov:BzCbsNYhMrjHiqZDTUASHg;spec/connections/1.0/response', '@id': '97d4758c-ee6e-4091-96cf-df3049e0d17d...5kcG9pbnQiOiAiaHR0cDovL3Rlc3QtY2VudGVyLmxvY2FsaG9zdCJ9XX19', 'signer': 'EMimZ99MRykjP2ZqbqSZJg86AN5EYmzCv8qzKhzCzibz'}}
apts    | args = (<aries_staticagent.static_connection.StaticConnection object at 0x7f5b0d65a390>,)
apts    | kwargs = {}, handler = None
apts    | 
apts    |     async def dispatch(self, msg: Message, *args, **kwargs):
apts    |         """ Dispatch message to handler. """
apts    |         handler = self.select_handler(msg)
apts    |         if not handler:
apts    |             raise NoRegisteredHandlerException(
apts    |                 'No suitable handler for message of type {}'.format(
apts    | >                   msg.type
apts    |                 )
apts    |             )
apts    | E           aries_staticagent.dispatcher.NoRegisteredHandlerException: No suitable handler for message of type did:sov:BzCbsNYhMrjHiqZDTUASHg;spec/connections/1.0/response
apts    | 
apts    | /usr/local/lib/python3.6/dist-packages/aries_staticagent/dispatcher.py:107: NoRegisteredHandlerException
apts    | ---------------------------- Captured stdout setup -----------------------------
apts    | Setup: config: {'host': 'apts', 'port': 4000, 'endpoint': 'http://apts:4000', 'backchannel': 'aut.AUTBackchannel', 'provider': 'indy_provider.IndyProvider', 'ledger_name': 'indy.igrant', 'ledger_url': 'https://indy.igrant.io/genesis', 'ledger_apts_seed': '000000000000000000000000STEWARD1', 'tests': ['connections.*', 'issue-credential.*', 'present-proof.*'], 'subject': {'name': 'AGENT UNDER TEST', 'version': '1.0.0', 'endpoint': 'http://test-center.localhost'}, 'save_path': None}
apts    | =============================== warnings summary ===============================
apts    | ../usr/lib/python3/dist-packages/setuptools/depends.py:2
apts    |   /usr/lib/python3/dist-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
apts    |     import imp
apts    | 
apts    | -- Docs: https://docs.pytest.org/en/stable/warnings.html
apts    | 
apts    | =============================== Interop Profile ================================
apts    | 
apts    | {
apts    |   "@type": "Aries Test Suite Interop Profile v1",
apts    |   "suite_version": "0.1.0",
apts    |   "under_test_name": "AGENT UNDER TEST",
apts    |   "under_test_version": "1.0.0",
apts    |   "test_time": "2021-02-22T19:10:52",
apts    |   "results": [
apts    |     {
apts    |       "name": "connections,1.0,inviter,can-be-inviter",
apts    |       "description": "Test a connection as started by the agent under test.",
apts    |       "pass": false
apts    |     }
apts    |   ]
apts    | }
apts    | =========================== short test summary info ============================
apts    | FAILED protocol_tests/connection/test_connection.py::test_connection_started_by_tested_agent
apts    | !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
apts    | ================= 1 failed, 11 deselected, 1 warning in 0.26s ==================
apts exited with code 1

need tests for general DIDComm correctness

All of the following tests are needed:

Messages with a minor version > current, but major version == current, are handled.
Messages with a major version > current generate a graceful error.
Problem reports contain useful fields and correct codes.
Problem reports use localization correctly enough that machine translation could be applied.

need tests for Feature Discovery Protocol

All of the following tests are needed:

Agent can be a responder.
Agent returns supported roles, not just supported protocols.
Agent returns empty list when asked about nonexistent protocol.
Agent returns at least one item when asked about the protocol discovery protocol itself.
Agent supports wildcards.

`docker-compose build` instructon does not work

Followed the instructions in the read me file: https://github.com/hyperledger/aries-protocol-test-suite/blob/master/README.md

Running the docker-compose build results in the following error:

The following packages have unmet dependencies:
 indy-plenum : Depends: python3-orderedset (= 2.0) but 2.0.3 is to be installed
               Depends: python3-psutil (= 5.4.3) but 5.6.6 is to be installed
               Depends: python3-psutil (= 5.4.3) but 5.6.6 is to be installed
               Depends: python3-pympler (= 0.5) but 0.8 is to be installed
E: Unable to correct problems, you have held broken packages.
ERROR: Service 'ledger' failed to build : The command '/bin/sh -c apt-get update -y && apt-get install -y         python3-pyzmq=17.0.0         indy-plenum=${indy_plenum_ver}         indy-plenum=${indy_plenum_ver}         indy-node=${indy_node_ver}         python3-indy-crypto=${python3_indy_crypto_ver}         libindy-crypto=${indy_crypto_ver}         vim' returned a non-zero code: 100

Base64URL encoding/decoding with and without padding

There are differences between Python's urlsafe_b64encode/decode and what is generally rendered by other language libraries. It appears standard for padding to be omitted in most cases; however, Python includes padding and expects padding on decoding values, even though omitting padding is "loseless" and even appears to be the correct behavior for Base64URL encoding. This is subtle yet significant enough to cause agents that are otherwise well behaved to fail tests and cause issues in interoperability between agents written in python and agents written in other languages.

We can either select with or without padding and fail agents who do not support the selection or accept both as the test suite and implement tests to show that the agent under test will accept both as well.

docker-compose file is not present in version 0.1.0

docker-compose.yaml is not present in the released version /v0.1.0

need tests for Trust Ping Protocol

All of the folllowing tests are needed:

Agent can be pingee.
Agent can be pinger.
A trust ping that asks for no response elicits no response.
A trust ping that asks for a delayed response elicits a delayed response.

eliminate ordering

Generally, test suites avoid constructs that require ordering because they can lead to unreliable reports if the order is short-circuited. I think we need to remove all support for ordering (the priority feature) and replace it with reliable setup/fixtures.

hyperledger-archives / aries-protocol-test-suite Goto Github PK

aries-protocol-test-suite's People

Contributors

Stargazers

Watchers

Forkers

aries-protocol-test-suite's Issues

Stacktrace for reference

Recommend Projects

Recommend Topics

Recommend Org