The messages are not cleared from topic once consumed. about mockafka-py HOT 7 CLOSED

Hey @ssmails ,
I don't have any idea how it could happen . You mean you received messages more than once ?

could you please provide a unit test? Or a code snippet that show it is happening?

from mockafka-py.

@ssmails please look at this PR

I think you forgot to add clean=True to your admin client

in this test, I added parameterized test case that checks the number of produced messages with the number of consumed message

from mockafka-py.

@alm0ra thanks for looking into this. I copied the test sample you had and added a print msg as part of the test after the consumer runs, and am able to reproduce the issue I mentioned.
I reduced the dataset to 4 runs as well, so its more clear.

In run 2 below, you see the 2 messages are the same. They should be different since a uuid is used as the msg produced per https://github.com/alm0ra/mockafka-py/pull/62/files#diff-6d0ff70a7ffbc6670cd1e7d92364fa4ad5c3e155734668e5f668aee512b59e8aR62

test_consumer_consistency.py::test_consumer_consistency[2] PASSED [ 50%]
recv: {'u': '34635fbb-bdf3-44d3-baf0-e5fb4681fc53'}
recv: {'u': '34635fbb-bdf3-44d3-baf0-e5fb4681fc53'}

Similarly in test 3 and 4 below, we see that sometimes messages are duplicates, and sometimes unique.
If I run the test multiple times, everytime, the behaviour is different.

============================= test session starts ==============================
collecting ... collected 4 items

test_consumer_consistency.py::test_consumer_consistency[1] PASSED        [ 25%]
recv: {'u': '01d35b5b-b7da-4975-b91c-35f1d8bc8ff4'}

test_consumer_consistency.py::test_consumer_consistency[2] PASSED        [ 50%]
recv: {'u': '34635fbb-bdf3-44d3-baf0-e5fb4681fc53'}

recv: {'u': '34635fbb-bdf3-44d3-baf0-e5fb4681fc53'}

test_consumer_consistency.py::test_consumer_consistency[3] PASSED        [ 75%]
recv: {'u': '111f5a57-9815-4f16-b1d1-ab5e42af5fe9'}

recv: {'u': '111f5a57-9815-4f16-b1d1-ab5e42af5fe9'}

recv: {'u': 'd6b99d87-9636-4f76-a351-8c8b98abe45f'}

test_consumer_consistency.py::test_consumer_consistency[4] PASSED        [100%]
recv: {'u': 'db027dff-c5a5-4a85-986f-e7007e9d3efe'}

recv: {'u': '2919cdc3-5570-475a-a647-6b0327e011f7'}

recv: {'u': '2919cdc3-5570-475a-a647-6b0327e011f7'}

recv: {'u': '1afacb8e-bc2f-435f-89ce-46d5b568e1a6'}


============================== 4 passed in 0.05s ===============================

Process finished with exit code 0

from mockafka-py.

@ssmails
thanks for your raising the issue.

i found it by adding

    values_list = [item["u"] for item in messages]
    assert len(messages) == len(set(values_list))

to my test

from mockafka-py.

Dear @ssmails
I have identified the problem. As outlined in the pull request (PR), we neglected to implement single message commit functionality. Consequently, I refrained from committing anything when you passed this parameter. However, I have now included a new assertion to ensure that we do not consume a message more than once after committing. Once the PR is merged, the issue will be resolved. Should you encounter any further problems, please do not hesitate to inform me, and I will be glad to assist.

from mockafka-py.

Fixed in this release-v0.1.54

from mockafka-py.

Thanks @alm0ra , will check.

from mockafka-py.