Giter Club home page Giter Club logo

Comments (12)

criccomini avatar criccomini commented on May 22, 2024

Note had to install Python 3.4 on Ubuntu Precise via third party package because the Travis Precise image comes with Python 3.2, and that was giving me trouble with pip/virtualenv.

from confluent-kafka-python.

criccomini avatar criccomini commented on May 22, 2024

Also, 3.4 is what's used in the Travis CI in #40, where we first saw the segfaults. A proper way to replicate the exact environment would be to use Travis' CLI and build plugin to generate the real build.sh command that's run in the actual environment, but this seemed good enough to replicate the seg faults.

from confluent-kafka-python.

criccomini avatar criccomini commented on May 22, 2024

Here's the output of master with the above thread patch applied:

# py.test -v --timeout 20 --ignore=tmp-build --import-mode append -s
=========================================================================================================================================================================== test session starts ============================================================================================================================================================================
platform linux -- Python 3.4.5, pytest-3.0.4, py-1.4.31, pluggy-0.4.0 -- /test/bin/python3.4
cachedir: .cache
rootdir: /root/confluent-kafka-python, inifile: tox.ini
plugins: timeout-1.2.0
timeout: 20.0s method: signal
collected 19 items 

tests/test_Consumer.py::test_basic_api OK: poll() timeout
PASSED
tests/test_Consumer.py::test_on_commit SKIPPED
tests/test_Consumer.py::test_subclassing PASSED
tests/test_KafkaError.py::test_error_cb %3|1479486831.870|FAIL|rdkafka#producer-3| 127.0.0.1:1/bootstrap: Connect to ipv4#127.0.0.1:1 failed: Connection refused
error_cb KafkaError{code=_TRANSPORT,val=-195,str="127.0.0.1:1/bootstrap: Connect to ipv4#127.0.0.1:1 failed: Connection refused"}
error_cb KafkaError{code=_ALL_BROKERS_DOWN,val=-187,str="1/1 brokers are down"}
PASSED
tests/test_KafkaError.py::test_subclassing PASSED
tests/test_Producer.py::test_basic_api delivery <class 'str'>
PASSED
tests/test_Producer.py::test_subclassing PASSED
tests/test_TopicPartition.py::test_sort PASSED
tests/test_TopicPartition.py::test_cmp PASSED
tests/test_TopicPartition.py::test_hash PASSED
tests/test_TopicPartition.py::test_subclassing PASSED
tests/test_docs.py::test_verify_docs PASSED
tests/test_enums.py::test_enums -168
27
PASSED
tests/test_enums.py::test_tstype_enums PASSED
tests/test_misc.py::test_version Using confluent_kafka module version 0.9.2 (0x90100)
Using librdkafka version 0.9.1.2-128-g7018d2 (0x902ff)
PASSED
tests/test_misc.py::test_error_cb %3|1479486833.213|FAIL|rdkafka#consumer-7| localhost:65531/bootstrap: Connect to ipv4#127.0.0.1:65531 failed: Connection refused
%3|1479486834.226|FAIL|rdkafka#consumer-7| localhost:65531/bootstrap: Connect to ipv4#127.0.0.1:65531 failed: Connection refused
PASSED
tests/test_misc.py::test_stats_cb PASSED
tests/test_threads.py::test_thread_safety 1 Flush took 1.000
2 Flush took 1.001
1 Intentional callback crash: ok
1 Done
4 Flush took 2.002
2 Flush took 1.001
2 Done
3 Flush took 2.002
4 Intentional callback crash: ok
4 Done
3 Flush took 1.000
3 Done
Done
PASSED
tests/test_threads.py::test_foo Fatal Python error: Couldn't create autoTLSkey mapping
Segmentation fault

from confluent-kafka-python.

criccomini avatar criccomini commented on May 22, 2024

This discussion makes me pretty nervous. The producer callback seems fine, but I noticed some callbacks in the consumer as well. I haven't dug into this much, but I did notice that I have to comment out EVERY instance of Consumer() and Producer() in the tests in order to get the segfault to go away.

If I remove test_Consumer.py, test_Producer.py, and test_threads.py(), the tests on #40 pass. If I add just a single call to Consumer() or Producer() the tests segfault.

from confluent-kafka-python.

criccomini avatar criccomini commented on May 22, 2024

Attaching a diff off of master since #40 has had a force push applied to it that lost the checksum I mentioned. Not super relevant, since you can reproduce this off of master with the thread diff above, but just for historical purposes.

broken-pr-40.diff.zip

from confluent-kafka-python.

edenhill avatar edenhill commented on May 22, 2024

The provided test_threads.py patch doesn't really work (undefined variables, func parameter order error), can you provide an example that works(but segfaults)?

from confluent-kafka-python.

criccomini avatar criccomini commented on May 22, 2024

Issues with that part of the code are irrelevant. It seg faults before it even gets there.

I have double checked. Follow the instructions at the top of the issue, but just use master (instead of PR 40). Put the thread diff in /tmp/diff.diff and apply it (with bad params or whatever):

patch -p1 </tmp/diff.diff

Then run the tests using the command shown at the top of the isssue. It seg faults.

If you feel compelled to re-order the params, and fix the exit_flag variable, go for it. It doesn't matter.

from confluent-kafka-python.

edenhill avatar edenhill commented on May 22, 2024

The crash occurs when Producer or Consumer initprocs raise an exception

from confluent-kafka-python.

criccomini avatar criccomini commented on May 22, 2024

:(

Any idea why?

from confluent-kafka-python.

edenhill avatar edenhill commented on May 22, 2024

Yes sir, fix is on the way!

from confluent-kafka-python.

edenhill avatar edenhill commented on May 22, 2024

Thank you for the repro env and everything

from confluent-kafka-python.

criccomini avatar criccomini commented on May 22, 2024

Woot!

from confluent-kafka-python.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.