Comments (12)
Note had to install Python 3.4 on Ubuntu Precise via third party package because the Travis Precise image comes with Python 3.2, and that was giving me trouble with pip/virtualenv.
from confluent-kafka-python.
Also, 3.4 is what's used in the Travis CI in #40, where we first saw the segfaults. A proper way to replicate the exact environment would be to use Travis' CLI and build plugin to generate the real build.sh command that's run in the actual environment, but this seemed good enough to replicate the seg faults.
from confluent-kafka-python.
Here's the output of master
with the above thread patch applied:
# py.test -v --timeout 20 --ignore=tmp-build --import-mode append -s
=========================================================================================================================================================================== test session starts ============================================================================================================================================================================
platform linux -- Python 3.4.5, pytest-3.0.4, py-1.4.31, pluggy-0.4.0 -- /test/bin/python3.4
cachedir: .cache
rootdir: /root/confluent-kafka-python, inifile: tox.ini
plugins: timeout-1.2.0
timeout: 20.0s method: signal
collected 19 items
tests/test_Consumer.py::test_basic_api OK: poll() timeout
PASSED
tests/test_Consumer.py::test_on_commit SKIPPED
tests/test_Consumer.py::test_subclassing PASSED
tests/test_KafkaError.py::test_error_cb %3|1479486831.870|FAIL|rdkafka#producer-3| 127.0.0.1:1/bootstrap: Connect to ipv4#127.0.0.1:1 failed: Connection refused
error_cb KafkaError{code=_TRANSPORT,val=-195,str="127.0.0.1:1/bootstrap: Connect to ipv4#127.0.0.1:1 failed: Connection refused"}
error_cb KafkaError{code=_ALL_BROKERS_DOWN,val=-187,str="1/1 brokers are down"}
PASSED
tests/test_KafkaError.py::test_subclassing PASSED
tests/test_Producer.py::test_basic_api delivery <class 'str'>
PASSED
tests/test_Producer.py::test_subclassing PASSED
tests/test_TopicPartition.py::test_sort PASSED
tests/test_TopicPartition.py::test_cmp PASSED
tests/test_TopicPartition.py::test_hash PASSED
tests/test_TopicPartition.py::test_subclassing PASSED
tests/test_docs.py::test_verify_docs PASSED
tests/test_enums.py::test_enums -168
27
PASSED
tests/test_enums.py::test_tstype_enums PASSED
tests/test_misc.py::test_version Using confluent_kafka module version 0.9.2 (0x90100)
Using librdkafka version 0.9.1.2-128-g7018d2 (0x902ff)
PASSED
tests/test_misc.py::test_error_cb %3|1479486833.213|FAIL|rdkafka#consumer-7| localhost:65531/bootstrap: Connect to ipv4#127.0.0.1:65531 failed: Connection refused
%3|1479486834.226|FAIL|rdkafka#consumer-7| localhost:65531/bootstrap: Connect to ipv4#127.0.0.1:65531 failed: Connection refused
PASSED
tests/test_misc.py::test_stats_cb PASSED
tests/test_threads.py::test_thread_safety 1 Flush took 1.000
2 Flush took 1.001
1 Intentional callback crash: ok
1 Done
4 Flush took 2.002
2 Flush took 1.001
2 Done
3 Flush took 2.002
4 Intentional callback crash: ok
4 Done
3 Flush took 1.000
3 Done
Done
PASSED
tests/test_threads.py::test_foo Fatal Python error: Couldn't create autoTLSkey mapping
Segmentation fault
from confluent-kafka-python.
This discussion makes me pretty nervous. The producer callback seems fine, but I noticed some callbacks in the consumer as well. I haven't dug into this much, but I did notice that I have to comment out EVERY instance of Consumer()
and Producer()
in the tests in order to get the segfault to go away.
If I remove test_Consumer.py, test_Producer.py, and test_threads.py(), the tests on #40 pass. If I add just a single call to Consumer()
or Producer()
the tests segfault.
from confluent-kafka-python.
Attaching a diff off of master since #40 has had a force push applied to it that lost the checksum I mentioned. Not super relevant, since you can reproduce this off of master with the thread diff above, but just for historical purposes.
from confluent-kafka-python.
The provided test_threads.py patch doesn't really work (undefined variables, func parameter order error), can you provide an example that works(but segfaults)?
from confluent-kafka-python.
Issues with that part of the code are irrelevant. It seg faults before it even gets there.
I have double checked. Follow the instructions at the top of the issue, but just use master (instead of PR 40). Put the thread diff in /tmp/diff.diff and apply it (with bad params or whatever):
patch -p1 </tmp/diff.diff
Then run the tests using the command shown at the top of the isssue. It seg faults.
If you feel compelled to re-order the params, and fix the exit_flag variable, go for it. It doesn't matter.
from confluent-kafka-python.
The crash occurs when Producer or Consumer initprocs raise an exception
from confluent-kafka-python.
:(
Any idea why?
from confluent-kafka-python.
Yes sir, fix is on the way!
from confluent-kafka-python.
Thank you for the repro env and everything
from confluent-kafka-python.
Woot!
from confluent-kafka-python.
Related Issues (20)
- AdminClient doesn't support specifying the logger HOT 7
- from confluent_kafka.avro import AvroProducer class Deprecated. what does it mean ? HOT 1
- SSL cert problem "self-signed certificate" and ignored SSL options HOT 5
- Transactional Produce Increments Offset by 2 HOT 1
- Segmentation fault happens when trying to commit a message after the max.poll.interval.ms passed without polling HOT 1
- Improper handling of Python C-API exceptions
- Kafka producer does not report _ISR_INSUFF as error code when ISR count is less; instead reports _MSG_TIMED_OUT
- PyPI package is badly built
- AdminClient is stuck at `CONNECT` when using `OAUTHBEARER` with `SASL_SSL` protocol HOT 5
- V2.3.0 not support python2.7.8 HOT 1
- String representation of a TopicPartition raises a SystemError on Windows with Python 3.12.1 and Confluent Kafka Python 2.3.0
- OverflowError: Python int too large to convert to C int when using confluent_kafka Avro deserializer HOT 2
- Kafka returning same set of records even after full records fetched from the table
- SASL Authentication issue HOT 4
- Missing log messages when closing the program HOT 1
- future is done but cannot return result() in delete_topics HOT 3
- confluent-kafka-python vs aiokafka for Asynchronous applications HOT 4
- Install failed when using 32bit python3.12 on Windows
- Not support validating fields with the `format: email` in JSON Schema.
- Initializing producer throws segfault
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from confluent-kafka-python.