Comments (6)
ppar
commented on August 20, 2024
Test 1
Environment:
- Host: OS X 10.11
- Kafka: Docker container / Ubuntu Xenial
- Gollum: Vagrant/VirtualBox/Ubuntu Trusty
Result:
- Gollum seems to reconnect succesfully under this Ubuntu / Virtualbox / OS X / Docker setup
Host: Start docker container for kafka
docker run -it -p 9092:9092 -p 45505:45505 ubuntu
Container: Install and run kafka
apt-get update
apt-get -y install default-jre-headless curl
cd /opt
curl -LO http://mirror.softaculous.com/apache/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
tar xpzf kafka_2.11-0.10.2.0.tgz
cd kafka_2.11-0.10.2.0
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
bin/kafka-topics.sh --list --zookeeper localhost:2181
Host or Container: Run kafkacat
Run kafkacat to consume the "test" topic and dump it to stdout
kafkacat -C -t test -b localhost
Host: Create and start vagrant machine for gollum:
cat > Vagrantfile <<EOF
# -*- mode: ruby -*-
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/trusty64"
config.vm.network "private_network", ip: "192.168.66.10"
config.vm.network "private_network", ip: "192.168.77.10"
end
EOF
vagrant up
vagrant ssh default
Vagrant: Make sure you can log into it from the console:
sudo -i
passwd -d root
Vagrant box now has 3 interfaces with 2 explicit routes to host machine:
eth0: 10.0.2.15/24 => 10.0.2.2 (VirtualBox NAT)
eth1: 192.168.66.10/24 => 192.168.66.1 (vboxnet)
eth2: 192.168.77.10/24 => 192.168.77.1 (vboxnet)
Host machine has matching interfaces:
vboxnet1: inet 192.168.66.1 netmask 0xffffffa00 broadcast 192.168.66.255
vboxnet2: inet 192.168.77.1 netmask 0xffffff00 broadcast 192.168.77.255
Vagrant: Add one of the host machine's IPs to /etc/hosts
(1341d5e1a542 is the hostname kafka uses to refer to itself)
echo "192.168.77.1 kafka1341d5e1a542 1341d5e1a542" >> /etc/hosts
Vagrant: Create gollum config
cat > kafka_producer_test.conf <<EOF
"StdIn":
Type: "consumer.Console"
Streams: "stream01"
"KafkaOut01":
Type: "producer.Kafka"
Streams: "stream01"
#Compression: "zip"
Topics:
"stream01": "test"
Servers:
- "kafka1341d5e1a542:9092"
EOF
Vagrant: TEST
Start gollum (on virtualbox console, not over SSH)
./gollum -ll3 -c kafka_producer_test.conf
<type something to console, verify that kafkact displays it>
Check that TCP connection is established:
root@vagrant-ubuntu-trusty-64:~|0 # netstat -ntp
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 192.168.77.10:42592 192.168.77.1:9092 ESTABLISHED 3497/gollum
tcp 0 0 10.0.2.15:22 10.0.2.2:52433 ESTABLISHED 2303/sshd: vagrant
tcp 0 0 192.168.77.10:42591 192.168.77.1:9092 ESTABLISHED 3497/gollum
root@vagrant-ubuntu-trusty-64:~|0 #
root@vagrant-ubuntu-trusty-64:~|0 # ip route get 192.168.77.1
192.168.77.1 dev eth2 src 192.168.77.10
cache
root@vagrant-ubuntu-trusty-64:~|0 #
Remove default route to eliminate ambiguity:
root@vagrant-ubuntu-trusty-64:~|0 # ip route del default
Type something to gollum's stdin => Result: kafkacat shows output
Take down interface:
root@vagrant-ubuntu-trusty-64:~|0 # ifdown eth2
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether 08:00:27:bc:b6:c8 brd ff:ff:ff:ff:ff:ff
root@vagrant-ubuntu-trusty-64:~|0 #
root@vagrant-ubuntu-trusty-64:~|0 # ip route get 192.168.77.1
RTNETLINK answers: Network is unreachable
2|root@vagrant-ubuntu-trusty-64:~|0 #
Add route via the vboxnet NAT interface (note: eth1/192.168.66.10 wouldn't work here)
root@vagrant-ubuntu-trusty-64:~|0 # ip route add 192.168.77.1 via 10.0.2.2
root@vagrant-ubuntu-trusty-64:~|0 # ip route get 192.168.77.1
192.168.77.1 via 10.0.2.2 dev eth0 src 10.0.2.15
cache
Test connectivity to gollum:
root@vagrant-ubuntu-trusty-64:~|0 # telnet 192.168.77.1 9092
Trying 192.168.77.1...
Connected to 192.168.77.1.
Escape character is '^]'.
kljslfdkjsdf
Connection closed by foreign host.
Check established TCP connections:
root@vagrant-ubuntu-trusty-64:~|0 # netstat -tn
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 25 192.168.77.10:42592 192.168.77.1:9092 FIN_WAIT1
tcp 0 0 10.0.2.15:22 10.0.2.2:52433 ESTABLISHED
root@vagrant-ubuntu-trusty-64:~|0 #
Test connection by typing into gollum's stdin => Result: gollum reconnects, kafkacat shows output
Check established TCP connections:
root@vagrant-ubuntu-trusty-64:~|0 # netstat -tn
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 10.0.2.15:22 10.0.2.2:52433 ESTABLISHED
tcp 0 0 10.0.2.15:51470 192.168.77.1:9092 ESTABLISHED
tcp 0 0 10.0.2.15:51471 192.168.77.1:9092 ESTABLISHED
root@vagrant-ubuntu-trusty-64:~|0 #
Conclusion
Gollum seems to reconnect succesfully under this Ubuntu / Virtualbox / OS X / Docker setup
from gollum.
ppar
commented on August 20, 2024
Test 2
Same as Test 1, except:
Environment:
- Gollum: Vagrant / VirtualBox / FreeBSD 11.0
Result:
- FreeBSD 11.0 Vagrant/VirtualBox box is too unstable for testing
Part 1
Vagrantfile:
# -*- mode: ruby -*-
Vagrant.configure("2") do |config|
# config.vm.box = "ubuntu/trusty64"
config.vm.box = "freebsd/FreeBSD-11.0-CURRENT"
config.vm.base_mac = "000AAAAAAAAA"
config.vm.network "private_network", ip: "192.168.66.10"
config.vm.network "private_network", ip: "192.168.77.10"
end
vagrant up
# Takes a _long_ time due to broken DHCP
vagrant ssh default
su -
# vagrant
Since DHCP doesnt' work out of the box:
em0: inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
em1 inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
em2: inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
Add addresses:
root@:~ # ifconfig em1 192.168.77.10 netmask 255.255.255.0 up
root@:~ # ifconfig em2 192.168.77.10 netmask 255.255.255.0 up
root@:~ # netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 10.0.2.2 UGS em0
10.0.2.0/24 link#1 U em0
10.0.2.15 link#1 UHS lo0
127.0.0.1 link#4 UH lo0
192.168.66.0/24 link#2 U em1
192.168.66.10 link#2 UHS lo0
192.168.77.0/24 link#3 U em2
192.168.77.10 link#3 UHS lo0
Crash the FreeBSD box with no survivors:
root@:~ # telnet 192.168.66.1 9092
Trying 192.168.66.1...
Connected to 192.168.66.1.
Escape character is '^]'.
slakdjalksjd
Part 2
vagrant up
vagrant ssh default
su -
# vagrant
Bring up em1
root@:~ # ifconfig em1 192.168.66.10 up
# Defaults to netmask 255.255.255.0
root@:~ # ping 192.168.66.1
PING 192.168.66.1 (192.168.66.1): 56 data bytes
64 bytes from 192.168.66.1: icmp_seq=0 ttl=64 time=0.763 ms
64 bytes from 192.168.66.1: icmp_seq=1 ttl=64 time=0.489 ms
64 bytes from 192.168.66.1: icmp_seq=2 ttl=64 time=0.360 ms
^C
This time it doesn't crash:
root@:~ # telnet 192.168.66.1 9092
Trying 192.168.66.1...
Connected to 192.168.66.1.
Escape character is '^]'.
sdlkajldkjasdasd
Connection closed by foreign host.
root@:~ # telnet 192.168.66.1 9092
Trying 192.168.66.1...
Connected to 192.168.66.1.
Escape character is '^]'.
ldkjlasjdasd
Connection closed by foreign host.
root@:~ #
- ditto for em2
Routing:
root@:~ # netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 10.0.2.2 UGS em0
10.0.2.0/24 link#1 U em0
10.0.2.15 link#1 UHS lo0
127.0.0.1 link#4 UH lo0
192.168.66.0/24 link#2 U em1
192.168.66.10 link#2 UHS lo0
192.168.77.0/24 link#3 U em2
192.168.77.10 link#3 UHS lo0
- Soon after this, the FreeBSD box crashed at uptime 9m50s
from gollum.
arnecls
commented on August 20, 2024
The OS having the issues was FreeBSD 10.x (I think it was 10.1).
Back than I assumed a problem either inside the golang core or even in FreeBSD itself because sarama normally does reconnect and the change was somewhat VERY invasive.
from gollum.
ppar
commented on August 20, 2024
Yep, the reconnect code in producer/kafka.go is fairly straightforward, so if the implementation was in the gollum version at the time (the blames are 11-24 months old), the problem is/was very likely caused by the OS and/or Golang core.
from gollum.
ppar
commented on August 20, 2024
Test 3
Environment:
- Host: OS X 10.11
- Gollum: FreeBSD 10.1 / VirtualBox, traditional install, no vagrant
- Kafka: running on a separate server in the network (192.168.XX.BB/NM)
Gollum box: Network configuration
#1 == em0 -> bridged to Apple USB eth -> LAN, DHCP: 192.168.XX.AA/NM -> kafka reachable in same subnet
#2 == em1 -> VirtualBox NAT network -> NAT, DHCP: 10.0.3.15/24 -> kafka reachable via vbox NAT routing
#3 == em2 -> Host-only vboxnet1 -> static IP: 192.168.66.11/24 -> management interface
Result
- Gollum running under FreeBSD 10.1 seems to reconnect succesfully when juggling between two
emXinterfaces.
Test
Host: Consume topics with kafkacat
$ kafkacat -C -b 192.168.XX.BB -t test
Configure network:
# dhclient em1
# DHCPACK from 10.0.3.2
# ifconfig em2 192.168.66.11 netmask 255.255.255.0 up
Start gollum:
root@freebsd-10:~ # cat kafka_producer_test_freebsd.conf
"StdIn":
Type: "consumer.Console"
Streams: "stream01"
"KafkaOut01":
Type: "producer.Kafka"
Streams: "stream01"
#Compression: "zip"
Topics:
"stream01": "test"
Servers:
- "kafka-server-hostname:9092"
#"StdOut01":
# Type: "producer.Console"
# Streams: "stream01"
root@freebsd-10:~ #
root@freebsd-10:~ # ./gollum.freebsd -ll 3 -c kafka_producer_test_freebsd.conf
...
Feed text into gollum's stdin => output by kafkacat succesfully
Connection established via em0:
root@freebsd-10:~ # netstat -n
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 192.168.XX.AA.22535 192.168.XX.BB.9092 ESTABLISHED
tcp4 0 0 192.168.66.11.22 192.168.66.1.65005 ESTABLISHED
tcp4 0 0 192.168.66.11.22 192.168.66.1.64997 ESTABLISHED
Take interface down:
ifconfig em0 down
Type messages to gollum's stdin => predictably no response
Default route over em0 still persists, remove it
root@freebsd-10:~ # route del default
del net default
root@freebsd-10:~ #
Add default route via NAT gw:
root@freebsd-10:~ # route add default 10.0.3.2
add net default: gateway 10.0.3.2
root@freebsd-10:~ # netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 10.0.3.2 UGS em1
10.0.3.0/24 link#2 U em1
10.0.3.15 link#2 UHS lo0
127.0.0.1 link#4 UH lo0
192.168.XX.AA link#1 UHS lo0
192.168.66.0/24 link#3 U em2
192.168.66.11 link#3 UHS lo0
Old connections dying:
tcp4 0 30 192.168.XX.AA.44462 192.168.XX.BB.9092 FIN_WAIT_1
tcp4 0 30 192.168.XX.AA.22535 192.168.XX.BB.9092 FIN_WAIT_1
In the meanwhile, Gollum had discarded messages and does not re-establish connection automatically:
root@freebsd-10:~ # netstat -n
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 192.168.66.11.22 192.168.66.1.65005 ESTABLISHED
tcp4 0 0 192.168.66.11.22 192.168.66.1.64997 ESTABLISHED
Verify that kafka host is reachable again:
root@freebsd-10:~ # ping 192.168.XX.BB
PING 192.168.XX.BB (192.168.XX.BB): 56 data bytes
64 bytes from 192.168.XX.BB: icmp_seq=0 ttl=63 time=1.110 ms
64 bytes from 192.168.XX.BB: icmp_seq=1 ttl=63 time=1.096 ms
^C
--- 192.168.XX.BB ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.096/1.103/1.110/0.007 ms
Type something into gollum's stdin
=> connection is re-established
=> kafkacat displays messages correctly
=> intermediate messages were lost (due to timeout / retry settings?)
root@freebsd-10:~ # netstat -n
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 10.0.3.15.58766 192.168.XX.BB.9092 ESTABLISHED
tcp4 0 0 10.0.3.15.56722 192.168.XX.BB.9092 ESTABLISHED
tcp4 0 0 192.168.66.11.22 192.168.66.1.64598 ESTABLISHED
tcp4 0 0 192.168.66.11.22 192.168.66.1.64561 ESTABLISHED
A
Test repeated a couple of times with same results
Conclusion
Gollum running under FreeBSD 10.1 / VirtualBox seems to reconnect succesfully when juggling between two emX interfaces.
from gollum.
arnecls
commented on August 20, 2024
I guess we can close it as this is a) seems not to be reproducible anymore and is b) a too exotic thing to be problematic anyways. Thanks for the detailed analysis @ppar
from gollum.
Related Issues (20)
- Support new field types for config struct tags
- Test updated AWS Firehose and Kinesis producer HOT 3
- Update producer.Spooling to use components.BatchedWriterAssembly
- Update producer.StatsdMetrics to use core.BatchedProducer
- Update producer.socket to use components.BatchedWriterAssembly
- Race condition in core.PluginRegistry
- Gollum with Kafka and Fallback Producer crashes at startup if Kafka Broker is unreachable HOT 1
- 0.5.0 ERROR Failed to read config error=yaml: unmarshal errors: HOT 7
- Coloured output can't be disabled with the -lc flag HOT 1
- Feature request: Metrics in Prometheus format HOT 1
- multi-line messages HOT 8
- format.MetadataCopy: unexpected change in metadata if msg.data.payload get modified HOT 4
- Missing Root CA Certificates in Docker image breaks AwsS3 Producer HOT 4
- Feature request: Allow overriding Syslog RFC protocols HOT 3
- is documentation outdated? HOT 4
- Syslogd consumer should support setting file permissions
- Windows build contains a Linux ELF executable HOT 2
- Compatibility with future Go versions HOT 3
- Mention the gollum wiki project to avoid confusion HOT 1
- `github.com/CrowdStrike/go-metrics-prometheus` seems gone HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gollum.