Comments (2)
bryancall
commented on July 20, 2024
Seeing something similar on docs:
Fatal: /home/bneradt/src/trafficserver_10/src/proxy/http/HttpSM.cc:2968: failed assertion `event == HTTP_TUNNEL_EVENT_DONE || event == VC_EVENT_INACTIVITY_TIMEOUT`
[Mar 22 14:21:12.346] traffic_crashlo NOTE: crashlog started, target=15587, debug=false syslog=true, uid=0 euid=0
[Mar 22 14:21:12.350] traffic_crashlo NOTE: logging to 0x615000003c80
[Mar 22 14:21:14.732] traffic_crashlo ERROR: wrote crash log to /opt/ats/var/log/trafficserver/crash-2024-03-22-142112.log
traffic_server: received signal 6 (Aborted)
traffic_server - STACK TRACE:
/opt/ats/bin/traffic_server(_Z20signal_crash_handleriP9siginfo_tPv+0x1c)[0x55cb6b15d02f]
/opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x180)[0x55cb6b0bc157]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fc04b15b420]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fc04af9800b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fc04af77859]
/opt/ats/bin/traffic_server(_Z11ink_warningPKcz+0x0)[0x55cb6b132e0a]
/opt/ats/bin/traffic_server(+0xbdac7e)[0x55cb6b12cc7e]
/opt/ats/bin/traffic_server(_ZN6HttpSM14tunnel_handlerEiPv+0x2bc)[0x55cb6b222de6]
/opt/ats/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x51c)[0x55cb6b21f11c]
/opt/ats/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x1e3)[0x55cb6b0ca6eb]
/opt/ats/bin/traffic_server(+0x13c6724)[0x55cb6b918724]
/opt/ats/bin/traffic_server(+0x13c6d70)[0x55cb6b918d70]
/opt/ats/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0x1231)[0x55cb6b91b2c9]
/opt/ats/bin/traffic_server(_Z12write_to_netP10NetHandlerP18UnixNetVConnectionP7EThread+0x67)[0x55cb6b91a095]
/opt/ats/bin/traffic_server(_ZN18UnixNetVConnection12net_write_ioEP10NetHandlerP7EThread+0x2f)[0x55cb6b91e805]
/opt/ats/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x477)[0x55cb6b9a97d3]
/opt/ats/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x4a2)[0x55cb6b9a9f06]
/opt/ats/bin/traffic_server(_ZN7EThread15execute_regularEv+0x93c)[0x55cb6ba409b6]
/opt/ats/bin/traffic_server(_ZN7EThread7executeEv+0x30c)[0x55cb6ba40efc]
/opt/ats/bin/traffic_server(+0x14ebd12)[0x55cb6ba3dd12]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7fc04b14f609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fc04b074133]
[Mar 22 14:21:20.115] traffic_server NOTE: records parsing completed.
[Mar 22 14:21:20.115] traffic_server NOTE: /opt/ats/etc/trafficserver/records.yaml finished loading
Traffic Server 10.0.0-0 Mar 19 2024 15:31:13 ip-72-167-35-147.ip.secureserver.net
traffic_server: using root directory '/opt/ats'
from trafficserver.
bneradt
commented on July 20, 2024
I reproduced this locally by running the ramp test in a loop with ATS run via gdb:
(gdb) bt
#0 __pthread_kill_implementation (threadid=281474795599200, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x0000fffff6f522f8 [PAC] in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2 0x0000fffff6f05800 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x0000fffff6ef0288 [PAC] in __GI_abort () at abort.c:79
#4 0x00000000008beae0 [PAC] in ink_abort (message_format=0xd1a908 "%s:%d: failed assertion `%s`") at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/tscore/ink_error.cc:99
#5 0x00000000008bac50 in _ink_assert (expression=0xd35a30 "event == HTTP_TUNNEL_EVENT_DONE || event == VC_EVENT_INACTIVITY_TIMEOUT", file=0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", line=2986)
at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/tscore/ink_assert.cc:35
#6 0x0000000000920edc in HttpSM::tunnel_handler (this=0xffffc466f710, event=103, data=0xffffc42605b0) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc:2986
#7 0x000000000091f350 in HttpSM::main_handler (this=0xffffc466f710, event=103, data=0xffffc42605b0) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc:2622
#8 0x000000000088a7ac in Continuation::handleEvent (this=0xffffc466f710, event=103, data=0xffffc42605b0) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/include/iocore/eventsystem/Continuation.h:228
#9 0x0000000000c4aea4 in write_signal_and_update (event=103, vc=0xffffc42602e0) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:121
#10 0x0000000000c4b430 in write_signal_done (event=103, nh=0xfffff5e1cba0, vc=0xffffc42602e0) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:166
#11 0x0000000000c4c834 in write_to_net_io (nh=0xfffff5e1cba0, vc=0xffffc42602e0, thread=0xfffff5e1c010) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:522
#12 0x0000000000c4bfa4 in write_to_net (nh=0xfffff5e1cba0, vc=0xffffc42602e0, thread=0xfffff5e1c010) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:356
#13 0x0000000000c4df9c in UnixNetVConnection::net_write_io (this=0xffffc42602e0, nh=0xfffff5e1cba0, lthread=0xfffff5e1c010) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:862
#14 0x0000000000c7b7b8 in NetHandler::process_ready_list (this=0xfffff5e1cba0) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/NetHandler.cc:286
#15 0x0000000000c7bab4 in NetHandler::waitForActivity (this=0xfffff5e1cba0, timeout=4965250) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/NetHandler.cc:364
#16 0x0000000000cbf164 in EThread::execute_regular (this=0xfffff5e1c010) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/eventsystem/UnixEThread.cc:300
#17 0x0000000000cbf300 in EThread::execute (this=0xfffff5e1c010) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/eventsystem/UnixEThread.cc:350
#18 0x0000000000cbd9dc in spawn_thread_internal (a=0x11deca0) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/eventsystem/Thread.cc:68
#19 0x0000fffff6f50584 in start_thread (arg=0xfffff7fb9760) at pthread_create.c:444
#20 0x0000fffff6fbfd4c [PAC] in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone3.S:76
(gdb) f 6
#6 0x0000000000921014 in HttpSM::tunnel_handler (this=0xffffc53434b0, event=103, data=0xffffc5866260) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc:2989
2989 ink_assert(event == HTTP_TUNNEL_EVENT_DONE || event == VC_EVENT_INACTIVITY_TIMEOUT);
(gdb) p event
$4 = 103
(gdb) p (int)TS_EVENT_VCONN_WRITE_COMPLETE
$5 = 103Thus the crash happens while processing an unexpected WRITE_COMPLETE event.
Checking the history value of the HttpSM shows this:
(gdb) p history
$1 = {history = {
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd34450 "state_api_callback", line = 1292}, event = 60000, reentrancy = 2},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd34488 "state_api_callout", line = 1332}, event = 60000, reentrancy = 2},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd33a20 "state_read_client_request_header", line = 523}, event = 100, reentrancy = 3},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd39128 "set_next_state", line = 7933}, event = 34463, reentrancy = 3},
{location = {file = 0xd50b60 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpCacheSM.cc", func = 0xd50bf8 "state_cache_open_read", line = 106}, event = 1104, reentrancy = -31073},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd36c58 "do_cache_lookup_and_read", line = 4954}, event = 22808, reentrancy = 3},
{location = {file = 0xd50b60 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpCacheSM.cc", func = 0xd50bf8 "state_cache_open_read", line = 106}, event = 1103, reentrancy = -31073},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd35348 "state_cache_open_read", line = 2534}, event = 1103, reentrancy = 1},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd39128 "set_next_state", line = 7893}, event = 34463, reentrancy = 1},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd39128 "set_next_state", line = 8047}, event = 34463, reentrancy = 1},
{location = {file = 0xd50b60 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpCacheSM.cc", func = 0xd50d80 "state_cache_open_write", line = 188}, event = 1108, reentrancy = -31073},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd351f0 "state_cache_open_write", line = 2413}, event = 1108, reentrancy = 2},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd39128 "set_next_state", line = 7942}, event = 34463, reentrancy = 2},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd34a38 "state_read_server_response_header", line = 1873}, event = 104, reentrancy = 1},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd38478 "setup_internal_transfer", line = 6908}, event = 34463, reentrancy = 1},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd35a08 "tunnel_handler", line = 2984}, event = 103, reentrancy = 1},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd38708 "setup_server_transfer", line = 7143}, event = 34463, reentrancy = 1},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd35af8 "tunnel_handler_server", line = 3038}, event = 2302, reentrancy = 1},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd35d30 "tunnel_handler_ua", line = 3409}, event = 103, reentrancy = 0},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd35a08 "tunnel_handler", line = 2984}, event = 2301, reentrancy = 1},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd34450 "state_api_callback", line = 1292}, event = 60000, reentrancy = 2},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd34488 "state_api_callout", line = 1332}, event = 60000, reentrancy = 2},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd38b10 "kill_this", line = 7520}, event = 34463, reentrancy = 0},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd38b10 "kill_this", line = 7520}, event = 34463, reentrancy = 0},
{location = {file = 0xd33838 "/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc", func = 0xd38b10 "kill_this", line = 7520}, event = 34463, reentrancy = 0},
{location = {file = 0xefbeaddeefbeadde <error: Cannot access memory at address 0xefbeaddeefbeadde>, func = 0xefbeaddeefbeadde <error: Cannot access memory at address 0xefbeaddeefbeadde>, line = -272716322}, event = 44510, reentrancy = -4162} <repeats 40 times>},
history_pos = 16}Note that the final events are kill_this. We don't expect WRITE_COMPLETE events anymore after kill_this is called. Apparently. The this HttpSM, while killed, is not yet deleted, so this is not technically a use after free yet:
(gdb) f 6
#6 0x0000000000921014 in HttpSM::tunnel_handler (this=0xffffc53434b0, event=103, data=0xffffc5866260) at /home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc:2989
2989 ink_assert(event == HTTP_TUNNEL_EVENT_DONE || event == VC_EVENT_INACTIVITY_TIMEOUT);
(gdb) p magic
$11 = 65261
(gdb) p (int)HTTP_SM_MAGIC_ALIVE
$12 = 65261
(gdb) p kill_this_async_done
$13 = falsefrom trafficserver.
Related Issues (20)
- Is this missed to pass argument in `plugins/multiplexer/dispatch.cc`? HOT 3
- Is this missed to forward argument in `Http2CommonSession::add_url_to_pushed_table`? HOT 1
- Possibly forgotten parameter of `Matcher::test_reg` in `plugins/header_rewrite /matcher.h`
- Is there interest to get rid of `-Wno-unused-parameter` warning suppression? HOT 1
- Logs don't display URL properly for plugins
- AddressSanitizer: heap-buffer-overflow on address in XpackDynamicTableStorage::write HOT 1
- Add AuTest to exercise http2.incomplete_header_timeout_in
- Missed to pass `length` parameter in the tscpp api - `HeaderField::append` HOT 1
- Status mismatch in header_rewrite HOT 2
- Autest for static hit plugin fails on release build
- Autest tls_tunnel fails because of a slightly different error message from python HOT 3
- What is the current status of http3 support in ATS 10.x? HOT 1
- Intermittent CI failures of ja3_fingerprint
- Build without hwloc broken on master
- Missing git tag for 9.2.4 HOT 1
- HTTP/3 benchmark results HOT 5
- Records parses uninitialized data for IP addresses
- ATS will not build with VERIFY_JTEST_DATA defined
- Some debug stats are counted under the wrong metrics
- Outgoing HTTPS connections fail with OpenSSL 3.2+ HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from trafficserver.