session-replay-tools / cetus Goto Github PK

Cetus is a high performance middleware that provides transparent routing between your application and any backend MySQL Servers.

License: GNU General Public License v2.0

CMake 1.05% Pawn 0.01% Lex 0.41% Yacc 2.62% C 60.02% Shell 10.08% Python 5.08% Perl 11.41% PLpgSQL 0.09% Makefile 0.89% M4 0.06% C++ 0.59% Roff 7.68%

mysql xa sharding distributed performance mysql-proxy database scaling middleware horizontal-scalable

cetus's People

Contributors

Stargazers

Watchers

Forkers

zondahuman yejr edogawakudo hulaoan linecode skyformat99 staysun thm1201 jions7ihj zhanglei oraclexbw lazio579 eagle518 b-xiang cuiopen heidsoft hashzhang greentor china-jiajia ezail3 wangtenfee deanwei lyw007 leonliu315 libo-bob archoncap callmedba shiliuxing sansom hugoren suker413 liuflylove666 mojianyuan yuanfeng0905 hongren149 burstzp shiftwinting duzhanyuan xubingyue trestea 307545758 mengxingpek a58457432 fanyeren wyxiao muyl acesigh ziyege tonyzhimin ppayaoxing chenld penghaige newlysoft lxdzy zhoudaqing beaver-company supersnailaixiang davexanderxu lukw00heck bobzhanggmail zeus911 fnet123 huihuiyule-www feixueck sod442995033 zhiwuya 2892931976 sdhzlzhk davidmr001 swinsey carycui yiitz windzzp cuijinquan hadoop835 bluestn viyond shuyunqi znavy louishust zhuyu19911016 baijinping sandy27494131 gongchengcheng errychen sharethings no2key csy512889371 yanmengjun azkoss digideskio dudu0614 wangjianbao tom2jack yanghongkjxy xlogin netkey rainleo comk88 showsmall

cetus's Issues

MySQL 8.0，group by 字段不再隐式排序，需要在proxy端补加order by

改进响应处理过程，使其性能达到最优

原始mysql proxy响应处理过程效率低下，无论代码怎么优化，都不如maxscale，需要彻底改变响应处理流程，使其性能超越maxscale（目前maxscale性能最好）

安全过滤

当用户有如下需求时，应该提供一个选项，供用户配置
1、比如阻断没有where条件的update，delete语句
2、全表扫描
3、类似拖库sql等

增加对Cetus是否处于维护状态的查询

目前已经存在设置维护状态的命令：
set maintain [true|false]
增加新的指令：
show maintain status;

cetus change db或者change user的时候，如果报错，应该采用更加友好的方式反馈给用户

目前是静默处理，不友好

自动根据mysql版本，选择采用reset conn 还是change user来复位连接

当客户端关闭连接的时候，如果连接不能正常放入连接池，需要对连接复位，在5.7版本之前，只支持change user的重开销复位，5.7版本以后，支持了轻量级reset conn复位连接

解决python框架访问admin接口发送set autocommit=0的问题

由于admin，默认设置了CLIENT_TRANSACTIONS，导致有些框架，比如python框架，发送了set autocommit=0语句，而这个语句，admin接口不支持

rang分片规则下，当分片KEY大于2^32 值时，会出现找不到正确分片的情况

show variables like '%autocommit%';是否需要cetus捕获处理，而非发往MySQL

session:
set autocommit = 0;
show variables like '%autocommit%'; # 返回的值有问题

show global varaibles like '%autocommit%';语句可以发往后端

Improve packet locality using SO_ATTACH_REUSEPORT_CBPF or SO_ATTACH_REUSEPORT_EBPF

内核4.6以后支持SO_ATTACH_REUSEPORT_EBPF或者SO_ATTACH_REUSEPORT_CBPF。
利用这个来提升cetus多进程版本性能，预期提升性能30～50%

select sum(field1)/10000 结果和 sum(field1/10000) 结果不一样

如题，在使用过程中，发现select sum(field1)/10000 结果和 sum(field1/10000) 结果不一样

利用event_base_update_cache_time提升libevent cached time准确度

event_base_update_cache_time
EVENT2_EXPORT_SYMBOL int event_base_update_cache_time ( struct event_base * base )

Update cached_tv in the 'base' to the current time.
You can use this function is useful for selectively increasing the accuracy of the cached time value in 'base' during callbacks that take a long time to execute.
This function has no effect if the base is currently not in its event loop, or if timeval caching is disabled via EVENT_BASE_FLAG_NO_CACHE_TIME.

子查询sql语句的field错误导致条件被忽略

例如：
`select sum(amount) from cashout where bet_id in (
select bet_id from bet where order_id in (

)
);
`
当cashout 包含bet_id但是bet 不包含bet_id时，语句检查通过，所有的cashout记录都被算上

admin端口查询命令会存在隐患

复现方法：

updaet user_pwd set password='abc' where user='cetus_app';
+--------------------------+---------------------+
| Status | Value |
+--------------------------+---------------------+
| Cetus version | v0.8.8-18-gc9e5ba1 |
| Startup time | 2018-05-11 07:42:36 |
| Loaded modules | proxy admin |
| Idle backend connections | 100 |
| Used backend connections | 0 |
| Client connections | 3 |
| Query count | 0 |
| QPS (1min, 5min, 15min) | 0.00, 0.00, 0.00 |
| TPS (1min, 5min, 15min) | 0.00, 0.00, 0.00 |
+--------------------------+---------------------+
目前匹配的方法是在用户输入的SQL语句中查询目标字串来进行匹配的，该匹配方法有隐患，如上例。

请问支持 MariaDB么

请问这套中间件支持 MariaDB 么..~~

无法使用load data命令

load` data concurrent local infile 'C:\\Users\\hqjfliuping\\Desktop\\data.csv' ignore into table `hqchiperp`.`tabblename` fields escaped by '\\' terminated by ',' optionally enclosed by '"' lines terminated by '\n' (`somefileds`)

ERROR 1105 (07000): (proxy) unable to retrieve command

参考proxysql，在admin接口实现kill命令

需要处理好不同进程之间产生thread id的问题，不能重复，否则admin去处理的时候不知道杀哪一个.

同时admin接口支持show connectionlist where pid = xxxx

(select emp_no,dept_no from employees_mix.dept_manager limit 2 ) union all (select emp_no,dept_no from employees_mix.dept_emp limit 2)

解析器还不支持这种语法，需要解决

save settings时当proxy.conf.old文件不存在时，应忽略该错误

1 admin_save_settings函数中增加：调用系统函数rename/remove/chmod等的debug日志，方便用户排查权限等引起的问题
2 save settings时当*.conf.old文件不存在时，应忽略该错误，不需要上报给用户
3 select * from help 中 save settings 行去掉“not implement”

select conn_num from backends where backend_ndx=? and user=? 有些情况显示不正确

复现方法：

配置文件配置
*.conf文件配置
default-username=ght
users.json文件配置
配置多个用户，例如：ght和 tmp
2 启动Cetus，登录admin命令查看
mysql> select conn_num from backends where backend_ndx=1 and user='ght';
+----------------+
| connection_num |
+----------------+
| 100 |
+----------------+
1 row in set (0.00 sec)
mysql> select conn_num from backends where backend_ndx=1 and user='tmp';
+----------------+
| connection_num |
+----------------+
| 100 |
+----------------+
1 row in set (0.00 sec)
tmp没有连接，但是显示的却是100，是默认连接池的连接数。显示不准确。

参考proxysql，实现一致性读

http://www.proxysql.com/blog/proxysql-gtid-causal-reads

libmysql-chassis-proxy.so: undefined reference to `g_list_free_full'

[root@mysql8 build]# make install
[ 3%] Built target mysql-chassis-glibext
[ 6%] Built target mysql-chassis-timing
[ 30%] Built target mysql-chassis
[ 33%] Built target lemon
[ 51%] Built target sqlparser
[ 90%] Built target mysql-chassis-proxy
[ 91%] Linking C executable cetus
libmysql-chassis-proxy.so: undefined reference to g_list_free_full' collect2: ld returned 1 exit status make[2]: *** [src/cetus] Error 1 make[1]: *** [src/CMakeFiles/cetus.dir/all] Error 2 执行make install报错libmysql-chassis-proxy.so: undefined reference to g_list_free_full'，请问大神们，我这个是缺包还是啥？查了半天没啥结果

请教一下读写分离版本读强一致性的实现思路

多进程实现思路

利用内核3.9+的SO_REUSEPORT机制，多个work进程共同监听一个端口，而多进程如果需要通信，则可以参考nginx socket pair来通信，确保多进程之间没有锁访问

admin用单独的进程来做，可以从共享内存里面获取相关信息

Compile error

clone git repo
$ git clone https://github.com/Lede-Inc/cetus ....

make build directory
$ mkdir build && cd build

$ cmake ../ -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/home/user/cetus_install -DSIMPLE_PARSER=OFF
-- The C compiler identification is GNU 7.3.1
-- Check for working C compiler: /bin/cc
-- Check for working C compiler: /bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PkgConfig: /bin/pkg-config (found version "1.3.12") 
-- building from v1.0.0-64-gc2b2757
-- libevent at libs: incl:
-- gthread  at libs: incl:
-- gmodule  at libs: incl:
-- libevent at libs: incl:
-- glib2.0  at libs: incl:
-- gthread  at libs: incl:
-- gmodule  at libs: incl:
-- Checking for module 'glib-2.0>=2.28'
--   Found glib-2.0, version 2.54.3
-- Checking for module 'gmodule-2.0>=2.28'
--   Found gmodule-2.0, version 2.54.3
-- Checking for module 'gthread-2.0>=2.28'
--   Found gthread-2.0, version 2.54.3
-- checking in dirs: /usr/include/mysql;/usr/include/mysql/mysql;/usr/include/glib-2.0;/usr/lib64/glib-2.0/include;/usr/include/glib-2.0;/usr/lib64/glib-2.0/include
-- Looking for include files sys/types.h, event.h
-- Looking for include files sys/types.h, event.h - found
-- Looking for include file inttypes.h
-- Looking for include file inttypes.h - found
-- Looking for include file netinet/in.h
-- Looking for include file netinet/in.h - found
-- Looking for include file net/if.h
-- Looking for include file net/if.h - found
-- Looking for include file net/if_dl.h
-- Looking for include file net/if_dl.h - not found
-- Looking for include file stddef.h
-- Looking for include file stddef.h - found
-- Looking for include file stdint.h
-- Looking for include file stdint.h - found
-- Looking for include file stdlib.h
-- Looking for include file stdlib.h - found
-- Looking for include file signal.h
-- Looking for include file signal.h - found
-- Looking for include file syslog.h
-- Looking for include file syslog.h - found
-- Looking for include file sys/filio.h
-- Looking for include file sys/filio.h - not found
-- Looking for include file sys/ioctl.h
-- Looking for include file sys/ioctl.h - found
-- Looking for include file sys/param.h
-- Looking for include file sys/param.h - found
-- Looking for include file sys/resource.h
-- Looking for include file sys/resource.h - found
-- Looking for include file sys/socket.h
-- Looking for include file sys/socket.h - found
-- Looking for include file sys/sockio.h
-- Looking for include file sys/sockio.h - not found
-- Looking for include file sys/time.h
-- Looking for include file sys/time.h - found
-- Looking for include file sys/types.h
-- Looking for include file sys/types.h - found
-- Looking for include file sys/uio.h
-- Looking for include file sys/uio.h - found
-- Looking for include file sys/un.h
-- Looking for include file sys/un.h - found
-- Looking for include file time.h
-- Looking for include file time.h - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for include file mysql.h
-- Looking for include file mysql.h - found
-- Looking for include file glib.h
-- Looking for include file glib.h - found
-- Looking for include file zlib.h
-- Looking for include file zlib.h - found
-- Looking for include file glib/gthread.h
-- Looking for include file glib/gthread.h - not found
-- Looking for include file pwd.h
-- Looking for include file pwd.h - found
-- Looking for inet_ntop
-- Looking for inet_ntop - found
-- Looking for getcwd
-- Looking for getcwd - found
-- Looking for signal
-- Looking for signal - found
-- Looking for strerror
-- Looking for strerror - found
-- Looking for srandom
-- Looking for srandom - found
-- Looking for writev
-- Looking for writev - found
-- Looking for sigaction
-- Looking for sigaction - found
-- Looking for getaddrinfo
-- Looking for getaddrinfo - found
-- Looking for g_thread_init in gthread-2.0
-- Looking for g_thread_init in gthread-2.0 - found
-- Looking for malloc in tcmalloc
-- Looking for malloc in tcmalloc - not found
-- OpenSSL version: 1.1.0h
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11") 
-- ZLIB version: 1.2.11
-- Check size of ulong
-- Check size of ulong - done
-- Check size of socklen_t
-- Check size of socklen_t - done
-- Check size of rlim_t
-- Check size of rlim_t - done
-- CMAKE_REQUIRED_LIBRARIES is /usr/lib64/libevent.so
-- Looking for event_base_new
-- Looking for event_base_new - found
-- Looking for event_base_free
-- Looking for event_base_free - found
-- Found FLEX: /bin/flex (found version "2.6.1") 
** Using full parser [-DSIMPLE_PARSER=OFF(default)]
-- Configuring done
-- Generating done
-- Build files have been written to: /src/github.com/Lede-Inc/cetus/build

$ make install
Scanning dependencies of target lemon
[  1%] Building C object lib/CMakeFiles/lemon.dir/__/tools/lemon.c.o
[  2%] Linking C executable lemon
@@@ USING FULL PARSER [-DSIMPLE_PARSER=OFF]
[  2%] Built target lemon
[  4%] Generating myparser.y.c
Parser statistics:
  terminal symbols...................   186
  non-terminal symbols...............    96
  total symbols......................   282
  rules..............................   300
  states.............................   359
  conflicts..........................     0
  action table entries...............  2720
  total table size (bytes)........... 13058
[  5%] [FLEX][MyLexer] Building scanner with flex 2.6.1
Scanning dependencies of target sqlparser
[  7%] Building C object lib/CMakeFiles/sqlparser.dir/sql-expression.c.o
[  8%] Building C object lib/CMakeFiles/sqlparser.dir/sql-operation.c.o
[  9%] Building C object lib/CMakeFiles/sqlparser.dir/sql-property.c.o
[ 11%] Building C object lib/CMakeFiles/sqlparser.dir/sql-context.c.o
[ 12%] Building C object lib/CMakeFiles/sqlparser.dir/sql-construction.c.o
[ 14%] Building C object lib/CMakeFiles/sqlparser.dir/sql-filter-variables.c.o
[ 15%] Building C object lib/CMakeFiles/sqlparser.dir/mylexer.l.c.o
[ 16%] Building C object lib/CMakeFiles/sqlparser.dir/myparser.y.c.o
[ 18%] Linking C shared library libsqlparser.so
[ 18%] Built target sqlparser
Scanning dependencies of target mysql-chassis-glibext
[ 19%] Building C object src/CMakeFiles/mysql-chassis-glibext.dir/glib-ext.c.o
[ 21%] Linking C shared library libmysql-chassis-glibext.so
[ 21%] Built target mysql-chassis-glibext
Scanning dependencies of target mysql-chassis-timing
[ 22%] Building C object src/CMakeFiles/mysql-chassis-timing.dir/chassis-timings.c.o
[ 23%] Linking C shared library libmysql-chassis-timing.so
[ 23%] Built target mysql-chassis-timing
Scanning dependencies of target mysql-chassis
[ 25%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-plugin.c.o
[ 26%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-event.c.o
[ 28%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-log.c.o
[ 29%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-mainloop.c.o
[ 30%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-shutdown-hooks.c.o
[ 32%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-keyfile.c.o
[ 33%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-path.c.o
[ 35%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-filemode.c.o
[ 36%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-limits.c.o
[ 38%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-frontend.c.o
[ 39%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-options.c.o
[ 40%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-options-utils.c.o
[ 42%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-unix-daemon.c.o
[ 43%] Building C object src/CMakeFiles/mysql-chassis.dir/chassis-config.c.o
[ 45%] Building C object src/CMakeFiles/mysql-chassis.dir/cJSON.c.o
[ 46%] Linking C shared library libmysql-chassis.so
[ 46%] Built target mysql-chassis
Scanning dependencies of target mysql-chassis-proxy
[ 47%] Building C object src/CMakeFiles/mysql-chassis-proxy.dir/network-mysqld.c.o
In file included from /src/github.com/Lede-Inc/cetus/src/network-mysqld.c:61:0:
/src/github.com/Lede-Inc/cetus/src/network-mysqld.c: In function ‘proxy_self_create_auth’:
/src/github.com/Lede-Inc/cetus/src/network-mysqld-packet.h:105:32: error: ‘CLIENT_BASIC_FLAGS’ undeclared (first use in this function); did you mean ‘CLIENT_DEFAULT_FLAGS’?
 #define COMPATIBLE_BASIC_FLAGS CLIENT_BASIC_FLAGS
                                ^
/src/github.com/Lede-Inc/cetus/src/network-mysqld-packet.h:108:30: note: in expansion of macro ‘COMPATIBLE_BASIC_FLAGS’
 #define CETUS_DEFAULT_FLAGS (COMPATIBLE_BASIC_FLAGS                     \
                              ^~~~~~~~~~~~~~~~~~~~~~
/src/github.com/Lede-Inc/cetus/src/network-mysqld.c:4436:33: note: in expansion of macro ‘CETUS_DEFAULT_FLAGS’
     auth->client_capabilities = CETUS_DEFAULT_FLAGS;
                                 ^~~~~~~~~~~~~~~~~~~
/src/github.com/Lede-Inc/cetus/src/network-mysqld-packet.h:105:32: note: each undeclared identifier is reported only once for each function it appears in
 #define COMPATIBLE_BASIC_FLAGS CLIENT_BASIC_FLAGS
                                ^
/src/github.com/Lede-Inc/cetus/src/network-mysqld-packet.h:108:30: note: in expansion of macro ‘COMPATIBLE_BASIC_FLAGS’
 #define CETUS_DEFAULT_FLAGS (COMPATIBLE_BASIC_FLAGS                     \
                              ^~~~~~~~~~~~~~~~~~~~~~
/src/github.com/Lede-Inc/cetus/src/network-mysqld.c:4436:33: note: in expansion of macro ‘CETUS_DEFAULT_FLAGS’
     auth->client_capabilities = CETUS_DEFAULT_FLAGS;
                                 ^~~~~~~~~~~~~~~~~~~
make[2]: *** [src/CMakeFiles/mysql-chassis-proxy.dir/build.make:63: src/CMakeFiles/mysql-chassis-proxy.dir/network-mysqld.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:993: src/CMakeFiles/mysql-chassis-proxy.dir/all] Error 2
make: *** [Makefile:163: all] Error 2

system info

$ cat /etc/redhat-release 
Fedora release 27 (Twenty Seven)

$ uname -a
Linux localhost.localdomain 4.16.16-200.fc27.x86_64 #1 SMP Sun Jun 17 03:06:00 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --enable-libmpx --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)

cetus支持全量日志

config set key=字符串值; 时如果字符串加了单引号或双引号，key的值会变成空，且不报错

config set key=字符串；字符串不应该加额外的单引号或是双引号。如果用户增加了单引号或双引号，给用户返回错误，不修改值。

mysql> config set default-username = "ght";
Query OK, 1 row affected (0.00 sec)

多实例启动的时候，判断端口已经被前一个实例占用

安装启动后报错 (critical) g_string_assign: assertion `rval != NULL' failed 2018-09-11 19:53:43: (critical) last message repeated 101 times 2018-09-11 19:53:43: (critical) g_string_assign: assertion `rval != NULL' failed 2018-09-11 19:53:51: (critical) last message repeated 101 times 2018-09-11 19:53:51: (critical) g_string_assign: assertion `rval != NULL' failed 2018-09-11 19:53:59: (critical) last message repeated 101 times

遇到一个问题一直报错，但是服务能用，不确实是否隐患，感觉像是lib库冲突，但是不清楚情况
希望团队帮确认下，要不用的揪心，感谢
#CentOS release 6.2 (Final)
#mysql Ver 14.14 Distrib 5.7.23, for Linux (x86_64) using EditLine wrapper
#2.6.32-220.7.1.el6.2.0.0.9.x86_64 #1 SMP Fri Jul 24 17:07:23 CST 2015 x86_64 x86_64 x86_64 GNU/Linux
2018-09-11 06:39:07: (message) starting cetus 1.0.0
2018-09-11 06:39:07: (message) glib version: 2.28.8
2018-09-11 06:39:07: (message) libevent version: 2.0.21-stable
2018-09-11 06:39:07: (message) monitor thread started
2018-09-11 06:39:07: (message) starting cetus 1.0.0
2018-09-11 06:39:07: (message) glib version: 2.28.8
2018-09-11 06:39:07: (message) libevent version: 2.0.21-stable
2018-09-11 06:39:07: (message) config dir: /usr/local/cetus/conf
2018-09-11 06:39:07: (message) read 2 users
2018-09-11 06:39:07: (message) src/cetus-1.0.0/src/chassis-unix-daemon.c:127: [angel] we try
to keep PID=39071 alive, nprocs:0
2018-09-11 06:39:07: (message) plugin proxy 1.0.0 started
2018-09-11 06:39:07: (message) plugin admin 1.0.0 started
2018-09-11 06:39:07: (message) set default pool size:1000
2018-09-11 06:39:07: (message) set max pool size:2000
2018-09-11 06:39:07: (message) set max resp len:10485760
2018-09-11 06:39:07: (message) set max alive time:7200
2018-09-11 06:39:07: (message) src/cetus-1.0.0/src/mysql-proxy-cli.c:657:set merged output si
ze:8192
2018-09-11 06:39:07: (message) src/cetus-1.0.0/src/mysql-proxy-cli.c:660:set max header size:
65536
2018-09-11 06:39:07: (message) set client_found_rows false
2018-09-11 06:39:07: (message) src/cetus-1.0.0/src/mysql-proxy-cli.c:674:xa_log_detailed fals
e
2018-09-11 06:39:07: (message) monitor thread started
2018-09-11 06:39:07: (message) sql thread is not start
2018-09-11 06:39:07: (message) check_slave monitor open.
2018-09-11 06:39:07: (message) proxy listening on port 0.0.0.0:6001, con:0x61adb0
2018-09-11 19:53:51: (critical) g_string_assign: assertion rval != NULL' failed 2018-09-11 19:53:59: (critical) last message repeated 101 times 2018-09-11 19:53:59: (critical) g_string_assign: assertion rval != NULL' failed
2018-09-11 19:54:07: (critical) last message repeated 101 times
2018-09-11 19:54:07: (critical) g_string_assign: assertion `rval != NULL' failed
2018-09-11 19:54:16: (critical) last message repeated 101 times

backend 删除后开始状态为delete，后自动转为down

backend 删除后开始状态为delete，后自动转为down，日志不停抛出后端服务异常的日志，正常情况下都已经主动delete，程序上是否该判定这个server不做检测呢？

连接独占且不切换后端连接的场景下，可以使用各种set指令

需要支持这种类型的应用

cetus多进程版本平滑升级方法

多进程版本（设置worker-processes=xx，其中xx是进程个数）平滑升级分为两种情况：
方法1
cetus前面有类似lvs负载均衡器的情况
跟单进程用法一样

方法2
应用直接连接cetus（不用类似lvs负载均衡器，一般可以采用dns来做高可用）
cetus支持类似nignx的平滑升级，步骤如下：
a）启动新的cetus，同时改pid文件名称
假设pid文件在路径/xx/yy/下面，名称为cetus.pid，执行如下命令会把旧的cetus.pid改名为cetus.pid.oldbin
kill -USR2 `cat /xx/yy/cetus.pid`

需要注意的是，访问admin需要用长连接，否则由于新旧cetus都在监听同一个端口，可能访问新的cetus admin，也有可能访问旧的cetus admin

b）旧cetus停止监听新的连接，并且设置旧cetus为set maintain true，释放空闲连接
kill -WINCH `cat /xx/yy/cetus.pid.oldbin`

以后会在log文件定期输出处于maintain true状态的cetus的客户端连接情况

c）关闭旧cetus
kill -QUIT `cat /xx/yy/cetus.pid.oldbin`