yellow-camper / libevhtp Goto Github PK
View Code? Open in Web Editor NEWCreate extremely-fast and secure embedded HTTP servers with ease.
Home Page: https://criticalstack.com/
License: BSD 3-Clause "New" or "Revised" License
Create extremely-fast and secure embedded HTTP servers with ease.
Home Page: https://criticalstack.com/
License: BSD 3-Clause "New" or "Revised" License
I'm sorry I can't find it.
Now that the parser war is over, let's figure out the best plan of action to replace htparser with httparser.
#0 0x00000000008a8ee5 in event_active_nolock_ ()
#1 0x00000000008aeb49 in evmap_io_active_ ()
#2 0x00000000008b5388 in epoll_dispatch ()
#3 0x00000000008abd96 in event_base_loop ()
#4 0x00000000007eef9c in _evthr_loop ()
#5 0x0000003dfde079d1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003dfdae88fd in clone () from /lib64/libc.so.6
I suspect that it is the usleep trick to avoid thread concurrence problem.
I start 16 evhtp thread with evhtp_use_threads_wexit.
evhtp_bind_sockaddr()
does not close created socket if bind()
fails:
https://github.com/criticalstack/libevhtp/blob/develop/evhtp.c#L3668
if evhtp_accept_socket()
fails for some reason, socked is leaked as well:
https://github.com/criticalstack/libevhtp/blob/develop/evhtp.c#L3673
Build with SSL disabled
Execute test_basic - segfault.
With enables santizer next report generated
ASAN:DEADLYSIGNAL
==14636==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x7ff95654f191 bp 0x7ffcb1ccc600 sp 0x7ffcb1ccc5e0 T0)
#0 0x7ff95654f190 in evconnlistener_free (/lib64/libevent-2.0.so.5+0x1d190)
#1 0x4125a6 in evhtp_unbind_socket /home/vromanov/work/libevhtp/evhtp.c:3631
#2 0x404854 in main /home/vromanov/work/libevhtp/examples/test_basic.c:53
#3 0x7ff955967b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
#4 0x404568 (/home/vromanov/work/libevhtp/cmake-build-debug/examples/test_basic+0x404568)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib64/libevent-2.0.so.5+0x1d190) in evconnlistener_free
==14636==ABORTING
master branch, Centos 7.3
memory leak
code as flows:
void
reqHandler(evhtp_request_t * req, void * arg)
{
struct evbuffer * buffer = evbuffer_new();
char buf[4096] = { 0 };
int i;
evhtp_send_reply_chunk_start(req, EVHTP_RES_OK);
for (i = 0; i < 10000; i++)
{
evbuffer_add(buffer, buf, sizeof(buf));
evhtp_send_reply_chunk(req, buffer);
evbuffer_drain(buffer, -1);
}
evhtp_send_reply_chunk_end(req);
evbuffer_free(buffer);
}
int main(int argc,char **argv)
{
evbase_t * evbase;
evhtp_t * evhtp;
evbase = event_base_new();
assert(evbase);
evhtp = evhtp_new(evbase, NULL);
assert(evhtp);
evhtp_set_gencb(evhtp, reqHandler, NULL);
int ret = evhtp_use_threads(evhtp, NULL, 64, NULL);
ret = evhtp_bind_socket(evhtp, "0.0.0.0", 9010, 1024);
ret = event_base_loop(evbase, 0);
}
1.2.12
Sorry if this has already been addressed. I noticed https://github.com/criticalstack/libevhtp/blob/70849e7a310718cdc2ece4793673435179d9d452/refcount.h#L81 unlocks the mutex and frees the data, but I do not see pthread_mutex_destroy
anywhere. Is the mutex leaked?
When I see the number exit_cb, I'm confused. Because the function type named evthr_init_cb
https://github.com/criticalstack/libevhtp/blob/77f20e606e01666a8b121c2ef4c09022456cdf50/thread.c#L48
And I find the type define of callback function.
https://github.com/criticalstack/libevhtp/blob/0da6bc9d3124e225d67f5d678f1a2961987757c1/include/evhtp/thread.h#L36
bufferevent_socket_connect()
calling in the evhtp_connection_ssl_new()@evhtp.c
returns a unzero value, so evhtp_assert(rc == 0)
failed.evhtp_assert()
, abort()
func is called, so the program is exit.libevhtp is common http(1.1) adaptor for the programs which use it. and for some unknown reason, the program exit from libevhtp is not gracefullly.
evhtp_connection_ssl_new() {
...
rc = bufferevent_socket_connect(conn->bev,
(struct sockaddr *)&sin, sizeof(sin));
// the following code is workaround
if(rc !=0 ) {
// log error info
// free conn
// set conn to NULL
}
return conn;
}
the newest libevhtp
@NathanFrench
how about the above case, any suggestions is welcome, thank you in advance.
evhtp_set_gencb don't work. Instead first evhtp_set_cb callback called.
This is modified version of test_basic.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <errno.h>
#include <evhtp.h>
void
testcb(evhtp_request_t * req, void * a) {
const char * str = a;
evbuffer_add(req->buffer_out, str, strlen(str));
evhtp_send_reply(req, EVHTP_RES_OK);
}
void
issue161cb(evhtp_request_t * req, void * a) {
struct evbuffer * b = evbuffer_new();
if (evhtp_request_get_proto(req) == EVHTP_PROTO_10) {
evhtp_request_set_keepalive(req, 0);
}
evhtp_send_reply_start(req, EVHTP_RES_OK);
evbuffer_add(b, "foo", 3);
evhtp_send_reply_body(req, b);
evbuffer_add(b, "bar\n\n", 5);
evhtp_send_reply_body(req, b);
evhtp_send_reply_end(req);
evbuffer_free(b);
}
int
main(int argc, char ** argv) {
evbase_t * evbase = event_base_new();
evhtp_t * htp = evhtp_new(evbase, NULL);
evhtp_set_cb(htp, "/simple/", testcb, "simple");
evhtp_set_cb(htp, "/1/ping", testcb, "one");
evhtp_set_cb(htp, "/1/ping.json", testcb, "two");
evhtp_set_cb(htp, "/issue161", issue161cb, NULL);
// VVVVVVVVVVVVVVVVVVVVVVVVV
evhtp_set_gencb(htp, testcb, "root");
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#ifndef EVHTP_DISABLE_EVTHR
evhtp_use_threads_wexit(htp, NULL, NULL, 8, NULL);
#endif
evhtp_bind_socket(htp, "0.0.0.0", 8081, 2048);
event_base_loop(evbase, 0);
evhtp_unbind_socket(htp);
evhtp_free(htp);
event_base_free(evbase);
return 0;
}
Latest
There was this exchange in the issues of elizy's repo that I would always keep going back to, and which never formally made it into the documentation, perhaps it is worth adding to the documentation, now that those old issues are now gone?
I have a question : why need pause and resume pairs.
This is actually a good question.
Pause and resume will essentially stop and start attempting to read from the client socket. But also has some extra logic around it. Since the parser is a state machine, you can get functions executed as data is passed to it. But what if you wanted to temporarily stop that input processing? Well you have to save the state on the current input, then turn off the reading.
You can do all the normal things you can do while the request is paused, add output data, headers, etc. But it isn't until you use the resume function will these actions actually happen on the socket. When resume is called, it does an event_active(), which in turn goes back to the original reader function.
For example, with my companies reverse proxy we wrote (https://github.com/mandiant/RProxy) we don't immediately send the request to a backend, they get queued up and only processed once a backend connection becomes available. If it was not for the pause functionality, if a connection started posting a crap-ton of data, it would buffer and eat up all the memory.
This source comment actually does a good job of describing: https://github.com/mandiant/RProxy/blob/master/src/rproxy.c#L1020
Starting program: /work/github/libevhtp-1.2.12/build/examples/test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
*** Error in `/work/github/libevhtp-1.2.12/build/examples/test': double free or corruption (out): 0x00007fffffffd6a0 ***
Program received signal SIGABRT, Aborted.
0x00007ffff6da4cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: 没有那个文件或目录.
(gdb) bt
#0 0x00007ffff6da4cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff6da80d8 in __GI_abort () at abort.c:89
#2 0x00007ffff6de1394 in libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7ffff6eefb28 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff6ded66e in malloc_printerr (ptr=, str=0x7ffff6eefc58 "double free or corruption (out)", action=1) at malloc.c:4996
#4 int_free (av=, p=, have_lock=0) at malloc.c:3840
#5 0x0000000000406ce4 in htp__free (ptr=0x7fffffffd6a0) at /work/github/libevhtp-1.2.12/evhtp.c:173
#6 0x000000000040c1e3 in evhtp_kv_free (kv=0x6847b0) at /work/github/libevhtp-1.2.12/evhtp.c:2968
#7 0x000000000040c327 in evhtp_kvs_free (kvs=0x6835a0) at /work/github/libevhtp-1.2.12/evhtp.c:3007
#8 0x0000000000408611 in htp__request_free (request=0x6833c0) at /work/github/libevhtp-1.2.12/evhtp.c:1091
#9 0x000000000040aae6 in htp__connection_writecb (bev=0x682a90, arg=0x680890) at /work/github/libevhtp-1.2.12/evhtp.c:2201
#10 0x00007ffff7bad0df in bufferevent_writecb (fd=, event=, arg=0x682a90) at bufferevent_sock.c:297
#11 0x00007ffff7ba2a10 in event_persist_closure (ev=, base=0x67f040) at event.c:1319
#12 event_process_active_single_queue (activeq=0x67f480, base=0x67f040) at event.c:1363
#13 event_process_active (base=) at event.c:1438
#14 event_base_loop (base=0x67f040, flags=0) at event.c:1639
#15 0x0000000000406c3e in main (argc=1, argv=0x7fffffffe408) at /work/github/libevhtp-1.2.12/examples/test.c:663
(gdb)
examples/test
libevhtp-1.2.12
libevent-2.0.2
When I tried to run test_client example I got following assertion error
Assertion failed: c && c->htp && c->request && c->parser && bev (htp__connection_writecb_: libevhtp/evhtp.c:2165)
Seems the reason of this is that a client connection is every time created with evhtp_t equal to NULL, like on line 5173 in evhtp.c
if (!(conn = htp__connection_new_(NULL, -1, evhtp_type_client)))
{
return NULL;
}
And there is an assertion in evhtp.c on line 2165
evhtp_assert(c && c->htp && c->request && c->parser && bev);
So with client request c->htp is every time NULL and assertion fails.
Simply run test_client.c
Latest libevhtp version.
There should be a way to disable the use of scratch_buf within the evhtp_connection_s structure.
Hi,
After compile libevhtp with oniguruma package in ubuntu-14.04, call evhtp_set_regex_cb() function makes crash.
#include <stdio.h>
#include <stdlib.h>
#include <evhtp.h>
void testcb(evhtp_request_t *req, void *a)
{
const char* str;
printf("Fired testcb.\n");
str = a;
evbuffer_add_printf(req->buffer_out, "%s", str);
evhtp_send_reply(req, EVHTP_RES_OK);
}
int main(int argc, char** argv)
{
evbase_t *evbase;
evhtp_t *htp;
evbase = event_base_new();
htp = evhtp_new(evbase, NULL);
#ifndef EVHTP_DISABLE_REGEX
printf("NOT defined EVHTP_DISABLE_REGEX.\n");
evhtp_set_regex_cb(htp, "/simple1/*", testcb, "/simple/");
#endif
evhtp_bind_socket(htp, "0.0.0.0", 8081, 1024);
event_base_loop(evbase, 0);
evhtp_unbind_socket(htp);
evhtp_free(htp);
event_base_free(evbase);
return 0;
}
...
$ ./main &
[1] 23672
$ NOT defined EVHTP_DISABLE_REGEX.
$ curl -X GET localhost:8081/simple1/test
curl: (52) Empty reply from server
[1]+ Segmentation fault (core dumped) ./main
$ git branch -v
evhtp_hook_on_request_fini called twice on second and next requests. This
add printf("fini\n"); to test_fini function in test.c file. Compile. Run.
make first request to localhost/bar using Chrome. One line with fini will be printed. Make another request. Two lines with "fini" will be printed.
I do some investigation and found that this issue possibly related to keepalive. Looks like evhtp closes connection after receiving request and browser send request again.
master, Centos 7.3
In case when evhtp_header_val_add()
is called with val="static_string" and val_alloc is greater than 1 evhtp_header_t
will contain non-heap value that evhtp_kv_free()
will attempt to free.
This also applies to key and key_alloc parameters of evhtp_kv_new()
function.
Create htp_header_t instance with evhtp_kv_new
with static string as key and key_alloc > 1 than attepmpt to call evhtp_kv_free.
#include <evhtp.h>
int
main(int argc, char ** argv)
{
evhtp_header_t * header = evhtp_header_new("Connection", "Keep-Alive", 2, 2);
evhtp_kv_free(header);
return 0;
}
1.2.16
code as flows:
void reqHandler(evhtp_request_t *req,void *arg)
{
evhtp_send_reply_chunk_start(req, EVHTP_RES_OK);
struct evbuf_t * buffer = evbuffer_new();
assert(buffer);
char buf[4096];
int i;
for(i=0; i < 10000; i++)
{
evbuffer_add(buffer, buf, 4096);
evhtp_send_reply_chunk(req, buffer);
evbuffer_drain(buffer, -1);
}
evhtp_send_reply(req, EVHTP_RES_OK);
evbuffer_free(buffer);
}
int main(int argc,char **argv)
{
evbase_t * evbase;
evhtp_t * evhtp;
evbase = event_base_new();
assert(evbase);
evhtp = evhtp_new(evbase, NULL);
assert(evhtp);
evhtp_set_gencb(evhtp, reqHandler, NULL);
int ret = evhtp_use_threads(evhtp, NULL, 4, NULL);
ret = evhtp_bind_socket(evhtp, "0.0.0.0", 9010, 1024);
ret = event_base_loop(evbase, 0);
}
memory leak ,how to do ? who can help me,thanks
libevhtp-1.2.12-1
evhtp_request_pause will freeze current connections from sending data. I don't entirely understand how evhtp works, so this could be a problem in the seafile code and not evhtp, but I'm not positive.
In version 1.2.9, [1] changed to [2] and in every version after it. This caused the main issue that I am experiencing and have reverted the change as shown at [3]. My guess is that EV_WRITE isn't being re enabled when it should. It is very difficult to debug the seafile code though because there isn't any debugging code in it. If you have a suggestion of how to debug the library I am open to suggestions.
[4] [5] is the seafile code using this function, I'm fairly certain [4] is where the bug is occurring. Also want to mention that [6] is a patch I am using that the previous libevhtp developer suggested, which didn't fix the issue unfortunately.
[1] https://github.com/criticalstack/libevhtp/blob/1.2.9/evhtp.c#L1984
[2] https://github.com/criticalstack/libevhtp/blob/1.2.13/evhtp.c#L2865
[3] https://github.com/freebsd/freebsd-ports/blob/branches/2018Q1/www/libevhtp/files/patch-evhtp.c
[4] https://github.com/haiwen/seafile-server/blob/v6.2.3-server/server/access-file.c
[5] https://github.com/haiwen/seafile-server/blob/v6.2.3-server/server/upload-file.c
[6] https://github.com/freebsd/freebsd-ports/blob/branches/2018Q1/net-mgmt/seafile-server/files/patch-server_access-file.c
[7] https://reviews.freebsd.org/D13742
[8] curl -v "https://OBSCURED/seafhttp/files/c6629c8c-5324-OBSCURED-a9a46bf4aaa8/OBSCURED.rar"
GET /seafhttp/files/c6629c8c-5324-OBSCURED-a9a46bf4aaa8/OBSCURED.rar HTTP/2
Host: OBSCURED
User-Agent: curl/7.57.0
Accept: /
1.2.9 (works) --> 1.2.10 (Fails), 1.2.11 (Fails), 1.2.13 (Fails)
1.2.12 Also seems to be not work. Removing EV_WRITE on the line mentioned in [2] will cause everything to work again as it did in 1.2.9.
This happens when the callbacks object is NULL. Reason is evhtp_set_cb()
not being called beforehand.
I'm trying upgrade libevhtp on FreeBSD ports, but running into compilation errors that I can't quite figure out how to fix.
Sorry, forgot to provide full error:
upload-file.c:2373:23: error: incomplete definition of type 'struct evhtp_callback_s'
evhtp_set_hook(&cb->hooks, evhtp_hook_on_headers, upload_headers_cb, NULL);
~~^
/usr/local/include/evhtp/evhtp.h:38:8: note: forward declaration of 'struct evhtp_callback_s'
struct evhtp_callback_s;
^
[1] Is a full log of the build if interested.
All the evhtp_set_hook function calls return this error in the below Example code link.
[2] https://reviews.freebsd.org/D13742
1.2.11 (works) --> 1.2.15 (breaks)
Thanks for taking over this library. It definitely has many areas that need improvement but is promising. Using an out dated Oniguruma was driving me absolutely crazy! Thank you so much for moving this out of the source.
evhtp_hook_on_conn_error
is never used. A quick search of the repo only shows this hook being set here evhtp.c:4325
evhtp_hook_on_error
isn't used if only set in struct evhtp_connection_s
. It must be explicitly set on every request struct. evhtp.c:758
Use evhtp_hook_on_path
to set it on the evhtp_request_t
with evhtp_request_set_hook()
. However on_error will only be called if the connection get's to this callback. For Example
perl -e '$|=1; print "GET /file00 HTTP/1.1\r\n"; while(<>){}' | nc localhost 8080
$ ./server 8080
error_cb
on_conn_error
and on_error
nc localhost 8080
evhtp_res
conn_error_cb(evhtp_connection_t *conn, evhtp_error_flags type, void *arg)
{
printf("%s\n", __FUNCTION__);
return EVHTP_RES_OK;
}
evhtp_res
error_cb(evhtp_request_t *req, evhtp_error_flags type, void *arg)
{
printf("%s\n", __FUNCTION__);
return EVHTP_RES_OK;
}
evhtp_res
pre_accept_cb(evhtp_connection_t *conn, void *arg)
{
evhtp_connection_set_hook(conn, evhtp_hook_on_conn_error, &conn_error_cb, NULL);
evhtp_connection_set_hook(conn, evhtp_hook_on_error, &error_cb, NULL);
return EVHTP_RES_OK;
}
int
main(int argc, const char ** argv)
{
struct timeval = { .tv_sec = 1, .tv_usec = 0 };
...
evhtp_set_timeouts(htp, &to, &to);
evhtp_set_pre_accept_callback(htp, &pre_accept_cb, NULL);
...
}
1.2.16
[removed]
When libevhtp included in another project using add_subdirectory(libevhtp), CMAKE_SOURCE_DIR points to source directory of main project, not to source directory of libevhtp. To fix this problem replace with CMAKE_SOURCE_DIR with CMAKE_CURRENT_SOURCE_DIR
master and last tag
The following program requires a large amount of memory during running。
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2315 root 20 0 4176m 3.4g 1564 S 18.3 92.1 0:04.76 test-chunk
#include <evhtp.h>
void
reqHandler(evhtp_request_t * req, void * arg)
{
struct evbuffer * buffer = evbuffer_new();
char buf[4096] = { 0 };
int i;
evhtp_send_reply_chunk_start(req, EVHTP_RES_OK);
for (i = 0; i < 500000; i++)
{
evbuffer_add(buffer, buf, sizeof(buf));
evhtp_send_reply_chunk(req, buffer);
evbuffer_drain(buffer, -1);
}
printf("begin.....\n");
sleep(10);
evhtp_send_reply_chunk_end(req);
printf("end .....\n");
evbuffer_free(buffer);
}
int main(int argc,char **argv)
{
evbase_t * evbase;
evhtp_t * evhtp;
evbase = event_base_new();
assert(evbase);
evhtp = evhtp_new(evbase, NULL);
assert(evhtp);
evhtp_set_gencb(evhtp, reqHandler, NULL);
int ret = evhtp_use_threads(evhtp, NULL, 4, NULL);
ret = evhtp_bind_socket(evhtp, "0.0.0.0", 9010, 1024);
ret = event_base_loop(evbase, 0);
}
1.2.12-1
Is it possible to get the http return code from an evhtp_hook_request_fini_cb
? E.g., the second parameter I supply to evhtp_send_reply()
.
@flokli emailed me today informing me that the pre-packaged version of onig is vulnerable to CVE-2017-9224
We will be removing this and be making it a third-party dependency from now until forever.
UNTIL THEN: libevhtp will use the global version of onig if it is found on the system (thus not compiling the pre-packaged version). So.. DO THIS!
I initially tried to implement chunked encoding using libevent but turned to this library when that didn't work out. Unfortunately, the same issue seems to persist here. I've been trying to send large amounts of data over a single HTTP request, but the amount is too large for one evhtp_send_reply. So, I turned to chunking, which seems like it should have been the perfect solution--it ostensibly sends the content in pieces so that a single buffer won't become overfilled (which causes the crashing on my machine). And yet, it doesn't work at all.
I stepped through the code using GDB (libevhtp's own test_chunking test) and found that evhtp_send_reply_chunk merely pushes the overflowing buffer back a step. The content is progressively added to req->conn->bev->output (where req is the evhtp_request_t) with each chunk call. This buffer grows until evhtp_send_reply_chunk_end flushes its content into the network. Nothing is sent over the network until all the chunks have been loaded into this intermediary buffer, a confounding discovery that seems to eliminate all utility of this encoding method. (The bufferevent_flush call at the end of chunk seems to do nothing at all--digging down, it ultimately calls libevent's be_socket_flush function, which just returns 0 because of course.) I furthermore did a practical test in which I inserted sleep calls in between the chunk calls and, indeed, the server simply hanged until all the sleep and chunk calls had gone through. If chunking were working properly, pieces of the request should have been sent incrementally.
Am I missing something? It just doesn't seem like chunking is working at all like it's supposed to. Any help would be greatly appreciated.
Please, do not use - sign in version, it breaks rpm build.
Just use 1.12.1.1
Currently, libevhtp is using SSL_CTX_use_certificate_file
to load a certificate file. That function lacks the ability to load the pinned certificate chain (if any) which has a consequence of connecting clients not trusting the received certificate. By using SSL_CTX_use_certificate_chain_file
we give the libssl the ability to read and send the entire certificate chain (if any), which clients can check against.
I am profiling my application, which implements a REST API on top of libevhtp. And I'm finding htparser_run using most of the processing time. Here's an example gprof output:
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls ms/call ms/call name
17.87 0.52 0.52 htparser_run
6.53 0.71 0.19 fe_contact_api_read
4.30 0.84 0.13 api_v1_cmp_endpoint
3.78 0.95 0.11 8600729 0.00 0.00 evbuffer_add
2.75 1.03 0.08 evhtp_kv_new
2.41 1.10 0.07 htp__request_parse_header_key_
2.06 1.16 0.06 600124 0.00 0.00 evmap_io_add
1.72 1.21 0.05 10001571 0.00 0.00 evbuffer_invoke_callbacks
1.37 1.25 0.04 200013 0.00 0.00 rest_api_v1_cb
1.37 1.29 0.04 30994 0.00 0.01 event_process_active_single_queue
1.37 1.33 0.04 htp__create_reply_
1.37 1.37 0.04 htp__malloc_
1.03 1.40 0.03 2000130 0.00 0.00 evbuffer_expand_singlechain
1.03 1.43 0.03 415868 0.00 0.00 evmap_io_active
1.03 1.46 0.03 200349 0.00 0.00 _evbuffer_expand_fast
1.03 1.49 0.03 35 0.86 0.86 evmap_io_init
1.03 1.52 0.03 evhtp_kvs_free
1.03 1.55 0.03 fe_get_1st_phoneline_by_contact
1.03 1.58 0.03 htp__create_headers_
1.03 1.61 0.03 htp__free_
1.03 1.64 0.03 htp__protocol_
1.03 1.67 0.03 htp__request_new_
1.03 1.70 0.03 htp__request_parse_header_val_
1.03 1.73 0.03 htp__request_parse_path_
1.03 1.76 0.03 htp__should_parse_query_body_
0.69 1.78 0.02 5200338 0.00 0.00 client_matches
0.69 1.80 0.02 2000130 0.00 0.00 evbuffer_expand
0.69 1.82 0.02 1029358 0.00 0.00 event_queue_insert
0.69 1.84 0.02 1029351 0.00 0.00 event_queue_remove
0.69 1.86 0.02 606808 0.00 0.00 event_add_internal
0.69 1.88 0.02 600119 0.00 0.00 evmap_io_del
0.69 1.90 0.02 400441 0.00 0.00 evbuffer_unfreeze
0.69 1.92 0.02 200014 0.00 0.00 bufferevent_init_common
0.69 1.94 0.02 200013 0.00 0.00 COPY_CHAIN
0.69 1.96 0.02 contact_record_check
0.69 1.98 0.02 event_get_method
I haven't done a lot of profiling work (it's been years..). Now the request headers aren't too complicated, and it's not SSL (handled elsewhere).
My app basically does some lookups in hash tables, builds JSON and returns it. Granted, that's not much, but I'm surprised by htparser_run being at the top of the list.
Here's a sample request:
GET /v1/contact HTTP/1.0
Connection: Keep-Alive
accept: application/json
content-type: application/json
Authorization: Bearer ekCM83ry3hmevxGiahoXTVhS1hEXkK9pGnfKIT1iGb
Host: nvast2:5657
User-Agent: ApacheBench/2.3
BTW, I came here because I had the same problem with the original libevhtp, which I incorporated into my source tree 1.5 year ago.
Don't know how this applies here
Git clone as of today. Version says v1.2.12
use libevhtp as http server, if request from client is a short connection(include "Connection":"close" header), i got a coredump
at call htparser_get_error()
in the _evhtp_connection_readcb()
func.
for the default callback registration in the _evhtp_connection_accept()
func, when receive a short connection request, the action flow of server is:
_evhtp_connection_readcb()
func and get the request data.htparser_run()
to verify and parse the request data. in the htparser_run()
, after completed parse msg, it will callback the func(for example,the func name is: handle_cmd
) that registered by evhtp_set_cb()
.handle_cmd
func, after process the request body, in common we will write the response data(for example, call evhtp_send_reply()
func) in the end._evhtp_connection_writecb
that registered in the _evhtp_connection_accept()
will be triggered by the write action._evhtp_connection_writecb
func, if the request is a short connection, it will call evhtp_connection_free()
to free the connection.htparser_run
func, the connection variable in the _evhtp_connection_readcb
func is become invalid. so call htparser_get_error
func get a coredump.static int _evhtp_connection_accept(...) {
...
bufferevent_enable(connection->bev, EV_READ);
bufferevent_setcb(connection->bev,
_evhtp_connection_readcb,
//_evhtp_connection_writecb, <-- after write response back to the accepted connection, don't call _evhtp_connection_writecb()
NULL,
_evhtp_connection_eventcb, connection);
...
}
static void _evhtp_connection_readcb(...) {
...
evbuffer_drain(bufferevent_get_input(bev), nread);
if (c->request && c->request->status == EVHTP_RES_PAUSE) {
evhtp_request_pause(c->request);
} else if (htparser_get_error(c->parser) != htparse_error_none) {
evhtp_connection_free(c);
} else if (nread < avail) {
// we still have more data to read (piped request probably)
evhtp_connection_resume(c);
} else { <-- add this else to deal with the connection.
if(c->type == evhtp_type_server) {
/*
* if there is a set maximum number of keepalive requests configured, check
* to make sure we are not over it. If we have gone over the max we set the
* keepalive bit to 0, thus closing the connection.
*/
if (c->htp->max_keepalive_requests) {
if (++c->num_requests >= c->htp->max_keepalive_requests) {
c->request->keepalive = 0;
}
}
if (c->request->keepalive == 1) {
_evhtp_request_free(c->request);
c->keepalive = 1;
c->request = NULL;
c->body_bytes_read = 0;
if (c->htp->parent && c->vhost_via_sni == 0) {
/* this request was servied by a virtual host evhtp_t structure
* which was *NOT* found via SSL SNI lookup. In this case we want to
* reset our connections evhtp_t structure back to the original so
* that subsequent requests can have a different Host: header.
*/
evhtp_t * orig_htp = c->htp->parent;
c->htp = orig_htp;
}
htparser_init(c->parser, htp_type_request);
htparser_set_userdata(c->parser, c);
} else {
evhtp_connection_free(c);
}
}
}
...
}
i have merged above changes in my project, it works correct now.
above modification is suitable or not, can you give some suggestions. Thanks in advance.
In SSL mode (I haven't observed the segfault using plain http, at least) the call to htparser_run at https://github.com/criticalstack/libevhtp/blob/develop/evhtp.c#L2100 can potentially free the request and apparently also c->parser
, so that the call to htparser_get_error
in line 2139 leads to a segfault. The case that c->parser
is NULL after line 2100 should be handled.
very recent develop branch (commit 2307737 Nov 3)
Running make install after compiling places headers in /usr/local/evhtp/
dir (under ubuntu). Using those global headers causes compilation to fail because evhtp.h includes files assuming they are installed in /usr/local/
dir instead.
libevent
/* Default values for max_single_read & max_single_write variables. */
#define MAX_SINGLE_READ_DEFAULT 16384
#define MAX_SINGLE_WRITE_DEFAULT 16384
http://www.wangafu.net/~nickm/libevent-book/Ref6a_advanced_bufferevents.html Limiting maximum single read/write size
Interface
int bufferevent_set_max_single_read(struct bufferevent *bev, size_t size);
int bufferevent_set_max_single_write(struct bufferevent *bev, size_t size);ev_ssize_t bufferevent_get_max_single_read(struct bufferevent *bev);
ev_ssize_t bufferevent_get_max_single_write(struct bufferevent *bev);
normal
evbuffer_add_reference(req->buffer_out, payload, 16000, NULL, NULL);
wrk -c64 -t4 -d5 --latency 'http://127.0.0.1:8080/test'
Running 5s test @ http://127.0.0.1:8080/test
4 threads and 64 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 664.27us 825.07us 19.51ms 93.01%
Req/Sec 20.70k 2.09k 29.85k 82.50%
Latency Distribution
50% 504.00us
75% 761.00us
90% 1.24ms
99% 4.27ms
412239 requests in 5.01s, 6.29GB read
Requests/sec: 82278.98
Transfer/sec: 1.26GB
throughput degradation
evbuffer_add_reference(req->buffer_out, payload, 16385, NULL, NULL);
wrk -c64 -t4 -d5 --latency 'http://127.0.0.1:8080/test'
Running 5s test @ http://127.0.0.1:8080/test
4 threads and 64 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 43.79ms 4.21ms 50.22ms 94.95%
Req/Sec 363.64 50.71 484.00 79.50%
Latency Distribution
50% 43.92ms
75% 44.19ms
90% 45.13ms
99% 48.62ms
7248 requests in 5.01s, 114.26MB read
Requests/sec: 1447.03
Transfer/sec: 22.81MB
v1.2.11n
There is no real regulation other than take_ownership
that allows for a user to specify who owns what (e.g., who opens and closes a file descriptor).
See initial discussion: #6
Cascading the structures and adding a flag would be optimal:
struct socket {
int flags; (EVHTP_SOCK_OWNER)
int sock;
};
struct evhtp {
int flags; (EVHTP_FLAG_OWNER)
struct socket socket;
...
};
struct connection {
int flags; (EVHTP_CONN_FLAG_OWNER)
...
};
struct request {
int flags; (EVHTP_REQ_FLAG_OWNER)
...
};
As a https server, for better security consideration, user may not keep the private key(for example: named as tls.key) in a clear text. it may use some encryption algo to encrypt the tls.key for protecting from server inbreak.
so when before use tls.key
to create a https connection, user first decrypt the tls.key.
now, in the evhtp_ssl_init()
, cfg->privfile
is used directly. this means that user should use a unencrypted privfile. user can not use a encrypted privfile.
SSL_CTX_use_PrivateKey_file(htp->ssl_ctx,
cfg->privfile ? cfg->privfile : cfg->pemfile, SSL_FILETYPE_PEM);
i'd like add a member callback
(for example: decrypt_privfile_cb) in the evhtp_ssl_cfg_t
struct. if this callback is set, it will be called to decrypt the privfile and used by SSL_CTX_use_PrivateKey()
.
@NathanFrench if this feature is welcome, i can make a commit. Thanks in Advance.
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
LIB_DL
linked by target "test" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_basic" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_client" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_proxy" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_query" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_vhost" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
LIB_RT
linked by target "test" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_basic" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_client" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_proxy" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_query" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
linked by target "test_vhost" in directory /data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10
-- Configuring incomplete, errors occurred!
make[3]: *** [/data/richard/qsdk/build_dir/target-mips_r2_uClibc-0.9.33.2/libevhtp-1.2.10/.configured_] Error 1
make[3]: Leaving directory `/data/richard/qsdk/package/utils/libevhtp'
make[2]: *** [package/utils/libevhtp/compile] Error 2
make[2]: Leaving directory `/data/richard/qsdk'
make[1]: *** [/data/richard/qsdk/staging_dir/target-mips_r2_uClibc-0.9.33.2/stamp/.package_compile] Error 2
make[1]: Leaving directory `/data/richard/qsdk'
make: *** [world] Error 2
From looking at the code, HTTP keepalive should work automatically if supported by the client. It seems however that evhtp_connection_free
is called from libevent before a response can be sent.
Running in gdb
, c->request->flags
seems to include EVHTP_REQ_FLAG_KEEPALIVE
so I do not know what's wrong. I'm still searching.
Context: I'm working on fixing haiwen/seafile#1119
Please note below that wget
is trying to reuse the TCP connection and fails because of no data received.
$ ./examples/test_basic & $ LANG=C wget http://localhost:8081/simple/ http://localhost:8081/simple/ --2017-09-25 12:45:42-- http://localhost:8081/simple/ Resolving localhost (localhost)... ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:8081... failed: Connection refused. Connecting to localhost (localhost)|127.0.0.1|:8081... connected. HTTP request sent, awaiting response... 200 OK Length: 6 [text/plain] Saving to: 'index.html' index.html 100%[===================>] 6 --.-KB/s in 0s 2017-09-25 12:45:42 (779 KB/s) - 'index.html' saved [6/6] --2017-09-25 12:45:42-- http://localhost:8081/simple/ Reusing existing connection to localhost:8081. HTTP request sent, awaiting response... No data received. Retrying. --2017-09-25 12:45:43-- (try: 2) http://localhost:8081/simple/ Connecting to localhost (localhost)|127.0.0.1|:8081... connected. HTTP request sent, awaiting response... 200 OK Length: 6 [text/plain] Saving to: 'index.html' index.html.3 100%[===================>] 6 --.-KB/s in 0s 2017-09-25 12:45:43 (548 KB/s) - 'index.html' saved [6/6] FINISHED --2017-09-25 12:45:43-- Total wall clock time: 1.1s Downloaded: 2 files, 12 in 0s (644 KB/s)
1.2.12-1
The Array of string pointers errstr_map only have 13 item
But. The Enum type htpparse_error had defined 14 item
This code if (e > (htparse_error_generic + 1))
may cause value e
large than that array errstr_map
size.
const char * htparser_get_strerror(htparser * p)
{
htpparse_error e = htparser_get_error(p);
if (e > (htparse_error_generic + 1))
{
return "htparse_no_such_error";
}
return errstr_map[e];
}
latest
base
-> server
-> connection
-> request
if I comment out the assert, everything appears to work fine
I had an issue with regexec()
always returning mismatch on any regex I tried. Removing the oniguruma's regex lib from the build solved the issue.
For example, ab (apache benchmarking tool) hangs in example_pause:
[oleg@osboxes ~]$ build/examples/example_pause
[INFO] example_pause.c:108 response delayed for 10s: curl http://127.0.0.1:43129/
[oleg@osboxes ~]$ time curl http://127.0.0.1:43129/
time start 1522242044
time end 1522242054
real 0m10.016s
user 0m0.003s
sys 0m0.003s
[oleg@osboxes ~]$ time ab -n 1 http://127.0.0.1:43129/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient)...apr_pollset_poll: The timeout specified has expired (70007)
real 0m40.047s
user 0m0.002s
sys 0m0.004s
Latest version from develop branch.
It helps in maintaining distro packages
Thanks!
Although there had judged upper limit
https://github.com/criticalstack/libevhtp/blob/bc52552641ef9d01713c6d8e9b41aa1c089091fe/parser.c#L728
But, if p->buf_idx
arrival at (PARSER_STACK_MAX-1
) value, The following will using two characters of space, then have a out of bounds memory write.
https://github.com/criticalstack/libevhtp/blob/bc52552641ef9d01713c6d8e9b41aa1c089091fe/parser.c#L776-L777
Proposed using if (p->buf_idx >= PARSER_STACK_MAX-1)
replace.
htpparse_error
htparser_get_error(htparser * p) {
return p->error;
}
I have encountered this problem many times in this project,Not just this one.
Compile and start test_basic, start second instance of test_basic. Second instance will be ended with segfault in evhtp_unbind_socket. htp->server==NULL
master
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.