finaldie / final_libs Goto Github PK

View Code? Open in Web Editor NEW

13.0 5.0 9.0 858 KB

Basic Libraries for Linux

License: MIT License

Python 1.60% Makefile 0.73% C 97.11% Shell 0.09% Pawn 0.46%

final_libs's Introduction

Common Libraries (Linux Platform)

Lastest Release

See Here

ChangeLog

See change log

Library contains

Lib Name	Description
flist	Lockfree FIFO single-linked list in one-producer one-consumer scenario
fdlist	Double-linked list
fhash	Hash table
flock	A wraper, which is safer and easier to use pthread condition
flog	A High Performance Logging Library
fmbuf	A light-weight ring-buffer
fconf	A simple configuration file library
ftime	Easy to create system timer
fthread_pool	Simple thread pool, which is easy to use
fnet	Wrap the system APIs, easy to use the networking api
fev	Event framework, including buffer, networking, timer service...
fcache	A simple cache with LRU
fco	C coroutine. Notes: Legacy library, use it carefully
~~fut~~	C Unit test framework. Notes: Migrated to fcunit
~~fmempool~~	Thread-Cache memory pool. Notes: Migrated to skull-malloc
~~fpcap~~	Pcap file conversion library. Notes: Migrated to fpcap

API Documents

See Wiki

Compile

git clone [email protected]:finaldie/final_libs.git flibs
cd flibs
git submodule update --init --recursive
make
make check
make valgrind-check

Benchmark

make bench
make bench-run

Flags

Change the compiler, such as using clang:

make CC=clang

Build shared library instead of static library:

make SHARED=true

Build debug version without any optimization parameters

make debug=true

Build 32bit libraries under 64bit platform

make BIT=32

Build in parallel

make -j4   # Adjust the number '4' according to the real cpu cores to speed up
the compile time

Skip building legacy library Currently, fco is defined as a legacy library, maybe it won't working well or lose support in some archs, use a specify macro to skip building it.

make FLIB_CFLAGS=-DFLIB_SKIP_LEGACY

Custom CFLAGS and LDFLAGS Sometimes, we want to define some different macros/compilation flags to control compiling/linking results, in flibs, instead of standard CFLAGS and LDFLAGS, we can use FLIB_CFLAGS and FLIB_LDFLAGS

Example (on 64bit platform)

Build 64bit static-link libraries

make -j4 && make -j4 check

Build 64bit dynamic-link libraries

make SHARED=true -j4 && make SHARED=true check

Install flibs to system After compilation, just call install target, to install flibs into system:

make install

notes: By default, the flibs will be installed in /usr/local/:

Headers in: /usr/local/include
Libraries in: /usr/local/lib If we want to change the location, maybe to /usr/, just run:

make prefix=/usr/ install

Use flibs in a Project

After installing flibs into system, basically we need few steps to use it:

Include the headers from the source file
Link the statis/dynamic library from the Makefile

Let's see an example Source file and Makefile, e.g. fhash:

// main.c
#include <stdlib.h>
#include <stdio.h>
#include <flibs/fhash.h>

int main(int argc, char** argv)
{
    // 1. Create string hash table
    fhash* tbl = fhash_str_create(0, FHASH_MASK_AUTO_REHASH);

    // 2. Set a key-value into table
    const char* key = "hello";
    fhash_str_set(tbl, key, "world");

    // 3. Get the value from table
    const char* value = fhash_str_get(tbl, key);
    printf("Key: %s, value: %s\n", key, value);

    // 4. Destroy the hash table
    fhash_str_delete(tbl);
    return 0;
}

# Makefile
all:
	gcc -Wall -g -O2 -o demo main.c -lflibs

Then Build and Run it:

final@ubuntu1404: ~/code/github/flibs/demo>make
gcc -Wall -g -O2 -o demo main.c -lflibs
final@ubuntu1404: ~/code/github/flibs/demo>./demo
Key: hello, value: world

Have fun :)

final_libs's People

Contributors

Stargazers

Watchers

Forkers

yutouhewo teethfairyyan wangguoxi zjxylc amyvmiwei lvshunxia noor-ho-tabbe zousenming ericma2014

final_libs's Issues

[FNET] Support writev

Support writev which can improve the multi-buffers sending performance.

[FLOG] Dead loop when no space on disk

Found a issue, we can see dead loop in fetcher thread when there is no space on disk, need to deep investigating it.

[FPCAP] Hash rules is not very suitable for tcp 4 elements

The current implementation is hash the source ip & source port, but this is depend on the dst ip and dst port are constant.

Unfortunately, in the real world, most of dst ip and dst port are not constant, so that means we need to adjust the hash table/algorithm to recombine tcp session.

[FNET] Fix the checking return code issue for fnet_get_peerip and fnet_get_localip

should check the return code for getsockname and getpeername

[FCONF] Auto configuration support

Current configuration system is too simple, which not support reload configure file. We need to do some improvement for taking it better :D

[FHASH] FNV Support

Since the current hash algorithm is not balanced, should try FNV instead of it

[FLOG] User Thread quit

When a User thread quit suddenly, the fetcher thread has already alive, so there is a case:

User thread do a cleanup, so the local buffer be freed.
Then the fetcher thread read a invalid memory during user thread cleanup period.

So, we need to use the notify mechanism to notice fetcher to free the thread data rather than a user thread when the buffer is not empty.

We can add protocol_id into message header, usually, we use protocol_id = 0 to write log message, and when a user thread quit, we use protocol_id = 1 to notice fetcher free this data. �Actually, when fetcher receive the protocol_id = 1's message, that means it is a last message in buffer, so we can do clean up in fetcher thread safety.

complie error based on "ubuntu 10.04 32bit"

I am sorry man, it can not work in the 32bit system,the errors such as:

(1)
gcc -Wall -Werror -g -O2 -std=c99 -c lhash.c
lhash.c: 在函数‘f_64itoa’中:
lhash.c:158:5: 错误：格式 ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘uint64_t’ [-Werror=format]
cc1：所有的警告都被当作是错误

(2)
gcc -Wall -Werror -g -O2 -c fco.c
fco.c: 在函数‘co_main’中:
fco.c:216:5: 错误：左移次数大于或等于类型宽度 [-Werror]
fco.c: 在函数‘fco_resume’中:
fco.c:259:25: 错误：右移次数大于或等于类型宽度 [-Werror]
cc1：所有的警告都被当作是错误

Hoping your solution....

[FCO] C coroutine support

Need to add a library for supporting coroutine.
Goals:

user friendly interface.
completely closure.
non-preemptive thread.
dead coroutine graceful terminated.

[MBUF] False Sharing issue

Separate head and tail, fill the gap align to cpu cache line
Make the bench mark report

[FLOG] Analysis with perf/systemtap

Ok, since last milestone, we improve more performance by valgrind tools, but here are some more professional tools such as perf and sytemtap. so when get time, need to use these tools to analysis flog performance again.

btw, perf need kernel version upper than 2.6.31.

[FLOG] Improve Log library

Maybe try to use a thread to fetch data msg better than notice.

database interfaces support

We need to add some interfaces for supporting some common database, such as mysql, redis, ...
Thus, we can move forward fast to deploy the components of server.

Http parser handler support

For the incoming http request, we can add a simple parser handler for this, and at user level, it will easily to handle http request not care the network operation.

[FLOG] Optimizate Log system

Currently, the performance of log system is:
single thread: 616970 msg/s
multithread(4 threads): 1229448 msg/s

Find out the hotspot in code, do some deep optimization. Action items:

hotspot
branch predict
hold time string within one second
...

[FCACHE]Cache Library

Build a cache library，which can use flist and fhash.

LRU support
Fast and Simple