fluent / fluent-bit Goto Github PK

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows

License: Apache License 2.0

BitBake 0.01% C 81.26% CMake 1.55% Shell 3.18% C++ 7.82% HTML 0.44% Makefile 1.30% Python 1.43% CSS 0.05% JavaScript 0.08% Perl 0.28% M4 1.03% XSLT 0.01% Ruby 0.12% Roff 0.61% Emacs Lisp 0.01% Awk 0.01% sed 0.01% Lua 0.75% Batchfile 0.06%

fluentd c logging data-collector fluent-bit forwarder cloudnative stream-processing sql-queries metrics

fluent-bit's Introduction

CI Status

CI Workflow	Status
Unit Tests (master)
Integration Tests (master)
Arm builds
Latest release

About

Fluent Bit is a fast Log Processor and Forwarder for Linux, Windows, Embedded Linux, MacOS and BSD family operating systems. It's part of the Graduated Fluentd Ecosystem and a CNCF sub-project.

Fluent Bit allows to collect log events or metrics from different sources, process them and deliver them to different backends such as Fluentd, Elasticsearch, Splunk, DataDog, Kafka, New Relic, Azure services, AWS services, Google services, NATS, InfluxDB or any custom HTTP end-point.

Fluent Bit comes with full SQL Stream Processing capabilities: data manipulation and analytics using SQL queries.

Fluent Bit runs on x86_64, x86, arm32v7, and arm64v8 architectures.

Features

High Performance at low CPU and Memory footprint
Data Parsing
- Convert your unstructured messages using our parsers: JSON, Regex, LTSV and Logfmt
Reliability and Data Integrity
- Backpressure Handling
- Data Buffering in memory and file system
Networking
- Security: built-in TLS/SSL support
- Asynchronous I/O
Pluggable Architecture and Extensibility: Inputs, Filters and Outputs
- More than 70 built-in plugins available
- Extensibility
  - Write any input, filter or output plugin in C language
  - Write Filters in Lua or Output plugins in Golang
Monitoring: expose internal metrics over HTTP in JSON and Prometheus format
Stream Processing: Perform data selection and transformation using simple SQL queries
- Create new streams of data using query results
- Aggregation Windows
- Data analysis and prediction: Timeseries forecasting
Portable: runs on Linux, MacOS, Windows and BSD systems

Fluent Bit in Production

Fluent Bit is used widely in production environments. As of 2022, Fluent Bit surpasses 3 Billion downloads and continues to be deployed over 10 million times a day. The following is a preview of who uses Fluent Bit heavily in production:

If your company uses Fluent Bit and is not listed, feel free to open a GitHub issue and we will add the logo.

Documentation

Our official project documentation for installation, configuration, deployment and development topics is located here:

https://docs.fluentbit.io

Quick Start

Build from Scratch

If you aim to build Fluent Bit from sources, you can go ahead and start with the following commands.

cd build
cmake ..
make
bin/fluent-bit -i cpu -o stdout -f 1

If you are interested into more details, please refer to the Build & Install section.

Requirements

CMake >= 3.0
Flex
Bison
YAML library/headers
OpenSSL library/headers

Linux Packages

We provide packages for most common Linux distributions:

Linux / Docker Container Images

Our Linux containers images are the most common deployment model, thousands of new installation happen every day, learn more about the available images and tags here.

Windows Packages

Fluent Bit is fully supported on Windows environments, get started with these instructions.

Running on s390x

Fluent Bit is supported on Linux on IBM Z(s390x), but the WASM and LUA filter plugins are not.

Plugins: Inputs, Filters and Outputs

Fluent Bit is based in a pluggable architecture where different plugins plays a major role in the data pipeline:

Input Plugins

name	title	description
collectd	Collectd	Listen for UDP packets from Collectd.
cpu	CPU Usage	measure total CPU usage of the system.
disk	Disk Usage	measure Disk I/Os.
dummy	Dummy	generate dummy event.
exec	Exec	executes external program and collects event logs.
forward	Forward	Fluentd forward protocol.
head	Head	read first part of files.
health	Health	Check health of TCP services.
kmsg	Kernel Log Buffer	read the Linux Kernel log buffer messages.
mem	Memory Usage	measure the total amount of memory used on the system.
mqtt	MQTT	start a MQTT server and receive publish messages.
netif	Network Traffic	measure network traffic.
proc	Process	Check health of Process.
random	Random	Generate Random samples.
serial	Serial Interface	read data information from the serial interface.
stdin	Standard Input	read data from the standard input.
syslog	Syslog	read syslog messages from a Unix socket.
systemd	Systemd	read logs from Systemd/Journald.
tail	Tail	Tail log files.
tcp	TCP	Listen for JSON messages over TCP.
thermal	Thermal	measure system temperature(s).

Filter Plugins

name	title	description
aws	AWS Metadata	Enrich logs with AWS Metadata.
expect	Expect	Validate records match certain criteria in structure.
grep	Grep	Match or exclude specific records by patterns.
kubernetes	Kubernetes	Enrich logs with Kubernetes Metadata.
lua	Lua	Filter records using Lua Scripts.
parser	Parser	Parse record.
record_modifier	Record Modifier	Modify record.
rewrite_tag	Rewrite Tag	Re-emit records under new tag.
stdout	Stdout	Print records to the standard output interface.
throttle	Throttle	Apply rate limit to event flow.
nest	Nest	Nest records under a specified key
modify	Modify	Modifications to record.

Output Plugins

name	title	description
azure	Azure Log Analytics	Ingest records into Azure Log Analytics
bigquery	BigQuery	Ingest records into Google BigQuery
counter	Count Records	Simple records counter.
datadog	Datadog	Ingest logs into Datadog.
es	Elasticsearch	flush records to a Elasticsearch server.
file	File	Flush records to a file.
flowcounter	FlowCounter	Count records.
forward	Forward	Fluentd forward protocol.
gelf	GELF	Flush records to Graylog
http	HTTP	Flush records to an HTTP end point.
influxdb	InfluxDB	Flush records to InfluxDB time series database.
kafka	Apache Kafka	Flush records to Apache Kafka
kafka-rest	Kafka REST Proxy	Flush records to a Kafka REST Proxy server.
loki	Loki	Flush records to Loki server.
nats	NATS	Flush records to a NATS server.
null	NULL	Throw away events.
s3	S3	Flush records to s3
stackdriver	Google Stackdriver Logging	Flush records to Google Stackdriver Logging service.
stdout	Standard Output	Flush records to the standard output.
splunk	Splunk	Flush records to a Splunk Enterprise service
tcp	TCP & TLS	Flush records to a TCP server.
td	Treasure Data	Flush records to the Treasure Data cloud service for analytics.

Contributing

Fluent Bit is an open project, several individuals and companies contribute in different forms like coding, documenting, testing, spreading the word at events within others. If you want to learn more about contributing opportunities please reach out to us through our Community Channels.

If you are interested in contributing to Fluent bit with bug fixes, new features or coding in general, please refer to the code CONTRIBUTING guidelines. You can also refer the Beginners Guide to contributing to Fluent Bit here.

Community & Contact

Feel free to join us on our Slack channel, Mailing List or IRC:

Slack (#fluent-bit channel)
Twitter

License

This program is under the terms of the Apache License v2.0.

Authors

Fluent Bit is sponsored and maintained by several companies in the Cloud Native community, including all the major cloud providers.

You can see a list of contributors here.

fluent-bit's People

Contributors

Stargazers

Watchers

Forkers

syohex enukane pandax381 hasegaw rkawajiri viirya loginsight udemirezen nokute78 coxnegative camilorodegheri salekseev tommaroh leahnp ecoblockchain cpg1111 chinhui-chen takahasi rissam sharadgaur ajoybharath rocky-keeling6 nhosoi h0tw1r3 lorisbc leitao dhananjaysathe donmartin76 gamer22026 gmi7x mahdi-asadpour-rr ii secsecsec ddelbondio micklonge torden hetile-ssense duythinht kushwiz sdmo0 jenningsloy318 raghu999 adrianlzt epcim marcosrmendezthd astroza davidnarayan ethan-zhengyw jsravn mrueg wangtechservices salewski 0x17de mortent onorua ryan2x ganezu79 ducky-hong justnoise mikn costypetrisor jcchavezs panta mwindower appcoreopc ziosyoung bboreham sjanulonoks ffscl brava-vinh michiel digitalshift abdulhaseebhussain amitkumarj441 erikluo nitisht cfs-pure intelliguy ashutoshdhundhara manuelluis trstringer tikyau sammerry flodila worrel rtartler-desres homer6 jessestuart bivas martinssipenko wangjieboy zozzz davidewatson ellerbrock cloud-architecture mikekap projoke brandoncole yuanfeng0905 saint1991

fluent-bit's Issues

Remove unnecessary null pointer checks

An extra null pointer check is not needed in functions like the following.

Would you like to apply the following semantic patch to find more update candidates?

@Remove_unnecessary_pointer_checks@
expression x;
@@
-if (x)
    free(x);

Add wildcards support for routing

Fluent Bit 0.7 is adding tag routing, we need to add wildcards support for the matches instead of support just static string match.

Input Plugins OS Support

It would be nice if some of the unsupported inputs were supported on these platforms;

in_cpu
in_mem
in_kmsg
in_tail
in_proc

I'm not a C developer, or a BSD developer, but I'm willing to start work on the in_tail plugin--I need that one for a work project (collecting logs from a PfSense router). I'm not sure where to begin with this though, so it would be awesome if someone could get me pointed in the right direction.

P.S: I do know Golang fairly well, so I could probably be more useful there if that is a viable route.

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

printf ⇒ flb_message
signal ⇒ flb_signal_init

Add Filtering support

Fluent Bit needs to get filtering plugins support, so data form inputs can be modified.

First priority is to get a Kubernetes metadata filter.

in_serial: Support JSON Format

in_serial plugin read a stream of bytes, so the only thing is that user need to specify a delimiter string so the plugin can know when a message ends and a new one begins.
But, adding delimiter to JSON format is not suitable.

The request is that in_serial supports JSON Format.

Support for File Input (in_tail)

I could not find any documentation on how to read a log file like how fluentD does to output to elasticsearch on fluent-bit. In my case, there's already a stdout that I'm processing using fluentD (in Kubernetes). I'm trying to read some additional log files inside a pod using a pod volume.

Is this feature supported? Is it been targeted for a release in the near future?

out_flowcounter can't count from multiple inputs.

out_flowcounter doesn't consider multiple inputs.

In this case, out_flowcounter counts data only from in_cpu.
I'm thinking how fix it...

$ bin/fluent-bit -i cpu -i mem -i random -o flowcounter -m '*'
Fluent-Bit v0.10.0
Copyright (C) Treasure Data

[2016/12/22 20:48:35] [ info] [engine] started
[out_flowcounter] cpu.0:[1482407375, {"counts":178, "bytes":10982, "counts/minute":2, "bytes/minute":183 }]
[out_flowcounter] cpu.0:[1482407435, {"counts":180, "bytes":11040, "counts/minute":3, "bytes/minute":184 }]
[out_flowcounter] cpu.0:[1482407495, {"counts":180, "bytes":11040, "counts/minute":3, "bytes/minute":184 }]
[out_flowcounter] cpu.0:[1482407555, {"counts":180, "bytes":11040, "counts/minute":3, "bytes/minute":184 }]
[out_flowcounter] cpu.0:[1482407615, {"counts":180, "bytes":11040, "counts/minute":3, "bytes/minute":184 }]

reserved identifier violation

I would like to point out that an identifier like "__flb_config_verbose" does eventually not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?

libxbee3 memory leak

the libxbee3 component have a memory leak, I filed an issue here:

attie/libxbee3#9

Build: failed to generate the executable file

$ make
...
Scanning dependencies of target fluent-bit-bin
[ 81%] Building C object src/CMakeFiles/fluent-bit-bin.dir/fluent-bit.c.o
Linking C executable ../bin/fluent-bit
../library/libmk_core.a(mk_event.c.o): In function `mk_utils_libc_error':
mk_event.c:(.text+0x81): undefined reference to `pthread_getspecific'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/fluent-bit] Error 1
make[1]: *** [src/CMakeFiles/fluent-bit-bin.dir/all] Error 2
make: *** [all] Error 2

out_nats: protocol info before TLS handshake

When a connection is issued to a NATS server, the client receives an INFO message as soon the connection is established, e.g:

{"server_id":"e1d84d38a2c8f11f4209f079d8c82203",
 "version":"0.7.5",
 "go": "go1.5.3", 
 "host":"0.0.0.0", 
 "port":4449,
 "auth_required":true,
 "ssl_required":true,
 "tls_required":true,
 "tls_verify":true,
 "max_payload":1048576
}

so based on this information the client (out_nats) must be able to switch to TLS mode. Fluent Bit already supports TLS but we need a tweak to do:

connect to NATS server
parse INFO message
if TLS is enable, use Fluent Bit I/O TLS support, otherwise raw TCP backend

High load / 100% CPU usage on Raspberry Pi

Problem: fluent-bit process maxes out a CPU core (100% usage). My build environment is the Raspberry Pi 1 (ARM). Followed the compile instructions in the README.

This is the command:

head events.json | ~/src/fluent-bit/build/bin/fluent-bit -i stdin -o stdout -f 1 > outtest

This only sends 10 events through fluent-bit. It should be idle after flushing those. The actual result is the high CPU usage.

Run strace on the pid and I see the following lines being repeated while the process maxes out the CPU:

read(5, "", 32766)                      = 0
epoll_wait(3, {{EPOLLHUP, {u32=3363536, u64=4298330832}}}, 256, -1) = 1
read(5, "", 32766)                      = 0
epoll_wait(3, {{EPOLLHUP, {u32=3363536, u64=4298330832}}}, 256, -1) = 1
read(5, "", 32766)                      = 0
epoll_wait(3, {{EPOLLHUP, {u32=3363536, u64=4298330832}}}, 256, -1) = 1
...

Buffer size of stdin?

Hi there, thanks for the cool utility.

I'm trying to find out how big the buffer size is when reading from stdin. Let's say I have a 10M file and I just want to cat that file through fluent bit to my target.

So I do: cat events.log | td-agent-bit -c td-agent-bit.conf where the conf file is:

[SERVICE]
    Flush     1
    Daemon    off
    Log_Level debug

[INPUT]
    Name stdin
    Tag  events

[OUTPUT]
    Name forward
    Match events
    Host example.com
    Port 24224

I would expect the entire 10M file to be sent through the pipeline, however the actual result is that I get:

TD-Agent Bit v0.8.3
Copyright (C) Treasure Data

[2016/07/25 16:27:12] [ info] starting engine
[2016/07/25 16:27:12] [debug] [router] match rule stdin.0:forward.0
[2016/07/25 16:27:14] [ warn] STDIN data incomplete, waiting for more data...

And only a tiny fraction of the file contents are delivered.

Now, I can slow down the pipe using the following:

cat events.log | pv -l -L 10 -q | td-agent-bit -c td-agent-bit.conf

But it still doesn't work as I'd expect, I get many more events delivered, but still just about 1/10th. And that's slowing the rate to 10 bytes / second.

Is there a configuration toggle for the size of the buffer? I would like it to keep buffering data up to a certain limit, bounded by the amount of RAM on the box, I guess.

Thanks!

Fix signal handler

The function "flb_debug" does not belong to the list of async-signal-safe functions.
How do you think about to delete its call from your function "flb_signal_handler"?

NATS: out_nats

Add NATS output support

Upgrade lib/jsmn

The jsmn library used had a bug and Fluent Bit code have a workaround for it. As soon as jsmn becomes fixed we should upgrade:

zserge/jsmn#81

Linking Fails in FreeBSD 10.3

Maybe I'm making a n00b mistake, but it looks like neither 0.10.0 or master will compile on FreeBSD, regardless of version;

root@FreeBSD-10:~/fluent-bit/build # cmake ..
-- The C compiler identification is Clang 3.4.1
-- The CXX compiler identification is Clang 3.4.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/CC
-- Check for working CXX compiler: /usr/bin/CC -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find GTest (missing:  GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY)
-- Found ZLIB: /usr/lib/libz.so (found version "1.2.8")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Performing Test MSGPACK_ENABLE_GCC_CXX_ATOMIC
-- Performing Test MSGPACK_ENABLE_GCC_CXX_ATOMIC - Failed
-- Performing Test FLB_HAVE_SYNC_SUB
-- Performing Test FLB_HAVE_SYNC_SUB - Success
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE)
-- Performing Test HAVE_STAT_H
-- Performing Test HAVE_STAT_H - Success
-- Performing Test HAVE_SYS_UIO_H
-- Performing Test HAVE_SYS_UIO_H - Success
-- Performing Test HAVE_UNISTD_H
-- Performing Test HAVE_UNISTD_H - Success
-- Performing Test HAVE_EPOLL
-- Performing Test HAVE_EPOLL - Failed
-- Performing Test HAVE_KQUEUE
-- Performing Test HAVE_KQUEUE - Failed
-- Event loop backend > select(2)
-- Performing Test HAVE_TIMERFD_CREATE
-- Performing Test HAVE_TIMERFD_CREATE - Failed
-- Performing Test HAVE_EVENTFD
-- Performing Test HAVE_EVENTFD - Failed
-- Found Perl: /usr/local/bin/perl (found version "5.20.3")
Can't open perl script "/root/fluent-bit/scripts/config.pl": No such file or directory
-- Performing Test FLB_HAVE_UCONTEXT
-- Performing Test FLB_HAVE_UCONTEXT - Success
-- Performing Test FLB_HAVE_VALGRIND
-- Performing Test FLB_HAVE_VALGRIND - Failed
-- Performing Test FLB_HAVE_FORK
-- Performing Test FLB_HAVE_FORK - Success
-- Performing Test FLB_HAVE_C_TLS
-- Performing Test FLB_HAVE_C_TLS - Success
-- Performing Test FLB_HAVE_SETJMP
-- Performing Test FLB_HAVE_SETJMP - Success
-- Performing Test FLB_HAVE_ACCEPT4
-- Performing Test FLB_HAVE_ACCEPT4 - Success
-- Performing Test FLB_HAVE_INOTIFY
-- Performing Test FLB_HAVE_INOTIFY - Failed
-- Looking for accept4
-- Looking for accept4 - found
-- Configuring done
CMake Warning (dev) at src/CMakeLists.txt:153 (add_library):
  Policy CMP0063 is not set: Honor visibility properties for all target
  types.  Run "cmake --help-policy CMP0063" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  Target "fluent-bit-static" of type "STATIC_LIBRARY" has the following
  visibility properties set for C:

    C_VISIBILITY_PRESET

  For compatibility CMake is not honoring them for this target.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at src/proxy/go/CMakeLists.txt:4 (add_library):
  Policy CMP0063 is not set: Honor visibility properties for all target
  types.  Run "cmake --help-policy CMP0063" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  Target "flb-plugin-proxy-go" of type "STATIC_LIBRARY" has the following
  visibility properties set for C:

    C_VISIBILITY_PRESET

  For compatibility CMake is not honoring them for this target.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /root/fluent-bit/build
root@FreeBSD-10:~/fluent-bit/build # make
Scanning dependencies of target msgpackc-static
[  0%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/objectc.c.o
[  1%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/unpack.c.o
[  1%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/version.c.o
[  2%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/vrefbuffer.c.o
[  2%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/zone.c.o
[  3%] Linking C static library ../../library/libmsgpackc.a
[  3%] Built target msgpackc-static
Scanning dependencies of target jsmn
[  4%] Building C object lib/jsmn/CMakeFiles/jsmn.dir/jsmn.c.o
[  4%] Linking C static library ../../library/libjsmn.a
[  4%] Built target jsmn
Scanning dependencies of target ejson
[  5%] Building C object lib/ejson/CMakeFiles/ejson.dir/ejson.c.o
/root/fluent-bit/lib/ejson/ejson.c:167:20: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
    ret = snprintf(buf + ctx->buf_pos,
                   ^~~~~~~~~~~~~~~~~~
/usr/include/stdio.h:282:32: note: passing argument to parameter here
int      snprintf(char * __restrict, size_t, const char * __restrict,
                                   ^
1 warning generated.
[  5%] Linking C static library ../../library/libejson.a
[  5%] Built target ejson
Scanning dependencies of target mk_core
[  6%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_rconf.c.o
[  6%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_string.c.o
[  7%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_memory.c.o
[  7%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_event.c.o
[  8%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_utils.c.o
[  8%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_file.c.o
[  8%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_iov.c.o
[  9%] Linking C static library ../../../library/libmk_core.a
[  9%] Built target mk_core
Scanning dependencies of target mbedcrypto
[  9%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/aes.c.o
[ 10%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/aesni.c.o
[ 10%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/arc4.c.o
[ 11%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/asn1parse.c.o
[ 11%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/asn1write.c.o
[ 12%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/base64.c.o
[ 12%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/bignum.c.o
[ 13%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/blowfish.c.o
[ 13%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/camellia.c.o
[ 14%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ccm.c.o
[ 14%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/cipher.c.o
[ 15%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/cipher_wrap.c.o
[ 15%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/cmac.c.o
[ 15%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ctr_drbg.c.o
[ 16%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/des.c.o
[ 16%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/dhm.c.o
[ 17%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecdh.c.o
[ 17%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecdsa.c.o
[ 18%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecjpake.c.o
[ 18%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecp.c.o
[ 19%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecp_curves.c.o
[ 19%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/entropy.c.o
[ 20%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/entropy_poll.c.o
[ 20%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/error.c.o
[ 21%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/gcm.c.o
[ 21%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/havege.c.o
[ 21%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/hmac_drbg.c.o
[ 22%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md.c.o
[ 22%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md2.c.o
[ 23%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md4.c.o
[ 23%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md5.c.o
[ 24%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md_wrap.c.o
[ 24%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/memory_buffer_alloc.c.o
[ 25%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/oid.c.o
[ 25%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/padlock.c.o
[ 26%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pem.c.o
[ 26%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pk.c.o
[ 27%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pk_wrap.c.o
[ 27%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pkcs12.c.o
[ 28%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pkcs5.c.o
[ 28%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pkparse.c.o
[ 28%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pkwrite.c.o
[ 29%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/platform.c.o
[ 29%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ripemd160.c.o
[ 30%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/rsa.c.o
[ 30%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/sha1.c.o
[ 31%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/sha256.c.o
[ 31%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/sha512.c.o
[ 32%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/threading.c.o
[ 32%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/timing.c.o
[ 33%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/version.c.o
[ 33%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/version_features.c.o
[ 34%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/xtea.c.o
[ 34%] Linking C static library ../../../library/libmbedcrypto.a
[ 34%] Built target mbedcrypto
Scanning dependencies of target mbedx509
[ 34%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/certs.c.o
[ 35%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/pkcs11.c.o
[ 35%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509.c.o
[ 35%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509_create.c.o
[ 36%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509_crl.c.o
[ 36%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509_crt.c.o
[ 37%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509_csr.c.o
[ 37%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509write_crt.c.o
[ 38%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509write_csr.c.o
[ 38%] Linking C static library ../../../library/libmbedx509.a
[ 38%] Built target mbedx509
Scanning dependencies of target mbedtls
[ 38%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/debug.c.o
[ 39%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/net_sockets.c.o
[ 39%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_cache.c.o
[ 40%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_ciphersuites.c.o
[ 40%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_cli.c.o
[ 41%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_cookie.c.o
[ 41%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_srv.c.o
[ 42%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_ticket.c.o
[ 42%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_tls.c.o
[ 43%] Linking C static library ../../../library/libmbedtls.a
[ 43%] Built target mbedtls
Scanning dependencies of target sqlite3
[ 43%] Building C object lib/sqlite-amalgamation-3150200/CMakeFiles/sqlite3.dir/sqlite3.c.o
[ 44%] Linking C static library ../../library/libsqlite3.a
[ 44%] Built target sqlite3
Scanning dependencies of target cJSON
[ 44%] Building C object lib/cjson/CMakeFiles/cJSON.dir/src/cJSON.c.o
[ 44%] Linking C static library ../../library/libcJSON.a
[ 44%] Built target cJSON
Scanning dependencies of target co
[ 45%] Building C object lib/flb_libco/CMakeFiles/co.dir/libco.c.o
[ 45%] Linking C static library ../../library/libco.a
[ 45%] Built target co
Scanning dependencies of target flb-plugin-proxy-go
[ 46%] Building C object src/proxy/go/CMakeFiles/flb-plugin-proxy-go.dir/go.c.o
[ 46%] Linking C static library ../../../library/libflb-plugin-proxy-go.a
[ 46%] Built target flb-plugin-proxy-go
Scanning dependencies of target fluent-bit-static
[ 46%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_lib.c.o
[ 47%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_log.c.o
[ 47%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_uri.c.o
[ 47%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_pack.c.o
[ 48%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_sha1.c.o
[ 48%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_pipe.c.o
[ 49%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_kernel.c.o
[ 49%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_input.c.o
[ 50%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_output.c.o
[ 50%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_config.c.o
[ 51%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_network.c.o
[ 51%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_utils.c.o
[ 52%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_engine.c.o
[ 52%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_engine_dispatch.c.o
[ 53%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_task.c.o
[ 53%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_scheduler.c.o
[ 54%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_io.c.o
[ 54%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_upstream.c.o
[ 54%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_router.c.o
[ 55%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_http_client.c.o
[ 55%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_worker.c.o
[ 56%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_io_tls.c.o
[ 56%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_plugin_proxy.c.o
[ 57%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_sqldb.c.o
[ 57%] Linking C static library ../library/libfluent-bit.a
[ 57%] Built target fluent-bit-static
Scanning dependencies of target flb-plugin-out_flowcounter
[ 57%] Building C object plugins/out_flowcounter/CMakeFiles/flb-plugin-out_flowcounter.dir/out_flowcounter.c.o
[ 58%] Linking C static library ../../library/libflb-plugin-out_flowcounter.a
[ 58%] Built target flb-plugin-out_flowcounter
Scanning dependencies of target flb-plugin-out_lib
[ 58%] Building C object plugins/out_lib/CMakeFiles/flb-plugin-out_lib.dir/out_lib.c.o
[ 59%] Linking C static library ../../library/libflb-plugin-out_lib.a
[ 59%] Built target flb-plugin-out_lib
Scanning dependencies of target flb-plugin-out_td
[ 60%] Building C object plugins/out_td/CMakeFiles/flb-plugin-out_td.dir/td_http.c.o
In file included from /root/fluent-bit/plugins/out_td/td_http.c:28:
/root/fluent-bit/plugins/out_td/miniz/miniz.c:1012:14: warning: unused function 'def_realloc_func' [-Wunused-function]
static void *def_realloc_func(void *opaque, void *address, size_t items, size_t size) { (void)opaque, (void)address, (void)items, (void)size; return MZ_REALLOC(address, items * size); }
             ^
1 warning generated.
[ 60%] Building C object plugins/out_td/CMakeFiles/flb-plugin-out_td.dir/td_config.c.o
[ 61%] Building C object plugins/out_td/CMakeFiles/flb-plugin-out_td.dir/td.c.o
[ 61%] Linking C static library ../../library/libflb-plugin-out_td.a
[ 61%] Built target flb-plugin-out_td
Scanning dependencies of target flb-plugin-out_stdout
[ 62%] Building C object plugins/out_stdout/CMakeFiles/flb-plugin-out_stdout.dir/stdout.c.o
[ 62%] Linking C static library ../../library/libflb-plugin-out_stdout.a
[ 62%] Built target flb-plugin-out_stdout
Scanning dependencies of target flb-plugin-out_plot
[ 62%] Building C object plugins/out_plot/CMakeFiles/flb-plugin-out_plot.dir/plot.c.o
/root/fluent-bit/plugins/out_plot/plot.c:163:13: warning: implicit declaration of function 'dprintf' is invalid in C99 [-Wimplicit-function-declaration]
            dprintf(fd, "%lu %" PRIu64 "\n", atime, val->via.u64);
            ^
1 warning generated.
[ 62%] Linking C static library ../../library/libflb-plugin-out_plot.a
[ 62%] Built target flb-plugin-out_plot
Scanning dependencies of target flb-plugin-out_null
[ 62%] Building C object plugins/out_null/CMakeFiles/flb-plugin-out_null.dir/null.c.o
[ 63%] Linking C static library ../../library/libflb-plugin-out_null.a
[ 63%] Built target flb-plugin-out_null
Scanning dependencies of target flb-plugin-out_influxdb
[ 64%] Building C object plugins/out_influxdb/CMakeFiles/flb-plugin-out_influxdb.dir/influxdb_bulk.c.o
[ 64%] Building C object plugins/out_influxdb/CMakeFiles/flb-plugin-out_influxdb.dir/influxdb.c.o
[ 65%] Linking C static library ../../library/libflb-plugin-out_influxdb.a
[ 65%] Built target flb-plugin-out_influxdb
Scanning dependencies of target flb-plugin-out_http
[ 66%] Building C object plugins/out_http/CMakeFiles/flb-plugin-out_http.dir/http.c.o
[ 66%] Linking C static library ../../library/libflb-plugin-out_http.a
[ 66%] Built target flb-plugin-out_http
Scanning dependencies of target flb-plugin-out_forward
[ 66%] Building C object plugins/out_forward/CMakeFiles/flb-plugin-out_forward.dir/__/__/src/flb_network.c.o
[ 67%] Building C object plugins/out_forward/CMakeFiles/flb-plugin-out_forward.dir/forward.c.o
[ 67%] Linking C static library ../../library/libflb-plugin-out_forward.a
[ 67%] Built target flb-plugin-out_forward
Scanning dependencies of target flb-plugin-out_es
[ 67%] Building C object plugins/out_es/CMakeFiles/flb-plugin-out_es.dir/es_bulk.c.o
[ 68%] Building C object plugins/out_es/CMakeFiles/flb-plugin-out_es.dir/es.c.o
[ 68%] Linking C static library ../../library/libflb-plugin-out_es.a
[ 68%] Built target flb-plugin-out_es
Scanning dependencies of target flb-plugin-out_counter
[ 68%] Building C object plugins/out_counter/CMakeFiles/flb-plugin-out_counter.dir/counter.c.o
[ 69%] Linking C static library ../../library/libflb-plugin-out_counter.a
[ 69%] Built target flb-plugin-out_counter
Scanning dependencies of target flb-plugin-in_random
[ 70%] Building C object plugins/in_random/CMakeFiles/flb-plugin-in_random.dir/random.c.o
[ 70%] Linking C static library ../../library/libflb-plugin-in_random.a
[ 70%] Built target flb-plugin-in_random
Scanning dependencies of target flb-plugin-in_forward
[ 71%] Building C object plugins/in_forward/CMakeFiles/flb-plugin-in_forward.dir/fw.c.o
/root/fluent-bit/plugins/in_forward/fw.c:91:41: warning: incompatible pointer types passing 'int (struct flb_config *, void *)' to parameter of type 'int (*)(struct flb_input_instance *, struct flb_config *, void *)' [-Wincompatible-pointer-types]
                                        in_fw_collect,
                                        ^~~~~~~~~~~~~
/root/fluent-bit/include/fluent-bit/flb_input.h:451:42: note: passing argument to parameter 'cb_new_connection' here
                                   int (*cb_new_connection) (struct flb_input_instance *,
                                         ^
/root/fluent-bit/plugins/in_forward/fw.c:124:21: warning: incompatible pointer types initializing 'int (*)(struct flb_input_instance *, struct flb_config *, void *)' with an expression of type 'int (struct flb_config *, void *)' [-Wincompatible-pointer-types]
    .cb_collect   = in_fw_collect,
                    ^~~~~~~~~~~~~
2 warnings generated.
[ 71%] Building C object plugins/in_forward/CMakeFiles/flb-plugin-in_forward.dir/fw_conn.c.o
[ 72%] Building C object plugins/in_forward/CMakeFiles/flb-plugin-in_forward.dir/fw_prot.c.o
[ 72%] Building C object plugins/in_forward/CMakeFiles/flb-plugin-in_forward.dir/fw_config.c.o
[ 73%] Linking C static library ../../library/libflb-plugin-in_forward.a
[ 73%] Built target flb-plugin-in_forward
Scanning dependencies of target flb-plugin-in_lib
[ 74%] Building C object plugins/in_lib/CMakeFiles/flb-plugin-in_lib.dir/in_lib.c.o
[ 74%] Building C object plugins/in_lib/CMakeFiles/flb-plugin-in_lib.dir/__/__/src/flb_pack.c.o
[ 75%] Linking C static library ../../library/libflb-plugin-in_lib.a
[ 75%] Built target flb-plugin-in_lib
Scanning dependencies of target flb-plugin-in_mqtt
[ 75%] Building C object plugins/in_mqtt/CMakeFiles/flb-plugin-in_mqtt.dir/mqtt.c.o
[ 76%] Building C object plugins/in_mqtt/CMakeFiles/flb-plugin-in_mqtt.dir/mqtt_conn.c.o
[ 76%] Building C object plugins/in_mqtt/CMakeFiles/flb-plugin-in_mqtt.dir/mqtt_prot.c.o
[ 77%] Building C object plugins/in_mqtt/CMakeFiles/flb-plugin-in_mqtt.dir/mqtt_config.c.o
[ 77%] Linking C static library ../../library/libflb-plugin-in_mqtt.a
[ 77%] Built target flb-plugin-in_mqtt
Scanning dependencies of target flb-plugin-in_tcp
[ 77%] Building C object plugins/in_tcp/CMakeFiles/flb-plugin-in_tcp.dir/tcp.c.o
[ 78%] Building C object plugins/in_tcp/CMakeFiles/flb-plugin-in_tcp.dir/tcp_conn.c.o
[ 78%] Building C object plugins/in_tcp/CMakeFiles/flb-plugin-in_tcp.dir/tcp_config.c.o
[ 79%] Linking C static library ../../library/libflb-plugin-in_tcp.a
[ 79%] Built target flb-plugin-in_tcp
Scanning dependencies of target flb-plugin-in_stdin
[ 80%] Building C object plugins/in_stdin/CMakeFiles/flb-plugin-in_stdin.dir/in_stdin.c.o
[ 80%] Building C object plugins/in_stdin/CMakeFiles/flb-plugin-in_stdin.dir/__/__/src/flb_pack.c.o
[ 81%] Linking C static library ../../library/libflb-plugin-in_stdin.a
[ 81%] Built target flb-plugin-in_stdin
Scanning dependencies of target flb-plugin-in_serial
[ 82%] Building C object plugins/in_serial/CMakeFiles/flb-plugin-in_serial.dir/in_serial.c.o
[ 82%] Building C object plugins/in_serial/CMakeFiles/flb-plugin-in_serial.dir/in_serial_config.c.o
[ 82%] Linking C static library ../../library/libflb-plugin-in_serial.a
[ 82%] Built target flb-plugin-in_serial
Scanning dependencies of target flb-plugin-in_health
[ 82%] Building C object plugins/in_health/CMakeFiles/flb-plugin-in_health.dir/health.c.o
[ 82%] Linking C static library ../../library/libflb-plugin-in_health.a
[ 82%] Built target flb-plugin-in_health
Scanning dependencies of target flb-plugin-in_head
[ 82%] Building C object plugins/in_head/CMakeFiles/flb-plugin-in_head.dir/in_head.c.o
[ 83%] Linking C static library ../../library/libflb-plugin-in_head.a
[ 83%] Built target flb-plugin-in_head
Scanning dependencies of target fluent-bit-shared
[ 84%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_lib.c.o
[ 84%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_log.c.o
[ 84%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_uri.c.o
[ 85%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_pack.c.o
[ 85%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_sha1.c.o
[ 86%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_pipe.c.o
[ 86%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_kernel.c.o
[ 87%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_input.c.o
[ 87%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_output.c.o
[ 88%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_config.c.o
[ 88%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_network.c.o
[ 89%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_utils.c.o
[ 89%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_engine.c.o
[ 90%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_engine_dispatch.c.o
[ 90%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_task.c.o
[ 90%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_scheduler.c.o
[ 91%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_io.c.o
[ 91%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_upstream.c.o
[ 92%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_router.c.o
[ 92%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_http_client.c.o
[ 93%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_worker.c.o
[ 93%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_io_tls.c.o
[ 94%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_plugin_proxy.c.o
[ 94%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_sqldb.c.o
[ 95%] Linking C shared library ../lib/libfluent-bit.so
/usr/bin/ld: cannot find -ldl
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Stop.
make[2]: stopped in /root/fluent-bit/build
*** Error code 1

Stop.
make[1]: stopped in /root/fluent-bit/build
*** Error code 1

Stop.
make: stopped in /root/fluent-bit/build
root@FreeBSD-10:~/fluent-bit/build #

Stdin plugin not working properly on Alpine linux

I've created a docker container with fluent-bit + alpine linux, however I attempt to run the example from site to pipe the script to fluent-bit stdin but it isn't work. Fluent bit not receive the script output.

test.sh

#!/bin/sh

while :; do
  echo -n "{\"key\": \"some value\"}"
  sleep 1
done

test.sh | fluent-bit -i stdin -o stdout

On Ubuntu works fine, only on alpine that I have this issue.

[Idea] Hiding plugin instance at library mode

Now, flb_output_new and flb_input_new return a pointer of plugin instance.
Would you move them to flb_ctx_t to hide from user ?

Like this.

struct flb_lib_ctx {
    struct mk_event_loop *event_loop;
    struct mk_event *event_channel;
    struct flb_config *config;
    struct flb_input_instance *in;
    struct flb_output_instance *out;
};

User specify plugin instance with plugin id (like fd).

   int in_id = 0;
   ...
   in_id = flb_input_new(ctx, "mem", NULL); /* In this function, generate a instance and set  ctx->in[in_id] to the instance */
   flb_input_set(in_id, ctx, "tag", "test", NULL); /* set ctx->in[in_id]->properties to "tag" */
   ...
   flb_destroy(ctx); /* release all of ctx->in */

Pros:

User takes care of only flb_ctx_t. User doesn't need to release plugin instances. Engine will do.
Engine can control stop and release timing of instances in flb_stop or flb_destroy. (Now, it depends on user code.)
User also can specify plugin with plugin id.

Cons:

Lose compatibility.

How about this?

out_nats: handle heartbeat

the current NATS output plugin do not implement the heartbeat check, this is useful to maintain a keepalive TCP connection.

"pre_exit" callback for plugins

Eventually some plugins would need "pre_exit" hook. In case of in_serial plugin, it would be better to have a chance to restore original UART parameters.

filter plugin causes segfault in some cases

I tested filter plugin and I found it causes segfault in some cases.

no options for filter plugin

These commands cause segfault. Because "match" is NULL.

$ fluent-bit -i cpu -o stdout -F stdout

and

$ fluent-bit -i cpu -o stdout -F stdout -m '*'

Is option '-m' for only output plugins?

Grep plugin also causes.

$ fluent-bit -i cpu -o stdout -F grep

parse error at set_rules

Illegal property (regex=aaa in this case)causes segfault.
This is a no regular expression error.

$ fluent-bit -i cpu -o stdout -F grep -p "match=*" -p "regex=aaa"

Segfault randomly happens with in_health

How to reproduce

$ bin/fluent-bit -i health://google.com:80 -o stdout -vvv

Note

Segfault is randomly happened.
However it seems to be happen after outputting this log and it's id >= 2.

[2016/11/18 22:54:49] [debug] [input] destroy input_thread id=3

Log

$ bin/fluent-bit -i health://google.com:80 -o stdout -vvv
Fluent-Bit v0.10.0
Copyright (C) Treasure Data

[2016/11/18 22:54:28] [ info] [engine] started
[2016/11/18 22:54:28] [debug] [router] default match rule health.0:stdout.0
[2016/11/18 22:54:29] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:30] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:31] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:32] [debug] [input] destroy input_thread id=0
[0] health.0: [1479477269, {"alive"=>true}]
[1] health.0: [1479477270, {"alive"=>true}]
[2] health.0: [1479477271, {"alive"=>true}]
[3] health.0: [1479477272, {"alive"=>true}]
[2016/11/18 22:54:33] [debug] [task] created task=0x1dc9cf0 id=0 OK
[2016/11/18 22:54:35] [debug] [task] destroy task=0x1dc9cf0 (task_id=0)
[2016/11/18 22:54:35] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:35] [debug] [input] destroy input_thread id=1
[2016/11/18 22:54:36] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:37] [debug] [input] destroy input_thread id=0
[0] health.0: [1479477275, {"alive"=>true}]
[1] health.0: [1479477275, {"alive"=>true}]
[2] health.0: [1479477276, {"alive"=>true}]
[3] health.0: [1479477277, {"alive"=>true}]
[2016/11/18 22:54:38] [debug] [task] created task=0x1dc81e0 id=0 OK
[2016/11/18 22:54:38] [debug] [task] destroy task=0x1dc81e0 (task_id=0)
[2016/11/18 22:54:38] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:39] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:40] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:41] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:42] [debug] [input] destroy input_thread id=0
[0] health.0: [1479477278, {"alive"=>true}]
[1] health.0: [1479477279, {"alive"=>true}]
[2] health.0: [1479477280, {"alive"=>true}]
[3] health.0: [1479477281, {"alive"=>true}]
[4] health.0: [1479477282, {"alive"=>true}]
[2016/11/18 22:54:43] [debug] [task] created task=0x1dc1f20 id=0 OK
[2016/11/18 22:54:47] [debug] [task] destroy task=0x1dc1f20 (task_id=0)
[2016/11/18 22:54:49] [debug] [task] created task=0x1dc2320 id=0 OK
[0] health.0: [1479477289, {"alive"=>true}]
[2016/11/18 22:54:49] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:49] [debug] [task] destroy task=0x1dc2320 (task_id=0)
[2016/11/18 22:54:49] [debug] [input] destroy input_thread id=2
[2016/11/18 22:54:49] [debug] [input] destroy input_thread id=3
Segmentation fault

Ability to flag features/plugins support by Operating System

As Fluent Bit is becoming portable between Linux, OSX and BSD, it should expose some mechanism to whitelist the features allowed for the target system (some plugins may not work on all systems)

Allow all HTTP 2xx codes in out_http to result in success

At https://github.com/fluent/fluent-bit/blob/master/plugins/out_http/http.c#L394 you only check for http return code 200 but all 2xx codes mean success per https://httpstatuses.com, in particular my application returns 204.

out_forward: support for Fluentd secure_forward (TLS)

Development version of Fluent Bit allows to set optional TLS encryption for network I/O operations. The out_fluentd plugin needs support for the secure_forward protocol which after the common TLS handshake and certs checking it performs an extra authentication described here:

https://github.com/tagomoris/fluent-plugin-secure-forward#handshake

Some optional properties that needs to be handled:

self_hostname
shared_key

Upgrade lib/msgpack to 1.0.1

We are using 0.5.9, we should upgrade to the latest one:

https://github.com/msgpack/msgpack-c/releases/download/cpp-1.1.0/msgpack-1.1.0.tar.gz

segfault caused by out_stdout

I tested 15047b7 and Segfault occurs.
(By the way, Default log level is debug in that commit. Is it intentional?)

How to reproduce

$ fluent-bit -i cpu -o stdout
wait a minute

~~On the other hand, segfault is not happened when output is out_null.~~

gdb log

It is caused by src/flb_task.c.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000417e53 in flb_task_retry_clean (task=0x6d3f90, data=0x3a3298f040)
    at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/src/flb_task.c:124
124     o_ins = out_th->o_ins;

Full log is

$ gdb bin/fluent-bit 
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-90.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/build/bin/fluent-bit...done.
(gdb) run -i cpu -o stdout -f 1
Starting program: /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/build/bin/fluent-bit -i cpu -o stdout -f 1
[Thread debugging using libthread_db enabled]
[New Thread 0x2aaaaacc1700 (LWP 27777)]
Fluent-Bit v0.9.0
Copyright (C) Treasure Data

[2016/10/03 22:07:00] [ info] [engine] started
[2016/10/03 22:07:00] [debug] [router] default match rule cpu.0:stdout.0
[2016/10/03 22:07:02] [debug] [task] created task=0x6d3f90 OK
[0] cpu.0: [1475500021, {"cpu_p"=>4.000000, "user_p"=>1.000000, "system_p"=>3.000000, "cpu0.p_cpu"=>4.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>3.000000}]
[2016/10/03 22:07:03] [debug] [task] created task=0x6d41f0 OK
[0] cpu.0: [1475500022, {"cpu_p"=>0.000000, "user_p"=>0.000000, "system_p"=>0.000000, "cpu0.p_cpu"=>0.000000, "cpu0.p_user"=>0.000000, "cpu0.p_system"=>0.000000}]
[2016/10/03 22:07:04] [debug] [task] created task=0x6d4320 OK
[0] cpu.0: [1475500023, {"cpu_p"=>2.000000, "user_p"=>1.000000, "system_p"=>1.000000, "cpu0.p_cpu"=>2.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>1.000000}]
[2016/10/03 22:07:05] [debug] [task] created task=0x6d4470 OK
[0] cpu.0: [1475500024, {"cpu_p"=>1.000000, "user_p"=>1.000000, "system_p"=>0.000000, "cpu0.p_cpu"=>1.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>0.000000}]
[2016/10/03 22:07:06] [debug] [task] created task=0x6d45e0 OK
[0] cpu.0: [1475500025, {"cpu_p"=>4.000000, "user_p"=>2.000000, "system_p"=>2.000000, "cpu0.p_cpu"=>4.000000, "cpu0.p_user"=>2.000000, "cpu0.p_system"=>2.000000}]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000417e53 in flb_task_retry_clean (task=0x6d3f90, data=0x3a3298f040)
    at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/src/flb_task.c:124
124     o_ins = out_th->o_ins;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.192.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb)

Is there stable tag?

It would be great to have Fluent Bit in homebrew, but they prefer to have clearly defined versions.

Could you please tag a release?
Or, it need more work to stable and it cannot to tag a stable release for now?

Packing on engine side.

Now, each input plugin packs their data.
I want to change it.

New packing method

Input plugin passes a input context,data,size and data format to new flb_input API. (not use msgpack)
Engine enqueues and packs them in a msgpack of each context.

Why ?

Engine will be able to add a data to msgpack before flush.

Usecase

I want to add unique ID(like MAC address, hostname, UUID, etc) to msgpack.
If thousands of fluent-bit send data to one fluentd, user wants to know which fluent-bit sent the data.

ID doesn't depend on input plugin, but we add codes to all input plugins.
However, using new method, we only add them to engine.

It is useful to another data which doesn't depend on input plugin. (e.g timestamp)

Add out_kafka output

Could be based on https://github.com/edenhill/librdkafka, payload should be JSON or MessagePack.

[question] Error code of fluent-bit

In flb_error.h, some defines are octal (not decimal)
Is it intentional?
If not, I think there is a risk of overlap in the future.

Note:
As you know, the number which starts '0' is octal in C language.
For example, this value is 8 (not 10).

#define FLB_ERR_CFG_FILE             010

Add in_forward plugin

Please. So it can be used as a lightweight agent to collect from docker containers for example.

Broken link on fluentbit.io

http://fluentbit.io/news/ is not found.
It is linked from http://fluentbit.io/

Segfault which is caused by flb_tls_context_destroy

flb_test_elasticsearch is not finished successfully.

At flb_output_init(), a variable p->flags is checked like bitmap.

        if (p->flags & FLB_IO_TLS) {
            ins->tls.context = flb_tls_context_new(ins->tls_verify,

On the other hand, FLB_IO_TLS is defined like enum at flb_io.h.

/* Network operation modes */
#define FLB_IO_TCP         1  /* use plain TCP                          */
#define FLB_IO_TLS         2  /* use TLS/SSL layer                      */
#define FLB_IO_OPT_TLS     3  /* use TCP and optional TLS               */
#define FLB_IO_ASYNC       4  /* use async mode (depends on event loop) */

Which is proper ?
Now, if (p->flags & FLB_IO_TLS) is true when p->flags is set FLB_IO_OPT_TLS (out_es sets like this).

Mistakes in copyright file

I read fluent-bit/debian/copyright file.

License: Apache-2.0
Licensed under the Apache License, Version 2.0 (the "License");

...

it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
On Debian systems, the complete text of the GNU General
Public License version 2 can be found in "/usr/share/common-licenses/Apache-2.0".

Hmm..., This sentence is funny.
GNU GPL is not Apache License 2.0
And, it was included in Apace License Section.

You should correct this file.

I suggest you inspect other copyright and license files.

High resolution interval support

We can pass a argument 'nanoseconds' to flb_input_set_collector_time().
But the value seems not to be used.(Only 'seconds' is used.)

I wrote a patch and it seemed to work.
But I modified some lib/monkey files which come from Monkey project.

# Changed but not updated:
...
#   modified:   ../lib/monkey/mk_core/include/mk_core/mk_event.h
#   modified:   ../lib/monkey/mk_core/mk_event.c
#   modified:   ../lib/monkey/mk_core/mk_event_epoll.c
#   modified:   ../src/flb_engine.c

So, which do you prefer?

Pull Request to fluent-bit.
Pull Request to Monkey Project. (I have never used , so I can't test it.)
Do nothing. Someone will fix. :)

Thanks.

missing "data/json_es.h"

flb_test_elasticsearch.cpp includes "data/json_es.h".
However that header file is not found.

Msgpack buffer is leaked with retry

Sbuffer of out_forward seems to be leaked when fluent-bit can't communicate with server.
This issue is caused by cb_forward_flush.

In some cases (=FLB_OUTPUT_RETURN(FLB_RETRY)), sbuffer is not destroyed.
Will it be released when fluent-bit will be able to communicate with server?

How to reproduce.

$ cmake .. -DFLB_MTRACE=yes -DFLB_DEBUG=yes && make
$ MALLOC_TRACE=hoge.txt bin/fluent-bit -i cpu -o forward -f 1
(wait a minute and Ctrl+C)
$ mtrace bin/fluent-bit hoge.txt

mtrace log

mtrace reports so many sbuffer are leaked.

$ mtrace bin/fluent-bit hoge.txt 

Memory not freed:
-----------------
           Address     Size     Caller
0x00002b93c00008c0     0x15  at 0x3a32217cd2
0x00002b93c00008e0    0x496  at 0x3a3220aebf
0x00002b93c0000d80     0x15  at 0x3a3220b174
0x00002b93c0000da0     0x38  at 0x3a3220cd5d
0x00002b93c0000de0    0x138  at 0x3a3220ff34
0x0000000000db2150    0x240  at 0x3a32211a83
0x0000000000db23a0      0xf  at 0x3a32681022
0x0000000000db23c0     0x14  at 0x3a3269cf82
0x0000000000db23e0     0x14  at 0x3a3269cf82
0x0000000000db2400     0x15  at 0x3a3269cf82
0x0000000000db2420     0x14  at 0x3a3269cf82
0x0000000000db2600     0xd6  at 0x3a3269e91c
0x0000000000db6500        0  at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/include/fluent-bit/flb_mem.h:53
0x0000000000db6810   0x2000  at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/lib/msgpack-c-068041f/include/msgpack/sbuffer.h:77
0x0000000000db8820   0x2000  at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/lib/msgpack-c-068041f/include/msgpack/sbuffer.h:77
.
.
0x0000000000fa9b50   0x2000  at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/lib/msgpack-c-068041f/include/msgpack/sbuffer.h:77

in_stdout: overflow when reading in_mem generated msgpack maps

$ bin/fluent-bit -i mem -o stdout

[2015/06/02 18:41:02] [ info] starting engine
[0] {"time"=>1433292063, "total"=>8081592, "free"=>1595860}
[1] {"time"=>1433292064, "total"=>8081592, "free"=>1596028}
[2] {"time"=>1433292065, "total"=>8081592, "free"=>1595900}
[3] {"time"=>1433292066, "total"=>8081592, "free"=>1600272}
[4] 0
[5] 0
[6] 0
[7] 0
[8] 0
[9] 0
[10] 0
[11] 0
[12] 33
[13] 1
[14] 0
[15] 0
[16] 0
[17] 0
[18] 0
[19] 0
[20] 104
[21] -19
[22] 86
[23] "d^?^@^@hVd^?^@^@^@"
[24] 0
[25] 0
[26] 0
[27] 0
[28] 0

init scripts for packaged versions

Fluent Bit when installed from a package (RPM/Deb) should include an init script and a basic configuration file. By default do nothing until the config changes and the service is restarted, it must support:

systemd
legacy init.d script

in_cpu outputs one time only

in_cpu outputs records one time only.
A variable flush_done causes this issue.

$ bin/fluent-bit -i cpu -o stdout -vvv
Fluent-Bit v0.9.0
Copyright (C) Treasure Data

[2016/08/22 21:15:11] [ info] [engine] started
[2016/08/22 21:15:11] [debug] [router] default match rule cpu.0:stdout.0
[2016/08/22 21:15:12] [trace] [in_cpu] CPU 8.00%
[2016/08/22 21:15:13] [trace] [in_cpu] CPU 11.00%
[2016/08/22 21:15:14] [trace] [in_cpu] CPU 8.00%
[2016/08/22 21:15:15] [trace] [in_cpu] CPU 2.00%
[2016/08/22 21:15:16] [trace] [engine dispatch] task created 0x1d166b0
[2016/08/22 21:15:16] [trace] [thread 0x1d16780] created
[0] cpu.0: [1471868112, {"cpu_p"=>8.000000, "user_p"=>2.000000, "system_p"=>6.000000, "cpu0.p_cpu"=>8.000000, "cpu0.p_user"=>2.000000, "cpu0.p_system"=>6.000000}]
[1] cpu.0: [1471868113, {"cpu_p"=>11.000000, "user_p"=>4.000000, "system_p"=>7.000000, "cpu0.p_cpu"=>11.000000, "cpu0.p_user"=>4.000000, "cpu0.p_system"=>7.000000}]
[2] cpu.0: [1471868114, {"cpu_p"=>8.000000, "user_p"=>1.000000, "system_p"=>7.000000, "cpu0.p_cpu"=>8.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>7.000000}]
[3] cpu.0: [1471868115, {"cpu_p"=>2.000000, "user_p"=>1.000000, "system_p"=>1.000000, "cpu0.p_cpu"=>2.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>1.000000}]
[2016/08/22 21:15:16] [trace] [engine] [task event] task_id=0 thread_id=0 return=OK
[2016/08/22 21:15:16] [trace] [thread] destroy thread_id=0
[2016/08/22 21:15:16] [trace] [engine] destroy task_id=0

(no output)

master repository is reverted

golang repository became master and some commits were lost.
https://github.com/fluent/fluent-bit/network

My forked repo may help us...
nokute78@73d9cc7

[MQTT] Packet flooding causes SIGSEGV

SIGSEGV is occured with this PoC.

PoC(shell script)

while :; do mosquitto_pub -m '{"key":"aaa"}' -t some/topic & done

Backtrace

pi@raspberrypi /tmp/fluent-bit/build $ gdb bin/fluent-bit
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /tmp/fluent-bit/build/bin/fluent-bit...done.
(gdb) run -i mqtt -o stdout
Starting program: /tmp/fluent-bit/build/bin/fluent-bit -i mqtt -o stdout
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Fluent-Bit v0.8.0
Copyright (C) Treasure Data

[2016/04/05 12:58:47] [ info] starting engine
[2016/04/05 12:58:47] [ info] MQTT Listen='0.0.0.0' TCP_Port=1883
[2016/04/05 12:58:47] [ info] [mqtt] binding 0.0.0.0:1883
[2016/04/05 12:58:52] [ warn] No input data
[2016/04/05 12:58:57] [ warn] No input data
[2016/04/05 12:59:02] [ warn] No input data
[2016/04/05 12:59:07] [ warn] No input data
[2016/04/05 12:59:12] [ warn] No input data
[2016/04/05 12:59:17] [ warn] No input data
[2016/04/05 12:59:22] [ warn] No input data

Program received signal SIGSEGV, Segmentation fault.
0x000a8ef0 in mk_event_add (loop=0x706f742f, fd=239, type=3, mask=1, data=0x1f7de8)
    at /tmp/fluent-bit/lib/monkey/mk_core/mk_event.c:88
88          ctx = loop->data;
(gdb) bt
#0  0x000a8ef0 in mk_event_add (loop=0x706f742f, fd=239, type=3, mask=1, data=0x1f7de8)
    at /tmp/fluent-bit/lib/monkey/mk_core/mk_event.c:88
#1  0x00026b38 in mqtt_conn_add (fd=239, ctx=0xd4690) at /tmp/fluent-bit/plugins/in_mqtt/mqtt_conn.c:103
#2  0x00026848 in in_mqtt_collect (config=0xd1008, in_context=0xd4690) at /tmp/fluent-bit/plugins/in_mqtt/mqtt.c:102
#3  0x0001b478 in flb_engine_handle_event (config=0xd1008, mask=1, fd=9) at /tmp/fluent-bit/src/flb_engine.c:223
#4  flb_engine_start (config=0xd1008) at /tmp/fluent-bit/src/flb_engine.c:358
#5  0x00017110 in main (argc=5, argv=0x7efff7f4) at /tmp/fluent-bit/src/fluent-bit.c:314
(gdb)

Here, 'loop' is broken by received data.
loop=0x706f742f and '0x706f742f' is 'pot/' in ASCII.
'pot/' comes from the argument of mosquitto_pub, some'/top'ic

$ mosquitto_pub -m '{"key":"aaa"}' -t some/topic

HTTP Input

support HTTP input, payload as JSON or MessagePack.

note: work in process.

add google cloud out plugin

Currently we have https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud for google cloud logging but It would be great if there was a fluent-bit plugin that could do the same

Scheduler: improve retries handling

On a high load test, if many retries are issued, the Scheduler creates a timer file descriptor for each one, but easily these retries could not be scheduled at some point if we run out of file descriptors.

[2016/11/02 10:41:03] [  Error] timerfd: Too many open files, errno=24 at /home/edsiper/coding/fluent-bit/lib/monkey/mk_core/mk_event_epoll.c:175

Possible workarounds:

Let the user increase their file descriptor limits.
Do not use a file descriptor for each retry, instead lookup a list of retries every second using just one timer.

The approach 1 will let the user same the issue before take some action, note that using timerfd() is good as we just get notifications upon a timeout, we don't query a list.

Approach 2 add some overhead as we need to iterate a list every second.

This needs more thinking...

Build error on FreeBSD

I found this issue when I tested #48.
My environment is FreeBSD 10.2 , ~~gcc 4.8.5.~~ clang 3.4.1.

There are 2 issues.
I want to know how we fix issue1

1. unknown type name 'ucontext_t'

This is compatibility issue.

This is caused these codes at lib/monkey/mk_core/mk_thread.c.
(Monkey project has same issue.)

#if defined (__linux__)
#include <ucontext.h>
#elif defined (__APPLE__)
#include <sys/ucontext.h>
#endif

There is no 'else' case.
It is very simple to replace #elif to #else or add FreeBSD define.

ucontext.h may be obsolete.

On the other hand, I found these articles.

That header is removed on 'POSIX issue7' .
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap13.html

OpenBSD doesn't have ucontext.h
https://groups.google.com/forum/#!topic/v8-dev/seLPS6grC2A

They recommended to replace these API to POSIX threads.
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xsh_chap03.html

getcontext(), makecontext(), swapcontext()
Due to portability issues with these functions, especially with the manipulation of contexts, applications are recommended to be rewritten to use POSIX threads.

Build log

[  3%] Built target msgpack-static
[  4%] Built target jsmn
[  4%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_thread.c.o
fluent-bit/lib/monkey/mk_core/mk_thread.c:60:5: error: unknown type name 'ucontext_t'
    ucontext_t context;
    ^
fluent-bit/lib/monkey/mk_core/mk_thread.c:72:5: error: unknown type name 'ucontext_t'
    ucontext_t main;
    ^
fluent-bit/lib/monkey/mk_core/mk_thread.c:258:5: warning: implicit declaration of function 'swapcontext' is invalid in C99 [-Wimplicit-function-declaration]
    swapcontext(&dt->context, &sch->main);
    ^
fluent-bit/lib/monkey/mk_core/mk_thread.c:287:9: warning: implicit declaration of function 'getcontext' is invalid in C99 [-Wimplicit-function-declaration]
        getcontext(&dt->context);
        ^
fluent-bit/lib/monkey/mk_core/mk_thread.c:287:9: warning: declaration of built-in function 'getcontext' requires inclusion of the header <ucontext.h> [-Wbuiltin-requires-header]
fluent-bit/lib/monkey/mk_core/mk_thread.c:299:9: warning: implicit declaration of function 'makecontext' is invalid in C99 [-Wimplicit-function-declaration]
        makecontext(&dt->context, (void (*)(void))_mk_thread_entry_point, 1, sch);
        ^
4 warnings and 2 errors generated.
*** Error code 1

Stop.
make[2]: stopped in fluent-bit/build
*** Error code 1

Stop.
make[1]: stopped in fluent-bit/build
*** Error code 1

Stop.
make: stopped in fluent-bit/build

Segfault caused by retry

How to reproduce

$ bin/fluent-bit -i cpu -o forward -f 1
(wait a minute)

This issue also happens with ~~in_http~~ out_http.
The point is "offline" not to communicate fluentd/http server.

gdb log

out_th is NULL when segfault is happened.

src/flb_task.c:81
81	    o_ins = out_th->o_ins;

$ gdb bin/fluent-bit 
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-90.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/build/bin/fluent-bit...done.
(gdb) run -i cpu -o http -vvv -f 1
Starting program: /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/build/bin/fluent-bit -i cpu -o http -vvv -f 1
[Thread debugging using libthread_db enabled]
[New Thread 0x2aaaaacc1700 (LWP 28315)]
Fluent-Bit v0.9.0
Copyright (C) Treasure Data

[2016/10/31 22:14:01] [ info] [engine] started
[2016/10/31 22:14:01] [error] [TLS] Invalid CA file: /etc/ssl/certs/ca-certificates.crt
[2016/10/31 22:14:01] [debug] [router] default match rule cpu.0:http.0
[2016/10/31 22:14:02] [trace] [in_cpu] CPU 1.00%
[2016/10/31 22:14:03] [debug] [task] created task=0x6ea280 OK
[2016/10/31 22:14:03] [trace] [engine dispatch] task #0 created 0x6ea280
[2016/10/31 22:14:03] [trace] [thread 0x6ea320] created (custom data at 0x6ea348, size=64
[2016/10/31 22:14:03] [trace] [upstream] connection in process
[2016/10/31 22:14:03] [trace] [in_cpu] CPU 5.00%
[2016/10/31 22:14:03] [trace] [engine] resuming thread=0x6ea320
[2016/10/31 22:14:03] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:03] [error] [out_http] no upstream connections available
[2016/10/31 22:14:03] [trace] [engine] [task event] task_id=0 thread_id=0 return=RETRY
[2016/10/31 22:14:03] [trace] [thread] destroy thread=0x6ea320
[2016/10/31 22:14:03] [debug] [sched] retry=0x6f2620 0 in 6 seconds
[2016/10/31 22:14:04] [debug] [task] created task=0x6f0610 OK
[2016/10/31 22:14:04] [trace] [engine dispatch] task #1 created 0x6f0610
[2016/10/31 22:14:04] [trace] [thread 0x6f05a0] created (custom data at 0x6f05c8, size=64
[2016/10/31 22:14:04] [trace] [upstream] connection in process
[2016/10/31 22:14:04] [trace] [in_cpu] CPU 6.00%
[2016/10/31 22:14:04] [trace] [engine] resuming thread=0x6f05a0
[2016/10/31 22:14:04] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:04] [error] [out_http] no upstream connections available
[2016/10/31 22:14:04] [trace] [engine] [task event] task_id=1 thread_id=0 return=RETRY
[2016/10/31 22:14:04] [trace] [thread] destroy thread=0x6f05a0
[2016/10/31 22:14:04] [debug] [sched] retry=0x6f0740 1 in 2 seconds
[2016/10/31 22:14:05] [debug] [task] created task=0x6f0800 OK
[2016/10/31 22:14:05] [trace] [engine dispatch] task #2 created 0x6f0800
[2016/10/31 22:14:05] [trace] [thread 0x6f06d0] created (custom data at 0x6f06f8, size=64
[2016/10/31 22:14:05] [trace] [upstream] connection in process
[2016/10/31 22:14:05] [trace] [in_cpu] CPU 3.00%
[2016/10/31 22:14:05] [trace] [engine] resuming thread=0x6f06d0
[2016/10/31 22:14:05] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:05] [error] [out_http] no upstream connections available
[2016/10/31 22:14:05] [trace] [engine] [task event] task_id=2 thread_id=0 return=RETRY
[2016/10/31 22:14:05] [trace] [thread] destroy thread=0x6f06d0
[2016/10/31 22:14:05] [debug] [sched] retry=0x6f0950 2 in 5 seconds
[2016/10/31 22:14:06] [trace] [thread 0x6f08e0] created (custom data at 0x6f0908, size=64
[2016/10/31 22:14:06] [trace] [upstream] connection in process
[2016/10/31 22:14:06] [debug] [task] created task=0x6f0b50 OK
[2016/10/31 22:14:06] [trace] [engine dispatch] task #3 created 0x6f0b50
[2016/10/31 22:14:06] [trace] [thread 0x6f05a0] created (custom data at 0x6f05c8, size=64
[2016/10/31 22:14:06] [trace] [upstream] connection in process
[2016/10/31 22:14:06] [trace] [in_cpu] CPU 3.00%
[2016/10/31 22:14:06] [trace] [engine] resuming thread=0x6f08e0
[2016/10/31 22:14:06] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:06] [error] [out_http] no upstream connections available
[2016/10/31 22:14:06] [trace] [engine] resuming thread=0x6f05a0
[2016/10/31 22:14:06] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:06] [error] [out_http] no upstream connections available
[2016/10/31 22:14:06] [trace] [engine] [task event] task_id=1 thread_id=1 return=RETRY
[2016/10/31 22:14:06] [trace] [thread] destroy thread=0x6f08e0
[2016/10/31 22:14:06] [debug] [sched] retry=0x6f0740 1 in 20 seconds
[2016/10/31 22:14:06] [trace] [engine] [task event] task_id=3 thread_id=0 return=RETRY
[2016/10/31 22:14:06] [trace] [thread] destroy thread=0x6f05a0
[2016/10/31 22:14:06] [debug] [sched] retry=0x6f0a60 3 in 9 seconds

Program received signal SIGSEGV, Segmentation fault.
0x000000000041a615 in flb_task_retry_create (task=0x6ea280, data=0x0) at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/src/flb_task.c:81
81	    o_ins = out_th->o_ins;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.192.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb)

fluent / fluent-bit Goto Github PK

fluent-bit's Introduction

CI Status

About

Features

Fluent Bit in Production

Quick Start

Build from Scratch

Requirements

Linux Packages

Linux / Docker Container Images

Windows Packages

Running on s390x

Plugins: Inputs, Filters and Outputs

Input Plugins

Filter Plugins

Output Plugins

Contributing

Community & Contact

License

Authors

fluent-bit's People

Contributors

Stargazers

Watchers

Forkers

fluent-bit's Issues

no options for filter plugin

parse error at set_rules

How to reproduce

Note

Log

How to reproduce

gdb log

New packing method

Why ?

Usecase

How to reproduce.

mtrace log

1. unknown type name 'ucontext_t'

ucontext.h may be obsolete.

Build log

How to reproduce

gdb log

Recommend Projects

Recommend Topics

Recommend Org