saaras-io / yastack Goto Github PK

YAStack: User-space network-stack based on DPDK, FreeBSD TCP/IP Stack, EnvoyProxy

License: Apache License 2.0

CMake 0.19% Makefile 0.41% Meson 0.06% C 83.58% Python 0.58% C++ 13.09% Shell 0.19% Objective-C 0.56% PureBasic 0.01% Batchfile 0.01% PowerShell 0.01% CSS 0.01% Go 0.01% Thrift 0.01% Emacs Lisp 0.01% Roff 0.48% Ruby 0.01% M4 0.04% sed 0.01% Assembly 0.79%

networking networking-stack tcpip network-stack envoyproxy envoy-proxy dpdk dpdk-driver proxy proxy-server

yastack's Introduction

yastack

YAStack brings the speed of F-stack to envoy proxy.

Envoy's modular C++ codebase, clear separation of control plane and its filter architecture provides a way to quickly add new functionality. F-stack's freebsd stack provides the ability to perform packet processing from L2-L6. This integration provides for a high-speed L2-L7 stack for use in performance critial applications. Since the network stack resides in the user-space, it allows for quick iterations without modifying kernel code.

Envoy uses libevent to collect OS events. Libevent integration with f-stack thus provides a way for other applications that use libevent to integrate with this work. Details of libevent integration can be found under fs/app/libevent

DPDK integration generally uses Receive Side Scaling (RSS) to scale flow across cores. This needs hardware support. Yastack implementation can work without RSS hardware support. The flow distribution logic is encoded in software for better control. Setting of a newly introduced flag allows for the flow distribution logic in software to take over. This also eases the development process since it removes the dependency on hardware during development.

Software versions used in this project:

  Envoy Proxy - 1.8.0
  DPDK        - 18.0
  F-Stack     - fs/last_commit.txt

Blogs and additional documentation at - yastack.io

Quick Start

yastack's People

Contributors

Stargazers

Watchers

yastack's Issues

Test KNI support in YAStack

For machines that have just one interface, it needs to be shared between management and data traffic. When DPDK takes over the interface using (rte_kni driver), we can define the packets that should be sent to the linux kernel.

This support for KNI is already programmed in f-stack and can be activated and programmed from the dpdk ini file.

Attempt running YAStack from within a docker container

Run a DPDK application from inside docker. Determine filesystem mounts and running container in privileged mode.

yastack benchmark

I don't know if there is a problem with my configuration. The performance of yastack is much worse than nginx

Traffic Path：

wrk -> envoy(f-stack) -> nginx
wrk -> nginx(linux kernel) -> nginx

Modify code, always use f-stack socket

diff --git a/ev/source/common/network/address_impl.cc b/ev/source/common/network/address_impl.cc
index a7db10f..96dfc2c 100644
--- a/ev/source/common/network/address_impl.cc
+++ b/ev/source/common/network/address_impl.cc
@@ -194,20 +194,9 @@ int64_t InstanceBase::socketFromSocketType(SocketType socketType) const {
       domain = AF_INET;
     }
     int64_t fd;
-    if (likely(provider_ == Envoy::Network::Address::SocketProvider::Fp)) {
-        // Take over only network sockets
-           // FP non-blocking socket
-        SET_FP_NON_BLOCKING(flags);
-        fd = ff_socket(domain, flags, 0);
-        SET_FP_SOCKET(fd);
-        // TODO: Do we need this?
-        //RELEASE_ASSERT(ff_fcntl(fd, F_SETFL, O_NONBLOCK) != -1, "");
-    } else {
-           // Linux non-blocking socket
-        SET_HOST_NON_BLOCKING(flags);
-        fd = ::socket(domain, flags, 0);
-        RELEASE_ASSERT(fcntl(fd, F_SETFL, O_NONBLOCK) != -1, "");
-    }
+    SET_FP_NON_BLOCKING(flags);
+    fd = ff_socket(domain, flags, 0);
+    SET_FP_SOCKET(fd);
     return fd;
   } else {
     ASSERT(type() == Type::Pipe);

envoy config file:

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9995, provider: HOST }

static_resources:
  listeners:
  - name: listener_0
    address:
        socket_address: { address: 0.0.0.0, port_value: 10000, provider: FP}
    filter_chains:
      filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route: { cluster: service_local}
          http_filters:
          - name: envoy.router
  clusters:
  - name: service_local
    connect_timeout: 0.25s
    type: STATIC
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts: [ { socket_address: { address: 10.182.2.88, port_value: 8090, provider: FP}}]

f-stack config file:

[dpdk]
lcore_mask=1
channel=4
promiscuous=1
numa_on=1
tso=0
vlan_strip=1
port_list=0

[port0]
addr=10.182.2.69
netmask=255.255.252.0
broadcast=10.182.3.255
gateway=10.182.0.1
lcore_list=0

nginx use kernel network stack, config file:

worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;
    keepalive_requests 1000000;
    upstream myupstream {
        server 10.182.2.88:8090;
        keepalive 100;
    }

    server {
        listen       9999 reuseport;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
                proxy_http_version 1.1;
                proxy_set_header Connection "";
                proxy_pass http://myupstream;
        }

test result:

envoy

taskset -c  15-50 wrk -c 100 -d 2m -t20 'http://10.182.2.69:10000/' -H 'Connection: Keep-Alive'                                                                                                                                              
Running 2m test @ http://10.182.2.69:10000/
  20 threads and 100 connections


  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    11.61ms    7.77ms  40.59ms   67.28%
    Req/Sec   436.19     42.19   590.00     70.73%
  1042807 requests in 2.00m, 148.36MB read
Requests/sec:   8683.60
Transfer/sec:      1.24MB

nginx

taskset -c  15-50 wrk -c 100 -d 2m -t30 'http://10.182.2.68:9999/' -H 'Connection: Keep-Alive'                                                                                                                                               
Running 2m test @ http://10.182.2.68:9999/
  30 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.41ms  192.00us  42.99ms   99.36%
    Req/Sec     1.25k    29.55     3.62k    81.31%
  4479077 requests in 2.00m, 627.92MB read
Requests/sec:  37306.43
Transfer/sec:      5.23MB

How to run yastack/envoy-proxy as transparent forward proxy

Hi there,

We have transparent HTTP forward proxy which runs on standard Linux. However, we are trying to find more performant alternatives.

We would like to test/benchmark the yastack/envoy-proxy when running as transparent forward proxy. We would like to test the simplest possible configuration to see the raw performance without any additional features turned ON.

So, here are my questions:

Is it possible to run yastack/envoy-proxy in such mode?
Can you suggest the best configuration for such scenario?
Do you have some observations how much traffic or requests per second the yastack/envoy-proxy can handle and with what configuration?

Thanks,
Pavel.

Pre-compile YAStack tree in docker image

This is an optimization to reduce the time it takes to build/deploy YAStack.

Compilation of YAStack can be optimized by pre-compiling source code that does not depend on DPDK. This can help build the image faster since now only DPDK and dependent code will have to be compiled.

envoy is on bazel, but yastack ev is on cmake

Are there any specific reasons for yastack to use cmake to build envoy?

The official envoy code base uses bazel instead. There will be a lot of works and efforts in order to port the latest official envoy code to yastack because the two use different build systems.

Even though there's a tool on github to convert bazel to cmake, but it's not usable as it's limited supported.

So, what's the proper way for porting official envoy code into yastack? Any comments?

Utility to display NIC information

DPDK configuration file holds information about the underlying platform in addition to the NIC configuration - eg: number of cores, configuration about a NIC. This information can be read from underlying platform metadata

eg:
curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/02:29:96:8f:6a:2d/subnet-id

Right now, if a machine has two interfaces (eth0, eth1), we figure out the configuration for eth1 (eg: IP/Netmask/Broadcast/Gateway). However a tool that reads the metadata to infer this information can help.

libevent not receiving listen callbacks

I opened a listening port on my custom proxy through libevent API using evconnlistener_new. I issued a curl request from the client through the proxy port. I see that F stack receives the HTTP connect packet. But libevent does not receive this. Libevent is using epoll. Do you see any reason why this would not work?