thombashi / tcconfig Goto Github PK

A tc command wrapper. Make it easy to set up traffic control of network bandwidth/latency/packet-loss/packet-corruption/etc. to a network-interface/Docker-container(veth).

Home Page: https://tcconfig.rtfd.io/

License: MIT License

Python 98.40% Shell 1.14% Makefile 0.46%

tc traffic-shaping network wrapper cli-app docker linux

tcconfig's Introduction

tcconfig

Summary

tcconfig is a tc command wrapper. Make it easy to set up traffic control of network bandwidth/latency/packet-loss/packet-corruption/etc. to a network-interface/Docker-container(veth).

Traffic control

Setup traffic shaping rules

Easy to apply traffic shaping rules to specific networks:

Outgoing/Incoming packets
Source/Destination IP-address/network (IPv4/IPv6)
Source/Destination ports

Available parameters

The following parameters can be set to network interfaces:

Network bandwidth rate [G/M/K bps]
Network latency [microseconds/milliseconds/seconds/minutes]
Packet loss rate [%]
Packet corruption rate [%]
Packet duplicate rate [%]
Packet reordering rate [%]

Targets

Network interfaces: e.g. eth0
Docker container (veth corresponding with a container)

Usage

Set traffic control (`tcset` command)

tcset is a command to add a traffic control rule to a network interface (device).

e.g. Set a limit on bandwidth up to 100Kbps

# tcset eth0 --rate 100Kbps

e.g. Set network latency

You can use time units (such as us/sec/min/etc.) to designate delay time.

Set 100 milliseconds of network latency

# tcset eth0 --delay 100ms

Set 10 seconds of network latency

# tcset eth0 --delay 10sec

Set 0.5 minutes (30 seconds) network latency

# tcset eth0 --delay 0.5min

You can also use the following time units:

Unit	Available specifiers (str)
hours	`h`/`hour`/`hours`
minutes	`m`/`min`/`mins`/`minute`/`minutes`
seconds	`s`/`sec`/`secs`/`second`/`seconds`
milliseconds	`ms`/`msec`/`msecs`/`millisecond`/`milliseconds`
microseconds	`us`/`usec`/`usecs`/`microsecond`/`microseconds`

e.g. Set 0.1% packet loss

# tcset eth0 --loss 0.1%

e.g. All of the above settings at once

# tcset eth0 --rate 100Kbps --delay 100ms --loss 0.1%

e.g. Specify the IP address of the traffic control

# tcset eth0 --delay 100ms --network 192.168.0.10

e.g. Specify the IP network and port of traffic control

# tcset eth0 --delay 100ms --network 192.168.0.0/24 --port 80

Set traffic control to a docker container

Execute tcconfig with --docker option on a Docker host:

# tcset <container name or ID> --docker ...

You could use --src-container/--dst-container options to specify the source/destination container.

Set traffic control within a docker container

You need to run a container with --cap-add NET_ADMIN option if you would like to set a tc rule within a container:

docker run -d --cap-add NET_ADMIN -t <docker image>

A container image that builtin tcconfig can be available at https://hub.docker.com/r/thombashi/tcconfig/

Delete traffic control (`tcdel` command)

tcdel is a command to delete traffic shaping rules from a network interface (device).

e.g. Delete traffic control of `eth0`

You can delete all of the shaping rules for the eth0 with -a/--all option:

# tcdel eth0 --all

Display traffic control configurations (`tcshow` command)

tcshow is a command to display the current traffic control settings for network interface(s).

Example

# tcset eth0 --delay 10ms --delay-distro 2  --loss 0.01% --rate 0.25Mbps --network 192.168.0.10 --port 8080
# tcset eth0 --delay 1ms --loss 0.02% --rate 500Kbps --direction incoming
# tcshow eth0
{
    "eth0": {
        "outgoing": {
            "dst-network=192.168.0.10/32, dst-port=8080, protocol=ip": {
                "filter_id": "800::800",
                "delay": "10.0ms",
                "delay-distro": "2.0ms",
                "loss": "0.01%",
                "rate": "250Kbps"
            }
        },
        "incoming": {
            "protocol=ip": {
                "filter_id": "800::800",
                "delay": "1.0ms",
                "loss": "0.02%",
                "rate": "500Kbps"
            }
        }
    }
}

For more information

More examples are available at https://tcconfig.rtfd.io/en/latest/pages/usage/index.html

Installation

Installation: pip

tcconfig can be installed from PyPI via pip (Python package manager) command.

sudo pip install tcconfig

Installation: dpkg (Debian/Ubuntu)

curl -sSL https://raw.githubusercontent.com/thombashi/tcconfig/master/scripts/installer.sh | sudo bash

Dependencies

Python 3.7+
Python package dependencies (automatically installed)

Linux packages

mandatory: required for tc command:
- Ubuntu/`Debian`: iproute2
- Fedora/`RHEL`: iproute-tc
optional: required when you use the --iptables option:
- iptables

Linux kernel module

sch_netem

Optional Python packages

Pygments

Documentation

https://tcconfig.rtfd.io/

Troubleshooting

https://tcconfig.rtfd.io/en/latest/pages/troubleshooting.html

Docker image

https://hub.docker.com/r/thombashi/tcconfig/

Sponsors

Become a sponsor

tcconfig's People

Contributors

Stargazers

Watchers

tcconfig's Issues

Unlimited thoughput during tcset

HI,
I am using tcset to limit the throughput of a connection while following a bandwidth profile. For this I use the tcset command every 30 seconds. However, I notice a spike in the available throughput right after/while the new limit is set. I assume that the previous limit is removed, before adding a new limit and between these there is a not insignificant amount of time when there is no limit.

Is there a possibility to avoid this period of unlimited bandwidth?

As you can see, the measured throughput (red) spikes after when the available throughput (green) has changed (@ 60, 150 and 210 seconds)

Internal Error while deleting rules

Sometimes I get this error when running sudo tcdel ens3 --all:

When I run the command again, it works.

[2187] INTERNAL ERROR: cannot create temporary directory!

Script's rules don't change class IDs.

tcset must change the class IDs when using the --add parameter. When running tcset commands, this works:

USER@MACHINE ~/ $ tcshow eth5
{
    "eth5": {
        "outgoing": {},
        "incoming": {}
    }
}
USER@MACHINE ~/ $ sudo tcset --loss 4 --add --network 192.168.4.0/24 eth5
USER@MACHINE ~/ $ sudo tcset --loss 2 --add --network 192.168.2.0/24 eth5
USER@MACHINE ~/ $ tcshow eth5
{
    "eth5": {
        "outgoing": {
            "dst-network=192.168.4.0/24, protocol=ip": {
                "loss": 4,
                "rate": "32G",
                "filter_id": "800::800"
            },
            "dst-network=192.168.2.0/24, protocol=ip": {
                "loss": 2,
                "rate": "32G",
                "filter_id": "800::801"
            }
        },
        "incoming": {}
    }
}

As you can see above, tcshow shows that there are different rules for the % of dropped packets.

However, scripts generated with the --tc-script parameter don't do this:


 USER@MACHINE ~/ $ tcshow eth5
{
    "eth5": {
        "outgoing": {},
        "incoming": {}
    }
}
 USER@MACHINE ~/ $ sudo tcset --loss 2 --add --network 192.168.2.0/24 --tc-script eth5
[INFO] tcconfig: written a tc script to 'tcset_eth5.sh'
 USER@MACHINE ~/ $ mv tcset_eth5.sh tcset_2.sh
 USER@MACHINE ~/ $ sudo ./tcset_2.sh
 USER@MACHINE ~/ $ sudo tcset --loss 4 --add --network 192.168.4.0/24 --tc-script eth5
[INFO] tcconfig: written a tc script to 'tcset_eth5.sh'
 USER@MACHINE ~/ $ mv tcset_eth5.sh tcset_4.sh
 USER@MACHINE ~/ $ sudo ./tcset_4.sh 
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
 USER@MACHINE ~/ $ tcshow eth5
{
    "eth5": {
        "outgoing": {
            "dst-network=192.168.2.0/24, protocol=ip": {
                "filter_id": "800::800",
                "loss": 2,
                "rate": "32G"
            },
            "dst-network=192.168.4.0/24, protocol=ip": {
                "filter_id": "800::801",
                "loss": 2,
                "rate": "32G"
            }
        },
        "incoming": {}
    }
}

As you can see, both networks are set to a loss of 2% where 192.168.2.0/24 should be 2% and 192.168.4.0/24 should be 4%.

ImportError: No module named _threading

tcset --help
Traceback (most recent call last):
File "/usr/local/bin/tcset", line 9, in
load_entry_point('tcconfig==0.12.0', 'console_scripts', 'tcset')()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 337, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2279, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1989, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/usr/local/lib/python2.7/dist-packages/tcconfig/tcset.py", line 15, in
import logbook
File "/usr/local/lib/python2.7/dist-packages/logbook/init.py", line 14, in
from .base import (
File "/usr/local/lib/python2.7/dist-packages/logbook/base.py", line 19, in
from logbook.concurrency import (greenlet_get_ident, thread_get_ident,
File "/usr/local/lib/python2.7/dist-packages/logbook/concurrency.py", line 31, in
from gevent._threading import (Lock as ThreadLock,
ImportError: No module named _threading

Return value when tcset faild

When tcset fails because there is already a qdisc, the return value is zero, indicating succes.

$ sudo tcset --device lo --rate 10M; echo $?
[NOTICE] tcconfig: failed to add qdisc: qdisc already exists (dev lo, handle=17ce:, algo=htb). execute with --overwrite option if you want to overwrite the existing rules.execute with --add option if you want to add a new rule in addition to the existing rules.
0

Change or append/create rules

I use tcconfig on multiple hosts where I execute commands via Ansible, therefore, I do not store state. I would like to be able to override rules (on per network basis) without affecting the other rules and, if the rules do not exist, I would like to append (--add) them in the current list of traffic shaping rules. This could be done either by introducing a new option or changing the logic of --change option.

Thanks.

Plans to support traffic limit for a MAC address

Hi,

Great project and I see a lot of momentum on it right now. Is there any plans to support traffic control using device MAC address ?

Unable to Delete Shaping Rule by ID

wwtatc@ls-tj-test-01:/var/lab-services/ls-trafficjam-api$ tcdel --version
tcdel 0.22.1

wwtatc@ls-tj-test-01:/var/lab-services/ls-trafficjam-api$ tcshow ens192.100
{
    "ens192.100": {
        "outgoing": {
            "dst-network=10.1.0.0/24, dst-port=52, protocol=ip": {
                "filter_id": "800::800",
                "delay": "10.0ms",
                "delay-distro": "None",
                "loss": "None",
                "duplicate": "None",
                "corrupt": "None",
                "reorder": "None",
                "rate": "10Gbps"
            }
        },
        "incoming": {}
    }
}

wwtatc@ls-tj-test-01:/var/lab-services/ls-trafficjam-api$ sudo tcdel ens192.100 --id 800::800
[ERROR] tcconfig: shaping rule not found (device = 'ens192.100', protocol = 'ip', [dst-network] = '10.1.0.0/24', [src-network] = '0.0.0.0/0', [dst-port] = 52, [src-port] = 'None').
wwtatc@ls-tj-test-01:/var/lab-services/ls-trafficjam-api$

ipaddress package missing from requirements.txt

I installed via pip install tcconfig and got the error:

ImportError: No module named ipaddress

It's listed as a required package in the docs, but is missing from requirements.txt. Easily fixed, but unexpected :-)

AttributeError: 'module' object has no attribute 'logger'

I'm failing to run tcconfig with the following error message:

sudo tcset --device eno1 --delay 50 --direction incoming
Traceback (most recent call last):
  File "/usr/local/bin/tcset", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/tcconfig/tcset.py", line 176, in main
    subprocrunner.logger.level = options.log_level
AttributeError: 'module' object has no attribute 'logger'

I have tcconfig working on an almost identical box from OS and Python perspective. This is the non working one:

uname -a
Linux Lab-desktop 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocrunner
>>> dir(subprocrunner)
['CommandNotFoundError', 'InvalidCommandError', 'SubprocessRunner', 'Which', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '_error', '_logger', '_subprocess_runner', '_which', 'absolute_import', 'set_log_level', 'set_logger']
>>>

This is the working one:

uname -a
Linux daniel-tc 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocrunner
>>> dir(subprocrunner)
['CommandNotFoundError', 'InvalidCommandError', 'SubprocessRunner', 'Which', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '_error', '_subprocess_runner', '_which', 'absolute_import', 'logger']
>>>

On the non working one it seems logger is missing from the dir(subprocrunner) command.

Not sure if this is an issue with Python somehow or if something changed in tcconfig?

Set parameter from / to

Would be possible to add, as a parameter, the possibility to specify a rule only from a specific port/ip versus another specific port/ip?
For example, add a 500ms latency only from port 13000 to 14000.

Unexpected latency due to rate limitation

I got unexpected high latency when use "rate" to limit bandwidth.

tcset enp2s0 --rate 300K --loss 0 --delay 0ms --direction incoming --src-network 192.168.3.101

When host 192.168.3.101 tx-rate exceeds 300K the RTT is keeping grow up.
How can i make tcconfig to drop packets when tx-rate higher than rate limitation ?

Thanks a lot.

tcshow show incorrect limits when overwriting existing limit with tcset --add

Commands:

➜  ~ sudo tcset --device lo --rate 10M                
➜  ~ sudo tcset --device lo --rate 5M --add 
➜  ~ tcshow --device lo                 
{
    "lo": {
        "outgoing": {
            "network=0.0.0.0/0, protocol=ip": {
                "rate": "5M"
            }
        },
        "incoming": {}
    }
}

iperf3 keeps measuring the actual limit as 10 Mbps while tcshow shows 5 Mbps

Illegal "rate" when running tcset

I ran --device eth0 --delay 25 --rate 1M but encountered an error complaining about Illegal rate.
My computer uses Ubuntu 14.04 (kernel 4.4.0) with a gigabit ethernet.

[DEBUG] subprocrunner: tc class add dev eth0 parent 1a1a: classid 1a1a:1 htb rate 1000Gbit
[WARNING] subprocrunner: returncode=1, stderr=Illegal "rate"
Usage: ... qdisc add ... htb [default N] [r2q N]
[direct_qlen P]
default minor id of class to which unclassified packets are sent {0}
r2q DRR quantums are computed as rate in Bps/r2q {10}
debug string of 16 numbers each 0-3 {0}

direct_qlen Limit of the direct queue {in packets}
... class add ... htb rate R1 [burst B1] [mpu B] [overhead O]
[prio P] [slot S] [pslot PS]
[ceil R2] [cburst B2] [mtu MTU] [quantum Q]
rate rate allocated to this class (class can still borrow)
burst max bytes burst which can be accumulated during idle period {computed}
mpu minimum packet size used in rate computations
overhead per-packet size overhead used in rate computations
linklay adapting to a linklayer e.g. atm
ceil definite upper class rate (no borrows) {rate}
cburst burst but for ceil {computed}
mtu max packet size we create rate map for {1600}
prio priority of leaf; lower are served first {0}
quantum how much bytes to serve from leaf at once {use r2q}

TC HTB version 3.3

Fix: It seems that setting the unlimited rate as 1000Gbit is not good on my machine. Changing shaper/htb.py:HtbShaper's __NO_LIMIT = "1000G" to "1G" solved the problem for me.

backup and restore not working in 0.22.0 with python2

following https://tcconfig.readthedocs.io/en/latest/pages/usage/backup_and_restore.html

problem:

user@host:~$ sudo tcshow eth0
{
    "eth0": {
        "outgoing": {},
        "incoming": {}
    }
}

user@host:~$ sudo tcset eth0 --delay 10ms --direction incoming

user@host:~$ sudo tcshow eth0 > tcconfig.json

user@host:~$ cat tcconfig.json 
{
    "eth0": {
        "outgoing": {},
        "incoming": {
            "protocol=ip": {
                "delay": "10.0ms",
                "rate": "32Gbps",
                "filter_id": "800::800"
            }
        }
    }
}

user@host:~$ sudo tcset tcconfig.json --import-setting
Traceback (most recent call last):
  File "/usr/local/bin/tcset", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/tcconfig/tcset.py", line 346, in main
    return set_tc_from_file(logger, options.device, options.overwrite)
  File "/usr/local/lib/python2.7/dist-packages/tcconfig/_importer.py", line 139, in set_tc_from_file
    loader.load_tcconfig(config_file_path)
  File "/usr/local/lib/python2.7/dist-packages/tcconfig/_importer.py", line 43, in load_tcconfig
    schema(self.__config_table)
  File "/usr/local/lib/python2.7/dist-packages/voluptuous/schema_builder.py", line 267, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python2.7/dist-packages/voluptuous/schema_builder.py", line 589, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/usr/local/lib/python2.7/dist-packages/voluptuous/schema_builder.py", line 427, in validate_mapping
    raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: required key not provided @ data[<type 'unicode'>]

user@host:~$ pip list | grep tcconfig
tcconfig (0.22.0)

user@host:~$ pip list | grep voluptuous
voluptuous (0.11.5)

I can not see it with the tc command

I installed by pip.

I used tcset to do the following
$ tcset --device eth0 --rate 400k --src-network 10.211.55.22/32 --direction incoming --add

It works well.

However, I can not confirm anything using the tc command.
$ tc class show dev eth0
$ tc filter show dev eth0

And one more thing, if I set the --port to the incoming direction, it does not work well. what's the reason?

OS Version: CentOS7.2

I wonder why.
Thanks.

--tc-command doesn't work with --direction incoming

The error given when running these together is "[ERROR] tcconfig: network interface not found: ifb4623". I assume this is because of --tc-command's suppression of the commands to create the ifb interface.

ValueError exception on tcshow with empty qdisc configuration

/ # tc qdisc show dev eth0
/ # tcset --device eth0 --delay 100
/ # tcshow --device eth0
{
    "eth0": {
        "outgoing": {
            "network=0.0.0.0/0": {
                "delay": "100.0",
                "rate": "10G"
            }
        },
        "incoming": {}
    }
}
/ # tc qdisc show dev eth0
qdisc htb 1a1a: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 direct_qlen 2
qdisc netem 1a9a: parent 1a1a:2 limit 1000 delay 100.0ms
/ # tcdel --device eth0
[NOTICE] tcconfig: failed to delete qdisc: no qdisc for incomming packets
/ # tc qdisc show dev eth0
/ # tcshow --device eth0
Traceback (most recent call last):
  File "/usr/bin/tcshow", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/tcconfig/tcshow.py", line 213, in main
    tc_param.update(TcShapingRuleParser(device, logger).get_tc_parameter())
  File "/usr/lib/python2.7/site-packages/tcconfig/tcshow.py", line 74, in get_tc_parameter
    TrafficDirection.OUTGOING: self.__get_shaping_rule(self.device),
  File "/usr/lib/python2.7/site-packages/tcconfig/tcshow.py", line 126, in __get_shaping_rule
    qdisc_param_list = self.__parse_tc_qdisc(device)
  File "/usr/lib/python2.7/site-packages/tcconfig/tcshow.py", line 178, in __parse_tc_qdisc
    run_tc_show(Tc.Subcommand.QDISC, device)))
  File "/usr/lib/python2.7/site-packages/tcconfig/parser.py", line 201, in parse
    raise ValueError("empty text")
ValueError: empty text

It can happen that no qdisc is configured for a specific device and I would have expected that an empty JSON dict to be output.

Regards,

Emmanuel

How can I setup a default sharing rate?

I'm trying to limit traffic by destination ports.
However, for other applications, I'd like to put them into a default class, which has a very small rate to them.

I haven't found any arguments to set priority.

Is there any solution for this?

{
    "lo": {
        "incoming": {},
        "outgoing": {
            "dst-network=127.0.0.1/32, dst-port=3000, protocol=ip": {
                "rate": "1000M"
            },
            "dst-network=127.0.0.1/32, dst-port=5000, protocol=ip": {
                "rate": "2000M"
            },
            "protocol=ip": {
                "rate": "10M"
            }
        }
    }
}

For example, I wanna make the last rule to be the default rule.
All traffic not going to 127.0.0.1:3000/5000 should apply this rule.

qdisc classful type htb

your tcset command use htb qdisc to limit bandwidth, does it a first in first out algorithm.If not, how should I limit bandwidth like physical device limit? like connect two 100Mb PC with a 10Mb switch.

Error while deleting rules with --docker

I'm using tcconfig to manage docker container network interfaces. Every time I'm trying to delete previously set rules with --docker, I'm getting this error:

[ERROR] tcconfig: command execution failed
  command=/usr/sbin/tc qdisc del dev veth7f0e1f0 ingress
  stderr=Error: Cannot find specified qdisc on specified device.

[WARNING] tcconfig: [Errno 2] Bad argument `iptables'
Try `iptables -h' or 'iptables --help' for more information.
 (can not delete iptables entries)

The rules, however, cease to affect container.

How to reproduce:

sudo docker run -d alpine --name test_ping ping 8.8.8.8
sudo  tcset --delay 100ms --docker test_ping
sudo tcdel --docker test_ping

Multiple rules per interface

I would like to do something like this

tcset --device bond0.310 --rate 500M --network 192.168.2.0/24
tcset --device bond0.310 --rate 100M --network 192.168.0.0/24

The first one succeeds w/o issue but the second command fails. Is this not something tcconfig can do?

root@ubuntu:~# tcset --device bond0.310 --rate 500M --network 192.168.2.0/24
root@ubuntu:~# tcset --device bond0.310 --rate 100M --network 192.168.0.0/24
[2017-01-12 20:53:30.357743] NOTICE: subprocrunner: failed to add qdisc: prio qdisc already exists. execute with --overwrite option if you want to overwrite the existing settings.
[2017-01-12 20:53:30.367540] NOTICE: subprocrunner: failed to add qdisc: netem qdisc already exists. execute with --overwrite option if you want to overwrite the existing settings.
[2017-01-12 20:53:30.377499] WARNING: subprocrunner: returncode=2, stderr=RTNETLINK answers: File exists
root@ubuntu:~#

Add an extra check for capabilities when executing the commands

Besides the execution of the commands as a root user, you can also employ Linux capabilities and apply CAP_NET_ADMIN and CAP_NET_RAW to tc and iptables in order to use them as a regular user.

In your root check , you could implement an extra check for those two capabilities and permit the execution or else raise an Exception or print an informative error. Currently, I am unable to use tcconfig tools due to that check and I have to modify the code and run it again.

How to use capabilities

Get the capabilities of a command:

 getcap /sbin/tc

Set capabilities to a command:

setcap cap_net_raw,cap_net_admin+ep /sbin/tc

Clear rules from interface

I am wondering if there is any way to remove rules from an interface using tcset? I cannot seem to find anything in the docs. http://tcconfig.readthedocs.io/en/latest/pages/usage/tcset/index.html

Not looking to remove a matching rule but looking to clear all rules like a tcset --clear. Hopefully that makes sense. "Basically tc qdisc del dev root"

change argument alters other defined rules

According to the help page of tcset:

 --change               change existing traffic shaping rules to new one. this
                        option reduce the shaping rule switching side effect
                        (such as traffic spike) compared to --overwrite
                        option. note: the tcset command will fail when there
                        is no existing shaping rules.

How to reproduce

Set multiple rules:

$ sudo tcset --device eth0 --delay 100 --rate 50k --network 192.168.1.2 --add
$ sudo tcset --device eth0 --delay 100 --rate 50k --network 192.168.1.3 --add

Show the existing rules:

$ sudo tcshow --device eth0
{
    "eth0": {
        "outgoing": {
            "dst-network=192.168.1.2/32, protocol=ip": {
                "delay": "100.0",
                "rate": "50K"
            },
            "dst-network=192.168.1.3/32, protocol=ip": {
                "delay": "100.0",
                "rate": "50K"
            }
        },
        "incoming": {}
    }
}

If I want to change the delay and rate in the dst-network 192.168.1.3 to 200 and 60k respectively:

$ sudo tcset --device eth0 --delay 200 --rate 60k --network 192.168.1.3 --change

However the command changes the rule for the dst-network 192.168.1.2:

$ sudo tcshow --device eth0
{                                              
    "eth0": {                               
        "outgoing": {                          
            "dst-network=192.168.1.2/32, protocol=ip": {                                       
                "delay": "100.0",              
                "rate": "60K"                  
            },                                 
            "dst-network=192.168.1.3/32, protocol=ip": {                                       
                "delay": "100.0",              
                "rate": "50K"                  
            }                                  
        },                                     
        "incoming": {}                         
    }                                          
}

The command replaced only the rate attribute in the wrong network and the delay attribute was not changed at all.

Thanks in advance.

broken sqlite

➜  ~ sudo tcset tctest --docker --direction incoming --rate 100Kbps

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/simplesqlite/core.py", line 545, in insert_many
    self.connection.executemany(query, records)
sqlite3.IntegrityError: UNIQUE constraint failed: if_index.peer_ifindex

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tcset", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/tcconfig/tcset.py", line 347, in main
    return TcSetMain(options).run()
  File "/usr/local/lib/python3.6/dist-packages/tcconfig/tcset.py", line 227, in run
    for device in self._fetch_tc_target_list():
  File "/usr/local/lib/python3.6/dist-packages/tcconfig/_main.py", line 35, in _fetch_tc_target_list
    self._dclient.create_veth_table(container)
  File "/usr/local/lib/python3.6/dist-packages/tcconfig/_docker.py", line 139, in create_veth_table
    return_code = self.__create_ifindex_table(container_info.name)
  File "/usr/local/lib/python3.6/dist-packages/tcconfig/_docker.py", line 218, in __create_ifindex_table
    peer_ifindex=int(peer_ifindex_prefix_regexp.sub("", peer_ifindex)),
  File "/usr/local/lib/python3.6/dist-packages/simplesqlite/model.py", line 197, in insert
    cls.__connection.insert(cls.get_table_name(), record)
  File "/usr/local/lib/python3.6/dist-packages/simplesqlite/core.py", line 499, in insert
    self.insert_many(table_name, [record])
  File "/usr/local/lib/python3.6/dist-packages/simplesqlite/core.py", line 554, in insert_many
    + "  records={}\n".format(records[:2])
simplesqlite.error.OperationalError: /usr/local/lib/python3.6/dist-packages/simplesqlite/model.py(197) insert: failed to execute query:
  query=INSERT INTO if_index VALUES (?,?,?,?)
  msg='UNIQUE constraint failed: if_index.peer_ifindex'
  db=:memory:
  records=[['tazthink', 20, 'veth4e6461a7', 3]]

tcdel support for deletion per-network

Currently, tcdel supports deletion of all rules. However, it would be great if there were more fine grained options to delete rules based on the src-network, src-port, dst-network, dst-port etc.

Examples:

Deletion based on Individual IP:

 $ sudo tcdel --device eth0 --dst-network 192.168.1.2

Deletion based on CIDR block:

$ sudo tcdel --device eth0 --dst-network 192.168.1.2/24

Filter routes, not just networks

If a Linux box is acting as a router, it'd be helpful to be able to specify impairments for a specific route (i.e. all traffic from X.X.X.X to Y.Y.Y.Y, possibly with protocols or ports specified). How hard would this be to add?

Does tcconfig has any doc for its code achieve?

Hi,
You job is great,I want to learn how its code achieve limit bandwidth.It is hard to read code,does any doc for it.
thank you.

IPv6

It's telling me that it expects 4 octets. Does this mean that tcconfig does not support IPv6? or is there a flag I need to set.

tcconfig kali

I am newbie to network and firewall system.

I installed kali linux latest version.
it contains python3.7.2+

tcconfig was installed with pip3

When I run ' tcset ens3 --delay 10sec --rate 1bps --network <remote_ip_address> --direction incoming ' on console, I got errors as following.

[ERROR] tcconfig: command execution failed
command=/usr/sbin/tc class add dev ifb5634 parent 1602: classid 1602:2 htb rate 0.001Kbit ceil 0.001Kbit burst 1.25e-05KB cburst 1.25e-05KB
stderr="rate" is required.

[ERROR] tcconfig: command execution failed
command=/usr/sbin/tc qdisc add dev ifb5634 parent 1602:2 handle 2bb2: netem delay 10sec
stderr=Error: Specified class not found.

Could someone explain why the errors mean and how to fix it ?

confuse with docker rate limit

Hi sir,
I recently use tcconfig to set up some tc rules, but when I use --docker options(like this: tcset vpn_service --docker --rate 1mbps) the vpn_service docker might can't connect the internet. please help.
thanks a lot!

Traceback when using tcset with --change

To reproduce:

tcdel -d eth0
tcset --device eth0 --rate 10M --src-network 172.17.0.2 --direction outgoing
tcset --device eth0 --rate 100M --src-network 172.17.0.2 --direction outgoing --change

Output:

[DEBUG] subprocrunner: tc qdisc show dev eth0
Traceback (most recent call last):
  File "/usr/local/bin/tcset", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/tcconfig/tcset.py", line 432, in main
    return_code = tc.set_tc()
  File "/usr/local/lib/python3.6/site-packages/tcconfig/traffic_control.py", line 348, in set_tc
    return self.__shaper.set_shaping()
  File "/usr/local/lib/python3.6/site-packages/tcconfig/shaper/htb.py", line 211, in set_shaping
    self._set_netem()
  File "/usr/local/lib/python3.6/site-packages/tcconfig/shaper/_interface.py", line 91, in _set_netem
    "handle {:s}".format(handle),
ValueError: Unknown format code 's' for object of type 'int'

tcconfig version is 0.13.2
Although the traceback is produced the change still takes effect.

Thanks

Add SrcPort matching

Hi,
currently you only support matching on the destination port with --port.
How about adding a source port match as well?
Thank you.

Issue configuring bandwidth rate on a certain network interface

I would like to change the bandwidth rate in a certain network interface, however I get the same error output in all values that I have tried.

tcset --device ens3 --rate 1k
[ERROR] tcconfig: bandwidth_rate must be less than -1000: actual=1.0

tcset --device ens3 --rate 1Gbps
[ERROR] tcconfig: bandwidth_rate must be less than -1000: actual=1000000.0

tcset --device ens3 --rate 10Mbps
[ERROR] tcconfig: bandwidth_rate must be less than -1000: actual=10000.0

The output of /sys/class/net/ens3/speed is -1.

If I apply raw tc commands like the following, there are no error messages.

tc qdisc add dev ens3 handle 1: root htb default 11
tc class add dev ens3 parent 1: classid 1:1 htb rate 1000Mbps
tc class add dev ens3 parent 1:1 classid 1:11 htb rate 4Mbit
tc qdisc add dev ens3 parent 1:11 handle 10: netem delay 50ms

Update:

According to the ABI doc of the kernel (re: speed):

What: /sys/class/net/<iface>/speed
Description:
Indicates the interface latest or current speed value. Value is
an integer representing the link speed in Mbits/sec.
 
Note: this attribute is only valid for interfaces that implement
the ethtool get_settings method (mostly Ethernet ).

From the Linux kernel source code the __ethtool_get_link_ksettings function seems to fail and that's why the -1 is returned. My guess is that the virtio drivers are causing that confusion. Paravirtualised network interfaces have no speed because there is no speed limit.

So would it make sense to add a handling in the code to default to the iproute2 upper bound when it encounters a -1 speed?

Add support for network exclusion parameter in tcset

Hi,

It would be great to see an exclusion parameter in tcconfig which will exclude the rules to a specific network.

e.g., Apply the delay to everyone but the 192.16.0.10 network:

 # tcset --device eth0 --delay 100 --exclude-network 192.168.0.10

or to a specific CIDR:

 # tcset --device eth0 --delay 100 --exclude-network 192.168.0.10/24

Example of tc command for network exclusion of packets of 192.168.1.15 with destination port 9001:

tc filter add dev eth0 protocol ip parent 1:0 prio 3 u32 match ip src 0/0 flowid 1:3
tc filter add dev eth0 protocol ipv6 parent 1:0 prio 4 u32 match ip6 src ::/0 flowid 1:3

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip dport 22 0xffff flowid 1:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 22 0xffff flowid 1:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip src 192.168.1.15 match ip dport 9001 0xffff flowid 1:2

tc filter add dev eth0 protocol ipv6 parent 1:0 prio 2 u32 match ip6 dport 22 0xffff flowid 1:2
tc filter add dev eth0 protocol ipv6 parent 1:0 prio 2 u32 match ip6 sport 22 0xffff flowid 1:2
tc filter add dev eth0 protocol ipv6 parent 1:0 prio 2 u32 match ip6 dport 9001 0xffff flowid 1:2

Thanks,
Pavlos

how to get the tc command that tcconfig is creating?

I checked in documentation and I did not see it, probably is not possible (would be nice): a command that shows you what is the tc script is generating tcconfig. I find this useful in two cases: as a way to learn tc, and to debug this program

iptables rules for src network?

Is there a reason why you use iptables for src network when filter rules could do it instead?

Example

tc qdisc add dev bond0.310 root handle 1: prio bands 10
tc qdisc add dev bond0.310 parent 1:1 handle 10: netem delay 0ms
tc qdisc add dev bond0.310 parent 10:1 handle 11: netem loss 0%
tc qdisc add dev bond0.310 parent 11:1 handle 12: htb default 1
tc class add dev bond0.310 parent 12: classid 0:1 htb rate 200mbit ceil 200mbit
tc filter add dev bond0.310 protocol ip parent 1:0 prio 1 u32 match ip src 192.168.2.0/24 match ip dst 192.168.1.0/24 flowid 10:1

False positive errors when trying to add multiple rules

Hi,

When I try to add multiple incoming/outgoing rules to distinct IPs I get an error from tcconfig. However, the error seems to be false positive since when I use tcshow, it shows that the rules have been applied.

How to reproduce

List the rules for the network iface:

# tcshow -d ens3
{
    "ens3": {
        "outgoing": {},
        "incoming": {}
    }
}

Set outgoing/incoming rules for a specific dst-network:

# tcset --direction outgoing --dst-network 192.168.122.205 --rate 500Mbps --device ens3 --add; tcset --direction incoming --dst-network 192.168.122.205 --device ens3 --rate 500Mbps --add

List available rules:

{
    "ens3": {
        "outgoing": {
            "dst-network=192.168.122.205/32, protocol=ip": {
                "rate": "500M"
            }
        },
        "incoming": {
            "dst-network=192.168.122.205/32, protocol=ip": {
                "rate": "500M"
            }
        }
    }
}

Set new incoming/outgoing rules to another IP:

# tcset --direction outgoing --dst-network 192.168.122.201 --rate 500Mbps --device ens3 --add; tcset --direction incoming --dst-network 192.168.122.201 --device ens3 --rate 500Mbps --add
[NOTICE] tcconfig: failed to add ip link: ip link already exists. execute with --overwrite option if you want to overwrite the existing rules. execute with --add option if you want to add a new rule in addition to the existing rules.
[NOTICE] tcconfig: failed to 'tc qdisc add': ingress qdisc already exists. execute with --overwrite option if you want to overwrite the existing rules. execute with --add option if you want to add a new rule in addition to the existing rules.

However, if I list the available rules in the interface, it shows that the new rules have been applied successfully:

# tcshow -d ens3
{
    "ens3": {
        "outgoing": {
            "dst-network=192.168.122.201/32, protocol=ip": {
                "rate": "500M"
            },
            "dst-network=192.168.122.205/32, protocol=ip": {
                "rate": "500M"
            }
        },
        "incoming": {
            "dst-network=192.168.122.201/32, protocol=ip": {
                "rate": "500M"
            },
            "dst-network=192.168.122.205/32, protocol=ip": {
                "rate": "500M"
            }
        }
    }
}

Thanks in advance.

Plan to add support for direct_qlen attribute

Hi,

Any plan to support direct_qlen attribute?
TC command like this

tc qdisc add dev eth0 root handle 1: htb default 1 direct_qlen 512

Thanks.

[Documentation] Show two host examples

At the moment there are examples for applying traffic shaping rules to individual hosts. It would be great to see the documentation include some examples i.e., how to set/enforce bandwidth between two hosts.

api

How that can be fixed? =\

tcset focused_bartik --docker --direction incoming --rate 15Mbps
Traceback (most recent call last):
  File "site-packages/docker/api/client.py", line 256, in _raise_for_status
  File "site-packages/requests/models.py", line 940, in raise_for_status
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.35/containers/json?limit=-1&all=0&size=0&trunc_cmd=0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cli_tcset.py", line 14, in <module>
  File "tcconfig/tcset.py", line 347, in main
  File "tcconfig/tcset.py", line 227, in run
  File "tcconfig/_main.py", line 34, in _fetch_tc_target_list
  File "tcconfig/_docker.py", line 79, in verify_container
  File "tcconfig/_docker.py", line 59, in __verify_container
  File "site-packages/docker/api/container.py", line 208, in containers
  File "site-packages/docker/api/client.py", line 262, in _result
  File "site-packages/docker/api/client.py", line 258, in _raise_for_status
  File "site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
docker.errors.APIError: 400 Client Error: Bad Request ("client is newer than server (client API version: 1.35, server API version: 1.24)")
[189860] Failed to execute script cli_tcset

Support extra time units in delay specification

At the moment when using the --delay option in tcset, you can only set time to milliseconds. According to the tc man page , it supports microseconds and seconds too. It would be great to have support for those time units plus minutes which could be easily converted to one of the supported time units.

Examples:

e.g. Set 100ms network latency

# tcset --device eth0 --delay 100ms

e.g. Set 100s network latency

# tcset --device eth0 --delay 100s

e.g. Set 1m network latency

# tcset --device eth0 --delay 1m

Thanks,
Pavlos

Tcset fails to set bandwidth rate of 100 Mbit/s

When trying to set a rate of 100 Mbit/s on an interface, tcset fails with the following message:

Traceback (most recent call last): File "/usr/local/bin/tcset", line 11, in <module> sys.exit(main()) File "/usr/local/lib/python2.7/dist-packages/tcconfig/tcset.py", line 218, in main tc.set_tc() File "/usr/local/lib/python2.7/dist-packages/tcconfig/traffic_control.py", line 102, in set_tc self.__set_rate(qdisc_major_id) File "/usr/local/lib/python2.7/dist-packages/tcconfig/traffic_control.py", line 400, in __set_rate max(rate_kbps, self.__MIN_BUFFER_BYTE)), # [byte] ValueError: Unknown format code 'd' for object of type 'float'

When debug option is used the following commands are output by tcset:

[2016-11-11 09:47:09.580771] DEBUG: subprocrunner: lsmod | grep sch_netem [2016-11-11 09:47:09.585109] DEBUG: subprocrunner: tc qdisc add dev ens192.190 root handle 1040: prio [2016-11-11 09:47:09.587466] DEBUG: subprocrunner: tc filter add dev ens192.190 protocol ip parent 1040: prio 2 u32 match ip dst 0.0.0.0/0 flowid 1040:1 [2016-11-11 09:47:09.589509] DEBUG: subprocrunner: tc qdisc add dev ens192.190 parent 1040:1 handle 104a: netem

When the command is used successfully such as setting rate to 100 kbit/s the following commands are logged:

[2016-11-11 09:47:41.265752] DEBUG: subprocrunner: tc qdisc add dev ens192.190 root handle 1040: prio [2016-11-11 09:47:41.268064] WARNING: subprocrunner: returncode=2, stderr=RTNETLINK answers: File exists [2016-11-11 09:47:41.268512] DEBUG: subprocrunner: tc filter add dev ens192.190 protocol ip parent 1040: prio 2 u32 match ip dst 0.0.0.0/0 flowid 1040:1 [2016-11-11 09:47:41.270733] DEBUG: subprocrunner: tc qdisc add dev ens192.190 parent 1040:1 handle 104a: netem [2016-11-11 09:47:41.273082] WARNING: subprocrunner: returncode=2, stderr=RTNETLINK answers: File exists [2016-11-11 09:47:41.273447] DEBUG: subprocrunner: tc qdisc add dev ens192.190 parent 104a:1 handle 20: tbf rate 100.000000kbit buffer 1600 limit 10000

Is this a bug in tcset or is it an issue with tc itself?

reordering not working

CentOS Linux release 7.5.1804 (Core)
Python 2.7.5
tcconfig 0.20.0

tcset ens33 --add --reordering 99
Traceback (most recent call last):
File "/usr/bin/tcset", line 11, in
sys.exit(main())
File "/usr/lib/python2.7/site-packages/tcconfig/tcset.py", line 338, in main
return_code = tc.set_tc()
File "/usr/lib/python2.7/site-packages/tcconfig/traffic_control.py", line 282, in set_tc
return self.__shaper.set_shaping()
File "/usr/lib/python2.7/site-packages/tcconfig/shaper/htb.py", line 215, in set_shaping
self._set_netem()
File "/usr/lib/python2.7/site-packages/tcconfig/shaper/_interface.py", line 123, in _set_netem
handle=handle,
File "/usr/lib/python2.7/site-packages/tcconfig/_common.py", line 162, in run_command_helper
logger.error(error_msg_format.format(command, proc.stderr))
KeyError: u'uniform|normal|pareto|paretonormal'

Docs won't display correctly in PyPI

It looks like tcconfig might have a typo in its documentation which prevents PyPI from displaying the docs correctly.
https://pypi.python.org/pypi/tcconfig

Tcset set incoming rate error

When I execute the command in your tutorial to set the incoming rate , it gives me this error
And Using the tcshow command , we can see it didn't set successfully
It probably the parsing's problem from the log

root@cubieboard:~# tcset --device eth0 --direction incoming --rate 1M
[2016-11-25 13:14:09.023382] WARNING: subprocrunner: returncode=1, stderr=modprobe: FATAL: Module ifb not found.
[2016-11-25 13:14:09.059458] WARNING: subprocrunner: returncode=2, stderr=RTNETLINK answers: Operation not supported
[2016-11-25 13:14:09.098593] WARNING: subprocrunner: returncode=1, stderr=Cannot find device "ifb6682"
[2016-11-25 13:14:09.120417] WARNING: subprocrunner: returncode=2, stderr=RTNETLINK answers: No such file or directory
[2016-11-25 13:14:09.149173] WARNING: subprocrunner: returncode=1, stderr=Cannot find device "ifb6682"
bad action parsing
parse_action: bad value (5:mirred)!
Illegal "action"
[2016-11-25 13:14:09.178688] WARNING: subprocrunner: returncode=1, stderr=Cannot find device "ifb6682"
[2016-11-25 13:14:09.207769] WARNING: subprocrunner: returncode=1, stderr=Cannot find device "ifb6682"
[2016-11-25 13:14:09.236099] WARNING: subprocrunner: returncode=1, stderr=Cannot find device "ifb6682"
[2016-11-25 13:14:09.265203] WARNING: subprocrunner: returncode=1, stderr=Cannot find device "ifb6682"

hfsc support

Hi,

Any plan of WIP for supporting HFSC ?

Thanks.

Cannot delete rules

After adding latency, I cannot clear up all rules using tcdel -d DEVICE --all as specified in the doc. I am using v0.17.1. Removing everything (root) using tc works and afterwards tcshow shows no rules.

Make pytest-runner a conditional requirement

thombashi/DataProperty#20

thombashi / tcconfig Goto Github PK

tcconfig's Introduction

Summary

Traffic control

Setup traffic shaping rules

Available parameters

Targets

Usage

Set traffic control (tcset command)

e.g. Set a limit on bandwidth up to 100Kbps

e.g. Set network latency

Set 100 milliseconds of network latency

Set 10 seconds of network latency

Set 0.5 minutes (30 seconds) network latency

e.g. Set 0.1% packet loss

e.g. All of the above settings at once

e.g. Specify the IP address of the traffic control

e.g. Specify the IP network and port of traffic control

Set traffic control to a docker container

Set traffic control within a docker container

Delete traffic control (tcdel command)

e.g. Delete traffic control of eth0

Display traffic control configurations (tcshow command)

Example

For more information

Installation

Installation: pip

Installation: dpkg (Debian/Ubuntu)

Dependencies

Linux packages

Linux kernel module

Optional Python packages

Documentation

Troubleshooting

Docker image

Sponsors

tcconfig's People

Contributors

Stargazers

Watchers

Forkers

tcconfig's Issues

How to use capabilities

How to reproduce

How to reproduce

e.g. Set 100ms network latency

e.g. Set 100s network latency

e.g. Set 1m network latency

Recommend Projects

Recommend Topics

Recommend Org

Set traffic control (`tcset` command)

Delete traffic control (`tcdel` command)

e.g. Delete traffic control of `eth0`

Display traffic control configurations (`tcshow` command)