Function time_resolution_init() located in file main/main.c estimates the frequency of the TSC register and sets its result in a couple global variable in different units (e.g. cycles_per_sec). DPDK has function rte_get_tsc_hz(), which is equivalent to the global variable cycles_per_sec.

Which function is more precise, time_resolution_init() or rte_get_tsc_hz()? If time_resolution_init(), we should submit a patch to DPDK. In any case, we should adopt rte_get_tsc_hz() in time_resolution_init() to simplify our code.

We should also investigate if the global variables are still needed. For example, should we replace variable cycles_per_sec with rte_get_tsc_hz() or keep it?

The GK blocks, GT blocks, and the GK-GT unit are the critical blocks of Gatekeeper and Grantor servers. In order to allow them to reach line speed, we need to maximize their cache hits, even if this reduces the cache hits of other blocks.

The code of Gatekeeper has a number of constants such as GATEKEEPER_MAX_PORTS, GATEKEEPER_MAX_QUEUES, GATEKEEPER_NUM_RX_DESC, GATEKEEPER_NUM_TX_DESC, GATEKEEPER_MBUF_SIZE, GATEKEEPER_CACHE_SIZE, and GATEKEEPER_MAX_PKT_BURST.

The list above is not exhaustive, and one must hunt all these constants throughout the code.

Not only do these parameters have a high impact on the performance of Gatekeeper, but they may potentially hinder deployment. For example, if a given deployment needs more network ports than GATEKEEPER_MAX_PORTS, that Gatekeeper instance will be unnecessarily hindered.

Currently, the flow tables of the GK blocks grow indefinitely. There must be a way to reclaim expired entries and another way to alleviate memory pressure when the table is full.

Currently, once all slave lcores receive their assignments, the master lcores sits busy waiting for each functional block to stop running. That is, the master lcore goes wasted.

Given that the LLS block must always be present, it could run at the master lcore. Alternatively, if the proposed solution of #16 has been already implemented, the configuration could allocate any block to run at the master lcore, and this block pushed to the very end of the configuration to be executed by the master lcore.

We should be able to get the relevant information to implement ECMP through the RTA_MULTIPATH attribute of the rtnetlink library. There's already a stub for handling this functionality in cps/kni.c.

We need to figure out how to use this information to implement ECMP, and do it only when running Grantor.

This feature should remove the relevant XXX in cps/kni.c.

Each interface that Gatekeeper uses needs to have a maximum number of receiving and transmitting queues. The calculation should be automated to avoid this unnecessary, error-prone burden on network administrators.

The issue has origin on the initialization order of DPDK: queues can only be set up after all of them are created. And functional blocks are the only ones that know how many queues they need, and how to set them up. Thus, it's a chicken and egg problem.

Solution proposal: make function XXX_run() return an initialization struct. So, once the Lua scripts finish, Gatekeeper can easily calculate the number of queues and call the initialization functions one by one. This change should make field configuring in struct net_config obsolete.

Given that the log entries of the functional blocks will no longer be in synch with the execution of the Lua scripts in the proposed solution, it may be necessary to review the log entries during initialization in order to keep problem diagnoses easy.

By default DPDK's logging system writes to the standard error device. Since Gatekeeper will run as a daemon, it would be more useful for all DPDK logs to be output to a file.

This should remove the relevant XXX tags in lib/acl.c and (if needed) in main/main.c.

If policies were responsible for reassembling fragmented packets, they would be way harder to write. In addition, policies would have to forgo all the optimizations built into DPDK to assemble packets due to the interface between LuaJit and DPDK. Therefore, GT blocks must assemble fragmented packets before making a policy decision.

A good news is that DPDK has a library to assemble packets:
http://dpdk.org/doc/guides/prog_guide/ip_fragment_reassembly_lib.html

This feature has some corner cases that need to be handled properly:

1. The fragmented packets have to be assembled without the outer IP header (i.e. the tunnel header), but passed with the outer IP header to the policy. We can use the outer IP header of the last fragment to do avoid keeping the outer header until it's needed.

2. gt_parse_incoming_pkt() should no longer require that the l4 header must have at least 4 bytes because we can't assume anything about the l4 header of fragmented packets.

3. Flows associated with fragments that have to be discarded before being fully assembled must be punished. Otherwise, an attacker could overflow the request channel with fragments that never complete, and policies wouldn't be able to do anything about it because they would not be aware of these fragments. The punishment is essentially a policy decision stated in the configuration files to be applied to these cases. For example, decline the flow for 10 minutes.

There are a few TODO markers in the Lua policy files.

For example, the Lua policy should be responsible for checking the necessary space for each L4 header type, and we should add support for other transport protocols.

We also need to add examples of operations to list the GK FIB entries, the ARP table, the ND table, etc.

Policy decisions (from Grantor servers to Gatekeeper servers) should be batched to reduce the number of packets that Gatekeeper has to handle. However, there is also a timing constraint: policy decisions should not wait for too long before being dispatched.

This feature should remove the relevant TODO in gt/main.c.

Flow entries should maintain a reference counter of the grantor FIB entry to avoid the entry to going away before the flow.

All lcores running GK initialize flow entries, so a solution to this issue needs to deal with concurrency.

The chosen solution must be very efficient due to the impact on the throughout of the GK blocks. For example, a simple atomic counter will slow down all GK blocks, especially if there is only one grantor FIB entry.

Idea for a solution: trade the need to maintain the reference counter of the grantor FIB entry for some expensive operation that is only needed while editing the FIB table.

This feature should remove the relevant TODO in gk/main.c.

LACP can take a long time (seconds) to properly negotiate and configure, and in the meantime there can be LLS transmission errors that are logged. We should ignore these error messages while still configuring, and also warn the user if LACP is taking an unusually long time to configure since a failure might have occurred. For example, the link partner may not have LACP configured.

This feature should remove the relevant TODO in lls/main.c.

This pull request must also include three other patches: one to drop the old code of RT, another to drop the old code of BP, and another to drop Catcher.

We need to get a polished version of the GK queue discipline, and get it merged in the Linux kernel. This will simplify a ton the deployment of Gatekeeper once the GK queues become available in stock kernels.

Currently, when the GT block is unable to send packets, it immediately drops them. We should investigate whether we can do something better: logging the packets, try to send them again, etc.

This feature should remove the relevant XXX in gt/main.c.

Supporting to a 1Gbps NIC is important for interconnecting with legacy network meshes. This is especially critical for Grantor servers. This feature would also enable people to test Gatekeeper in cheap, commonplace test environments.

A suggestion on how to do this is to implement a virtual NIC over a real dummy NIC. This virtual NIC only has to implement the filters Gatekeeper needs. Moreover, this virtual NIC only has to reach 1Gbps of throughput, for anything above that the installer has to acquire one of the NICs Gatekeeper supports.

We need to make sure that prefixes for existing drop and Grantor entries can only be superseded by new prefixes for drop or Grantor entries.

For example, assume there's a prefix 10.1.1.0/24 that leads to a gateway and the prefix being added to a grantor is 10.1/16. The gateway entry would be a security hole.

The solution is to extend RTE LPM table to have an iterator over its entries. The iterator should require a prefix as a parameter and should list all entries as long as the given prefix (or longer). To list all entries, the given prefix would have length zero.

This feature should remove the relevant TODO in gk/fib.c.

Gatekeeper doesn't need jumbo frames, but the code must be able to handle jumbo frames to avoid the chance of existing a jumbo, malformed packet that can knock out a Gatekeeper server.

Gatekeeper writes log entries to report general information, warnings, and diagnosis information. These log entries are essential for operating a Gatekeeper deployment. However, during high loads or DDoS attacks, there is a potential for the log subsystem to become too heavy of a burden and impact the capacity of Gatekeeper servers. Especially, if attackers find an easy way to trigger log entries. To dodge this risk entirely, Gatekeeper must rate limit the log entries.

The Linux kernel has already a solution for this issue, namely functions: net_err_ratelimited(), net_warn_ratelimited(), net_notice_ratelimited(), among others. The proposed solution here is to study these functions to verify if we can/should adopt them in Gatekeeper. If not, we'll need to devise our own solution for this problem.

This patch needs to update the TTL/hop_limits field in IP headers. As a result, one also needs to update the checksum of the IPv4 header. Moreover, packets with reached TTL/hop limit should be dropped, etc. We need to look at how the Linux kernel deals with TTL/Hop limit field in production.

The LLS block currently checks to make sure that we're only receiving packets that are destined to us, which is one of four L2 addresses. The NIC should be able to handle this functionality, so we should verify that it can (and that all NICs in our deployment do the same).

This feature should remove the relevant XXX in lls/main.c.

The code of Gatekeeper assumes that IP headers always have the same size, but due to options, the headers can vary in size. Moreover, the mechanism is different for IPv4 and IPv6.

The code that parses packets must deal with the variable size. Perhaps, DPDK has some functions to help to deal with it.

Client requests for dynamic configuration must be received under a specified timeout, or the request should be aborted. The code that's currently in place needs to be further tested.

This feature should remove the relevant XXX in config/dynamic.c.

Issue #27 makes it so that the BGP speakers that use the KNI can edit the routing table of GK blocks.

The problem is that this routing table is duplicated: once in the kernel and once in Gatekeeper. Instead, it would be better to have the BGP speaker directly communicate with the CPS block using Netlink.

We need to find which BGP speaker(s) are relevant for our deployment(s), and patch them to include this functionality. The BGP speaker Bird (http://bird.network.cz/) is a strong candidate to be patched given its popularity. Another positive factor is that they have documentation for developers:

http://bird.network.cz/?get_doc&f=prog.html

All remaining TODO tags must be implemented.

All XXX tags must be reviewed and decided if they must be implemented for the first production release, or left in the code.

While allowing one to map which logical core to each functional block in the configuration of Gatekeeper is helpful for rare fine tunings and debugging, it's a pain in the neck for regular operation. One needs to automate this allocation.

The GT-GK Unit should be verifying the UDP checksum for packets that it receives.

This feature should remove the relevant XXX in ggu/main.c.

The secret key of the RSS hash must be random in order to prevent hackers from knowing it.

This feature should remove the relevant XXX in lib/net.c.

At least three files (gt/main.c, config/static.c, config/dynamic.c) hard code the path to the Lua directory. We should find a way of making this dynamic. Since it effects the basic configuration of Gatekeeper -- even before Lua is invoked -- we may need to pass it from the Makefile.

This also relates to issue #14, and might be solved in that patch if it's simple enough.

Any sort of examples for the config would be helpful. Planning to take some benchmarks of this.

The dynamic configuration block only supports running on Gatekeeper servers, but we should enable it to run on Grantor servers as well.

This feature should remove the relevant TODO in config/dynamic.c.

The memory pools per NUMA node held in struct net_config are always allocated; even when a given NUMA node is not being used.

We need to check which NUMA nodes are necessary, and not to allocate the not used nodes.

This will save memory that can be used by other critical data structures such as the flow table.

We should customize the neighbor table hash function for IPv4.

Why?

This feature should remove the relevant TODO in include/gatekeeper_fib.h.

If a colluding attacker is present in the destination network, then the attacker could in theory spoof Grantor command packets to Gatekeeper servers. We should explore ways of defending against this kind of inside attack, perhaps using an authentication handshake between Grantor and Gatekeeper servers.

The function gk/fib.c:synchronize_gk_instances() is analogous to RCU's synchronize_rcu(), which suggests that we may be able to profit from RCU.

We should consider whether Gatekeeper could profit from RCU for this function and others, and whether it's worth porting RCU to Gatekeeper. If so, implement it.

This feature should remove the relevant XXX in gk/fib.c.

We've been assuming that RTE_ASSERT() works similarly to assert(), that is, it's always there, unless removed in compile time. But RTE_ASSERT() works the other way around as one can find looking at include/rte_debug.h of DPDK:

#if RTE_LOG_LEVEL >= RTE_LOG_DEBUG
#define RTE_ASSERT(exp)	RTE_VERIFY(exp)
#else
#define RTE_ASSERT(exp) do {} while (0)
#endif
#define	RTE_VERIFY(exp)	do {                                                  \
	if (unlikely(!(exp)))                                                           \
		rte_panic("line %d\tassert \"" #exp "\" failed\n", __LINE__); \
} while (0)

We need to review all RTE_ASSERT() in the code and decide if we should keep them as they're, or change to RTE_VERIFY(). Likely, we need to replace all RTE_ASSERT().

The IPv6 ntuple filter is unable to verify the destination IP address of the packet, so it's possible that Gatekeeper could receive a packet that is not destined for it. In that case, we should redirect the packet.

This has only been marked as a TODO for the GT-GK Unit (the back interface), since the front interface (the GK block) already forwards IPv6 traffic as needed.

This feature should remove the relevant TODO in ggu/main.c.

When a network administrator changes a policy in response to an attack, she may need to flush all policy decisions cached at the Gatekeeper servers associated to a given destination in order to quickly re-establish the network. Thus, we need to extend the protocol of the GK-GT unit to support these flush requests.

We have a number of small patches to DPDK in https://github.com/cjdoucette/dpdk that Gatekeeper relies on. We should get these patches merged upstream, so Gatekeeper can use the latest version of DPDK.

The BGP speakers that use the KNI interfaces exported by the CPS block must have a way to edit the routing table of the GK blocks, so Gatekeeper servers can send the outgoing traffic to the IX.

The editing of the routing table in the Linux kernel is usually done through the Netlink interface. If the CPS can support Netlink, it would be easier to patch BGP speakers to redirect the editing commands to Gatekeeper.

The BGP speaker Bird (http://bird.network.cz/) is a strong candidate to be patched given its popularity. Another positive factor is that they have documentation for developers:

http://bird.network.cz/?get_doc&f=prog.html

It might be useful to dynamically dump the GK FIB tables for debugging purposes. This feature should add this ability as well as the component of the dynamic configuration block that would enable it.

This feature should remove the TODO in include/gatekeeper_fib.h.

Although supporting VLAN tags is essential only at the front interface of Gatekeeper servers due to network requirements of IXes, this feature should be available on both interfaces and for Grantor servers as well in order to keep the network code as symmetric as possible.

The code should use hardware acceleration where it's available, but be able to do all in software if not. The reason for this is to help with issue #22 .

We need to implement more functionality that allows for dynamic interaction with running Gatekeeper and Grantor servers.

For example, both need to be able to dump their ARP and ND tables for network diagnosis, and we need to have a way to update the policy given to Grantor at runtime. There might be more functionality required for the first deployment as well.

This feature should remove the relevant TODO in config/dynamic.c.

For the 82599 NIC, the EtherType value of each EtherType filter should be passed in little endian order. Is this the case with all NICs we are using in our deployments that support the EtherType filter?

This feature should remove the relevant XXX in lib/net.c.