brownsys / pane Goto Github PK

Let's emit FML for bandwidth requirements for starters

Delete users, delete shares, delete accepted requests, ...

During ZooKeeper testing we hacked-up support for Open vSwitch queues in this commit: 89548f8

We should a) integrate this back into the main branch and b) learn how to use the Switch Features messages to determine the type of switch, so the NIB knows whether the switch's queues should be configured with this approach, or with the OF 1.0 extensions, etc.

Log to files
Log messages received by the controller
etc.

see: http://hackage.haskell.org/package/hslogger
and http://hackage.haskell.org/package/hslogger-template

state passing has been awful

there should be some unit tests which are affected by the passage of time, which can be faked using the new sysTime variable

Initial Questions for next time:

1. What types of messages do Zookeper nodes send between each other?
2. Where in the code do those messages come from?
3. What is the effect of a message being delayed?
4. How long is such a delay? for example, 4 seconds? 10 seconds?
5. Are there benefits to tightening the bound on the length of a delay? That is, if delays are currently allowed to be up to 4 seconds, would there be any benefit to only allowing delays to be up to 2 seconds?

These should help us work on the first of the three phases of this work:
phase 1) What can go wrong if Zookeeper traffic is delayed?
phase 2) Can we cause Zookeper traffic to be delayed by using non-Zookeeper traffic? To work on this, we will setup an isolated test network using a Cisco switch we have.
phase 3) Finally, if Zookeeper traffic can be delayed by non-Zookeeper traffic, can we use PANE (the project Rodrigo & I are developing) to protect the traffic?

Questions for later:

1. Does this scenario apply to other highly available services such as: HDFS High Availability (HA), the GPFS quorum, or Hadoop heartbeat messages?
2. Some questions for companies such as Yahoo and Twitter: a) what do they use Zookeeper for? b) how is it configured? c) how frequently does it have outages? d) do our failure scenarios make sense in their environments?

We have some links to papers, etc here: https://github.com/brown-csci1680/dnp-project/wiki/ZooKeeper-links

Make it possible to put rate-limit restrictions on the reservations. For example, "arjun can request this reservation once per hour" or perhaps "arjun gets 1 minute of guaranteed bandwidth every 6 minutes and can burst up to 20 minutes of guaranteed bandwidth"

There are two additional types we need to match on:

1. End hosts (both sending host and receiving host)
2. Conglomerate types: groups (sets of users), networks (sets of hosts), and applications (sets of ports)
   • I think this is as simple as keeping a data structure which is a registry of these sets and then having commands to add and remove members

The server will keep absolute time, but clients should be able to submit relative times such as "Now", "Forever" "+60m" "+2h" etc.

Seriously ... what are the best practices for using this framework? We have been making things up as we go along.

Several API functions now return values other than True or False. Need to improve unit testing code to be able to test these functions, and then write actual tests for them.

To permit accountReferences to be assigned to "all users" (which includes future users)

Code removed in 12b1499.

They are not magic.

We refer to "hints" at several points in the Hot-ICE paper. We should clarify what they are for the Hot-SDN paper.

Implement either:

• reserveUpto(N) to complement the existing reserveExactly(N)
• reserve(MIN, MAX) where reserveExactly(N) = reserve(N,N) and reserveUpto(N) = reserve(0, N)

Add units for bandwidth so we can reserve "10Mb" or "25kb" etc.

Find references by user: Add method to query tree to return all references user can use, including default references. -- at the moment, this requires enumeration of the full tree, but we could simply augment the speaker datatype to have a list of references ... this we would have keep updated as well, but it's just book-keeping work

Find shares by flow or flow group -- Can walk down the tree, checking each child to see if the flow/flow group fit inside that child, then do recursive descent

Find implemented rules by flow / flow group (ie, which rule is dropping my flow, etc?) -- Check against the live queue of resources to see matches

We're using Haskell.

Imagine building a tree of FML reservations, where keys are flows and values are bandwidth reservations. Keys are partially-ordered by most exact match. We should be able to build a test case where this tree has a different shape from the tree of shares. This would show that the language can express policies which have exceptional cases.

Major steps:

1. Stream a video using the VLC client from some easy-to-access URL.
2. Identify in the VLC code base where it opens the streaming video, and determine the video's bitrate, etc.
3. Once the bitrate is known, open a connection to the DNP server
   • Determine available shares for the user
   • Request a guaranteed reservation for the bandwidth
   • Prevent VLC from automatically adapting the bitrate of the request (if it currently does that)
4. Augment VLC GUI to present the user with alternatives if the bandwidth guarantee cannot currently be met:
   • Offer to make the guarantee later (eg, in 5 minutes)
   • Offer to make a smaller guarantee (eg, for 500 KB/s rather than 800 KB/s)
   • Offer to continue without a guarantee

Rate-limit? Latency? Jitter? Allow / deny ? Also, waypoints.

Parse errors and server errors should be sent to the client, rather than appear in the server console. To quote Arjun:

I'll have to write it from safari, but here is the basic idea.

The operations in FlowController.hs return (Maybe State), which means on error they simply return Nothing (no error message).

Change the return type to Either String State. On success, return Left errMessage and on failure return Right newState

You can do this to just one function without changing everything.

• Write a stub client in Java which can integrate into Hadoop
• Augment VLC to send bandwidth reservations
• Write a simple firewall which detects bad traffic and then installs DENY commands in the network

What should we do when there are two overlapping reservations made? (which can happen if two shares have overlapping responsibilities) … this problem has come to a head with allow/deny since the ultimate action is binary, but it actually already exists with bandwidth reservations

Variable names, etc.

Reservations should have a time associated with them! (eg, "for 30 minutes" or "for 10 minutes, 5 minutes from now"). After this is implemented, we can have the basic REPL.

Support requests such as "provide guaranteed bandwidth for this 1 hour transfer any time after 8pm and before 6am" ... if this request is received at noon and accepted, it should be possible to automatically re-schedule the transfer for another time. This would significantly increase the complexity of the scheduling algorithm since multiple requests would be re-scheduled at once (now it is more like a jobshop problem suitable for Comet) ... there would also need to be a way to tell the client about the rescheduled guarantee.

The schedules printed in 520d365 look like this:

2 # number of events 12 900 30 # timestamp bandwidth available tokens available 34 400 20

Let's instead print them like this:

2; 12,900,30; 34,400,20

I think the latter is just as amenable to parsing with scanf/PCRE/etc., and substantially easier for unit testing.

first reservation is from now to 10. First time is relative, but the second is absolute.

It should be from now to +10 ,

I don't think mixing absolute and relative time is wrong, but it allows some interesting requests. Imagine the distinction between

from asap to +10 vs. from asap to 10.

.... Also, SharePerms -> SharePrivs (as per the HotICE paper writing)

Integrate all bandwidth limits into the token buckets, and put the token bucket inside the SharePriv syntax: [ ]

reserve >= and reserve <= are both limits on the drain rate of the token bucket, so the bucket is really:
level, capacity, fill rate, drain lower bound, drain upper bound

This changes the GetSchedule function to only return a list of 2-tuples because available capacity is now equivalent to available tokens (so let's make them into a single "available capacity"). Now, the available capacity is restricted by two things: the token bucket, and the previously accepted reservations.

At the moment, parseStmt in Parser.hs can only return DNP Bool's, which makes it impossible to return the time, or return sets of alternatives, etc.

The Pour command allows us to migrate tokens from a parent token bucket to a child token bucket, en mass. This is to permit users to immediately use a new token bucket which might have a very slow fill rate.

We currently have a pour function stubbed out in TokenGraph.hs, but it does not pour from a bucket, and it does not respect the capacity of the receiving bucket.

Test cases are also required.

1. IsAvailable -- simple true/false response

2. HoldIfAvailable -- reservation which is held for a short period of time (to be defined) ... for the purposes of rate-limiting reservations, we count tokens used for this reservation as having been spent ... for the purposes of accepting/rejecting other reservations, we also count these tokens as having been spent ... the difference is that when we design markets for resources on top of this platform, this call will be useful because it is not a reservation which has been locked-in for the actual time, as we won't actually install it and won't actually withdraw the tokens (unless it is turned into a real reservation, of course)

Now that everything is working with mininet, we should incorporate some automatic tests into the script which launches mininet and creates a network with the PANE controller, etc.

Such a test script could easily do traffic tests with allow & deny.

Attractive graphical display of reservations / shareTree, etc.

Andreas' latest Nettle should build with the haskell-platform that comes with ubuntu 12.04 LTS.

We've started to add comments for where there should be invariants, for example, the start time in a reservation should always be before the end time. We need to add code to enforce these.

suggestion from Rodrigo

A Help command would be very useful during interactive testing. Should also provide examples.

The hardware testbed is almost setup. We need to finish getting it ready for guaranteed-minimum-bandwidth (GMB) experiments, which means changing the OpenFlow switch implementation to use the reference userspace implementation rather than the Open vSwitch implementation with HP extensions (which supports rate-limiting, not GMB).

A reservation in the discarded past should fail.

We need to be able to specify the transport protocol from the surface syntax. During ZooKeeper testing we had to hardcode matchIPProtocol = Just ipTypeTcp

There is no need to do this every second. Pick any interval.