gisson / cnv1617 Goto Github PK

part of #7

Part of #24

Wait for #7 ?

The code of the application that performs the raytracing (called by the web server) is written in the
Java programming language and compiled into bytecode. The application is to be further instrumented
with a Java instrumentation tool (such as BIT that will be presented in the labs) in order to extract and
persistently store the dynamic performance metrics regarding the code executed. The explicit duration
(wall time) of each request handled should not be stored in the MSS.

These nodes will process the RenderFarm requests using code that was previously instrumented by the students, in order to collect relevant dynamic performance metrics regarding the application code executed (e.g. number of function calls executed, invocation stack depth, bytecodes executed, and others deemed relevant). They will allow estimating task complexity realistically, irrespective of variable delays, that could be caused by resource overcommit by the cloud provider.
The final choice of the metrics extracted, instrumentation code, and system used to store the metrics
data is free and subject to analysis and decision by the students

I think the metrics can be stored locally in the Checkpoint milestone.

E.g.

• memory
• number of instructions

• install AWS SDK in node images (renderer/load-balancer)

cd ~
wget "http://sdk-for-java.amazonwebservices.com/latest/aws-java-sdk.zip"
unzip aws-java-sdk.zip

make sure it's version aws-java-sdk-1.11.131

• install MSS DB access keys
  ~/.aws/credentials

• make the init script download the ... jar from somewhere or include it in the image

Part of #24

This load balancer should be our proprietary load balancer with super awesome skillz. Our proprietary balancer must be l33t and n0t l1k3 7h3 4w5 1.

Mimic the AWS load balancer.

part of #10

The the same shenanigan all over again but in the init send the load-balancer parameter.
Why me tho? 🥇
part of #25

It working during the checkpoint. Now ant gives the message:

run:
     [java] Error occurred during initialization of VM
     [java] Could not reserve enough space for object heap
     [java] Picked up _JAVA_OPTIONS: -XX:-UseSplitVerifier
     [java] Error: Could not create the Java Virtual Machine.
     [java] Error: A fatal exception has occurred. Program will exit.
     [java] Java Result: 1

Load balancer that uses the metrics to decide where to forward stuff.

wait for #25

part of #10

Remove the HttpServer default timeout.

Using the amazon database with mr wrap ist we should store all of the metrics in this system where the the load-balancer should get the metrics from.

• Receive request and parse the arguments.
• Reply parsed arguments

The URL for the renderer should have the following format:
http://<load-balancer-DNS-name>/r.html?f=<model-filename>&sc=<scene-columns>&sr=<scene-rows>&wc=<window-columns>&wr=<window-rows>&coff=<column-offset>&roff=<row-offset>
where r.html points to the code that receives the request and the parameters express the total size of the scene in rows and columns to be considered (sc,sr), the size of the (partial) viewport window to be rendered (wc,wr), and the offsets, regarding the top left corner of the image, where the viewport window should start.

part of #4

After this is done, proceed with invoking renderer classes #5

use an Amazon AWS Autoscaling group that adaptively decides how many web server nodes should be active at any given moment. It is up to the students the challenge to design the autoscaling rules that will provide the best balance between performance and cost. It should detect that the web app is overloaded and start new instances and, conversely, reduce the number of nodes when the load decreases.

The algorithms for load balancing, auto-scaling and the MSS need not be fully implemented at this stage (metrics can be stored temporarily in the computing nodes) but should be already thought out.

Make the raytracer run on the EC2 nodes.

Extend the webserver class to operate multithreaded.

part of #4

Make the Render classes build. It already includes a makefile, just it's just a matter of:

• invoking this makefile (both travis CI and EC2 instances)
• consider making the EC2 instances call some script in the root of the repo. Then the script compiles and runes everything
• defining the classpath that's used to launch the WebServer.

part of #7

Part of #10

part of #31

The images running on the render nodes should, on startup (rc.local) switch to the webserver user and run the $HOME/autostart script which:

• clones the repo: git clone -b release --depth 1 [email protected]:Gisson/CNV1617.git CNV1617
• runs the init script in the repo like so: cd CNV1617 && while bash init.sh render-node; do sleep 1; done, so the autostart script exits when init.sh fails.
• rc.local should shut-down the machine if the $HOME/autostart script exits. This should come in handy since the init.sh script can not shutdown the machine (needs root). For instance:

(
sudo --login -u webserver -- /home/webserver/autostart
sleep 5
poweroff
) &

Also, make sure that:

• the github deploy key is installed
• ant is installed
• BIT is already installed in $HOME/BIT for improved startup speed

Make extension for the load balancer
Part of #25

Can I switch this project to public and add a license?

The checkpoint submission bundle must include intermediate report (2-page, double column)
describing clearly:

• a) what is already developed in the current implementation (architecture, data
  structures and algorithms);
• b) the specification of what remains to be implemented.

The instrumentation code needs to know to what request a thread belongs to.
This will allow us to know that, for instance, request /r.html?f=test05.txt&sc=640&sr=360&wc=640&wr=360&coff=0&roff=0 needed 92069358 method calls, 29574356 allocations, etc.
A simple way to do this is for the server to store a <request, threadID> list which the instrumented code can then access before writing the metrics.

part of #10

The load balancer is the only entry point into the system: it receives a sequence of web requests and
selects one of the active web server cluster nodes to handle each of the requests.

In a first phase, this job can be performed by an off-the-shelf load balancer such as the ones available at Amazon AWS. Later in the project, you should design a more advanced load balancer that, using metrics data stored in the Metrics Storage System, will pick the best web server node to handle a request.

We need a load balancer for the Checkpoint, but I'm not sure whether In a first phase means this checkpoint or not.

The algorithms for load balancing, auto-scaling and the MSS need not be fully implemented at this stage (metrics can be stored temporarily in the computing nodes) but should be already thought out.

WebServer class invoking functionality of class to perform Rendering.
Assume these classes are magically in the classpath until #9 is finished.

• new RayTracer ()
• reply with bmp file
• set content-type to image/bmp 😄

part of #4