robzenn92 / eptodocker Goto Github PK

View Code? Open in Web Editor NEW

0.0 2.0 0.0 16.79 MB

A real distributed system managed by Kubernetes and composed of Docker containers which run EpTO developed in Python

License: MIT License

Shell 9.86% Python 90.14%

p2p peer-to-peer python docker kubernetes cyclon epto

eptodocker's People

Watchers

eptodocker's Issues

Every EpTO peer should be able to run for a limited number of rounds

As stated in Section 4.1 Gossiping with Balls and Bins of the original paper:

The gossip protocol terminates after m rounds.

print type(class) returns <type 'instance'> rather than the real class

According to this the reason behind this behavior is that the implemented classes don't inherit from the Object class.

Introduce Smoke tests as part of all the deployment processes

We need a simple test to determine whether there are any basic problems that will prevent it from working at all.
Can we consider the route http://localhost:500[0|1]/hello ?

Sometimes partialView.merge raises an IndexError exception

Sometimes PartialView.py raises an exception in merge while Cyclon container is running.
The exception raised is the following:

[2017-12-13,17:16:13.464 - ERROR] base.py:131 | run_job - Job "Cyclon.shuffle_partial_view (trigger: interval[0:00:15], next run at: 2017-12-13 17:16:28 CET)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/usr/src/cyclon_project/api/cyclon.py", line 101, in shuffle_partial_view
    self.partialView.merge(neighbors, received_partial_view)
  File "/usr/src/partialView/partialView.py", line 127, in merge
    i += 1 if self.add_peer(unknown[i]) else 0
IndexError: list index out of range

More info in the following stack trace.

[2017-12-13,17:15:58.510 - INFO] cyclon.py:78 | shuffle_partial_view - My partialView:
{'ip': '172.17.0.6', 'limit': 3, 'peer_list': [{'age': 1, 'ip': '172.17.0.8'}, {'age': 2, 'ip': '172.17.0.5'}, {'age': 2, 'ip': '172.17.0.9'}], 'shuffle_length': 3, 'size': 3}
[2017-12-13,17:15:58.511 - INFO] cyclon.py:81 | shuffle_partial_view - Selected oldest: 172.17.0.5
[2017-12-13,17:15:58.513 - INFO] cyclon.py:84 | shuffle_partial_view - Selected neighbors:
{'ip': '172.17.0.6', 'limit': 3, 'peer_list': [{'age': 2, 'ip': '172.17.0.9'}, {'age': 1, 'ip': '172.17.0.8'}], 'shuffle_length': 3, 'size': 2}
[2017-12-13,17:15:58.513 - INFO] cyclon.py:87 | shuffle_partial_view - Selected neighbors + myself (will be sent to 172.17.0.5):
{'ip': '172.17.0.6', 'limit': 3, 'peer_list': [{'age': 2, 'ip': '172.17.0.9'}, {'age': 1, 'ip': '172.17.0.8'}, {'age': 0, 'ip': '172.17.0.6'}], 'shuffle_length': 3, 'size': 3}
[2017-12-13,17:15:58.514 - INFO] cyclon.py:110 | send_message - I am sending Message:
{'source': 'http://172.17.0.6:5000', 'destination': 'http://172.17.0.5:5000', 'data': {'ip': '172.17.0.6', 'limit': 3, 'peer_list': [{'age': 2, 'ip': '172.17.0.9'}, {'age': 1, 'ip': '172.17.0.8'}, {'age': 0, 'ip': '172.17.0.6'}], 'shuffle_length': 3, 'size': 3}}
[2017-12-13,17:15:58.519 - INFO] cyclon.py:112 | send_message - I got the following response:
<Response [200]>
[2017-12-13,17:15:58.520 - INFO] cyclon.py:94 | shuffle_partial_view - I received (from 172.17.0.5):
{'ip': '172.17.0.5', 'limit': 3, 'peer_list': [{'age': 2, 'ip': '172.17.0.8'}, {'age': 8, 'ip': '172.17.0.2'}, {'age': 2, 'ip': '172.17.0.6'}], 'shuffle_length': 3, 'size': 3}
[2017-12-13,17:15:58.520 - INFO] cyclon.py:98 | shuffle_partial_view - My partialView after removing oldest:
{'ip': '172.17.0.6', 'limit': 3, 'peer_list': [{'age': 1, 'ip': '172.17.0.8'}, {'age': 2, 'ip': '172.17.0.9'}], 'shuffle_length': 3, 'size': 2}
[2017-12-13,17:15:58.520 - INFO] cyclon.py:102 | shuffle_partial_view - My partialView after merging:
{'ip': '172.17.0.6', 'limit': 3, 'peer_list': [{'age': 1, 'ip': '172.17.0.8'}, {'age': 2, 'ip': '172.17.0.9'}, {'age': 8, 'ip': '172.17.0.2'}], 'shuffle_length': 3, 'size': 3}
[2017-12-13,17:15:58.520 - INFO] base.py:135 | run_job - Job "Cyclon.shuffle_partial_view (trigger: interval[0:00:15], next run at: 2017-12-13 17:16:13 CET)" executed successfully
[2017-12-13,17:15:59.759 - INFO] views.py:75 | get_k_view - {'ip': '172.17.0.6', 'limit': 3, 'peer_list': [{'age': 1, 'ip': '172.17.0.8'}, {'age': 2, 'ip': '172.17.0.9'}, {'age': 8, 'ip': '172.17.0.2'}], 'shuffle_length': 3, 'size': 3}
[2017-12-13,17:16:13.456 - INFO] base.py:123 | run_job - Running job "Cyclon.shuffle_partial_view (trigger: interval[0:00:15], next run at: 2017-12-13 17:16:13 CET)" (scheduled at 2017-12-13 17:16:13.445482+01:00)
[2017-12-13,17:16:13.456 - INFO] cyclon.py:74 | shuffle_partial_view - Shuffling
[2017-12-13,17:16:13.456 - INFO] cyclon.py:78 | shuffle_partial_view - My partialView:
{'ip': '172.17.0.6', 'limit': 3, 'peer_list': [{'age': 2, 'ip': '172.17.0.8'}, {'age': 3, 'ip': '172.17.0.9'}], 'shuffle_length': 3, 'size': 2}
[2017-12-13,17:16:13.456 - INFO] cyclon.py:81 | shuffle_partial_view - Selected oldest: 172.17.0.9
[2017-12-13,17:16:13.457 - INFO] cyclon.py:84 | shuffle_partial_view - Selected neighbors:
{'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 2, 'ip': '172.17.0.8'}], 'shuffle_length': 2, 'size': 1}
[2017-12-13,17:16:13.457 - INFO] cyclon.py:87 | shuffle_partial_view - Selected neighbors + myself (will be sent to 172.17.0.9):
{'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 2, 'ip': '172.17.0.8'}, {'age': 0, 'ip': '172.17.0.6'}], 'shuffle_length': 2, 'size': 2}
[2017-12-13,17:16:13.457 - INFO] cyclon.py:110 | send_message - I am sending Message:
{'source': 'http://172.17.0.6:5000', 'destination': 'http://172.17.0.9:5000', 'data': {'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 2, 'ip': '172.17.0.8'}, {'age': 0, 'ip': '172.17.0.6'}], 'shuffle_length': 2, 'size': 2}}
[2017-12-13,17:16:13.463 - INFO] cyclon.py:112 | send_message - I got the following response:
<Response [200]>
[2017-12-13,17:16:13.463 - INFO] cyclon.py:94 | shuffle_partial_view - I received (from 172.17.0.9):
{'ip': '172.17.0.9', 'limit': 1, 'peer_list': [{'age': 2, 'ip': '172.17.0.7'}], 'shuffle_length': 1, 'size': 1}
[2017-12-13,17:16:13.463 - INFO] cyclon.py:98 | shuffle_partial_view - My partialView after removing oldest:
{'ip': '172.17.0.6', 'limit': 3, 'peer_list': [{'age': 2, 'ip': '172.17.0.8'}], 'shuffle_length': 3, 'size': 1}
[2017-12-13,17:16:13.464 - ERROR] base.py:131 | run_job - Job "Cyclon.shuffle_partial_view (trigger: interval[0:00:15], next run at: 2017-12-13 17:16:28 CET)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/usr/src/cyclon_project/api/cyclon.py", line 101, in shuffle_partial_view
    self.partialView.merge(neighbors, received_partial_view)
  File "/usr/src/partialView/partialView.py", line 127, in merge
    i += 1 if self.add_peer(unknown[i]) else 0
IndexError: list index out of range

Introduce an endpoint to retrieve all the delivered events

Introduce in EpTO a new endpoint to retrieve all the delivered events.
As an example, the endpoint could be:

GET http://localhost:5001/v1/delivered-events

Add minikube status checks to bash scripts

Since version 0.24.0, Minikube has introduces different exit codes for the command minikube status. There are several bash scripts in the project that require a minikube cluster up and running. In order to be able to execute the scripts, some checks about the cluster status are needed.

Refer to this: #1944

PartialView.merge should not allow self ip in peer_list

Change the receive-ball endpoint with send-ball

Right now the EpTO's endpoint to send a nextBall is http://localhost:5001/v1/receive-ball. Lets rename it to http://localhost:5001/v1/send-ball.

Introduce API versioning

API endpoints should start with the version of API currently used.
For example, localhost:5000/exchange-view should become to localhost:5000/v1/exchange-view.

Add a new endpoint in EpTO to broadcast a custom event

I would like to have an endpoint such as http://localhost:5001/v1/broadcast that should add a custom event to the NextBall set. The endpoint should be targeted via a POST method.

Introduce Structured logging

Add EpTO endpoint to get the Ball set at any time for debugging purposes

I would like to have an endpoint such as http://localhost:5001/v1/ball that should return the Ball set as is. The endpoint should be targeted via a GET method.

Sometimes exchange_view() raises an IndexError exception

The exception is the following:

Internal Server Error: /exchange-view
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/src/cyclon_project/api/views.py", line 102, in exchange_view
    cyclon.partialView.merge(to_send, received_partial_view)
  File "/usr/src/partialView/partialView.py", line 131, in merge
    self.remove_peer(can_be_replaced[i - how_many])
IndexError: list index out of range
[2017-12-13,17:22:00.424 - ERROR] exception.py:118 | handle_uncaught_exception - Internal Server Error: /exchange-view
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/src/cyclon_project/api/views.py", line 102, in exchange_view
    cyclon.partialView.merge(to_send, received_partial_view)
  File "/usr/src/partialView/partialView.py", line 131, in merge
    self.remove_peer(can_be_replaced[i - how_many])
IndexError: list index out of range

More info in the following stack trace.

[2017-12-13,17:21:58.446 - INFO] cyclon.py:78 | shuffle_partial_view - My partialView:
{'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 1, 'ip': '172.17.0.9'}, {'age': 1, 'ip': '172.17.0.5'}], 'shuffle_length': 2, 'size': 2}
[2017-12-13,17:21:58.447 - INFO] cyclon.py:81 | shuffle_partial_view - Selected oldest: 172.17.0.9
[2017-12-13,17:21:58.447 - INFO] cyclon.py:84 | shuffle_partial_view - Selected neighbors:
{'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 1, 'ip': '172.17.0.5'}], 'shuffle_length': 2, 'size': 1}
[2017-12-13,17:21:58.447 - INFO] cyclon.py:87 | shuffle_partial_view - Selected neighbors + myself (will be sent to 172.17.0.9):
{'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 1, 'ip': '172.17.0.5'}, {'age': 0, 'ip': '172.17.0.6'}], 'shuffle_length': 2, 'size': 2}
[2017-12-13,17:21:58.447 - INFO] cyclon.py:110 | send_message - I am sending Message:
{'source': 'http://172.17.0.6:5000', 'destination': 'http://172.17.0.9:5000', 'data': {'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 1, 'ip': '172.17.0.5'}, {'age': 0, 'ip': '172.17.0.6'}], 'shuffle_length': 2, 'size': 2}}
[2017-12-13,17:21:58.452 - INFO] cyclon.py:112 | send_message - I got the following response:
<Response [200]>
[2017-12-13,17:21:58.453 - INFO] cyclon.py:94 | shuffle_partial_view - I received (from 172.17.0.9):
{'ip': '172.17.0.9', 'limit': 1, 'peer_list': [{'age': 1, 'ip': '172.17.0.8'}], 'shuffle_length': 1, 'size': 1}
[2017-12-13,17:21:58.453 - INFO] cyclon.py:98 | shuffle_partial_view - My partialView after removing oldest:
{'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 1, 'ip': '172.17.0.5'}], 'shuffle_length': 2, 'size': 1}
[2017-12-13,17:21:58.453 - INFO] cyclon.py:102 | shuffle_partial_view - My partialView after merging:
{'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 1, 'ip': '172.17.0.5'}, {'age': 1, 'ip': '172.17.0.8'}], 'shuffle_length': 2, 'size': 2}
[2017-12-13,17:21:58.453 - INFO] base.py:135 | run_job - Job "Cyclon.shuffle_partial_view (trigger: interval[0:00:15], next run at: 2017-12-13 17:22:13 CET)" executed successfully
[2017-12-13,17:21:59.753 - INFO] views.py:75 | get_k_view - {'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 1, 'ip': '172.17.0.5'}, {'age': 1, 'ip': '172.17.0.8'}], 'shuffle_length': 2, 'size': 2}
[2017-12-13,17:22:00.424 - INFO] views.py:90 | exchange_view - My view before the exchange is:
{'ip': '172.17.0.6', 'limit': 2, 'peer_list': [{'age': 1, 'ip': '172.17.0.8'}], 'shuffle_length': 2, 'size': 1}
[2017-12-13,17:22:00.424 - INFO] views.py:95 | exchange_view - I got (from http://172.17.0.7:5000) the following:
{'ip': '172.17.0.7', 'limit': 3, 'peer_list': [{'age': 1, 'ip': '172.17.0.2'}, {'age': 2, 'ip': '172.17.0.8'}, {'age': 0, 'ip': '172.17.0.7'}], 'shuffle_length': 3, 'size': 3}
[2017-12-13,17:22:00.424 - INFO] views.py:99 | exchange_view - I will send (to http://172.17.0.7:5000) the following:
{'ip': '172.17.0.6', 'limit': 1, 'peer_list': [{'age': 1, 'ip': '172.17.0.8'}], 'shuffle_length': 1, 'size': 1}.
Internal Server Error: /exchange-view
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/src/cyclon_project/api/views.py", line 102, in exchange_view
    cyclon.partialView.merge(to_send, received_partial_view)
  File "/usr/src/partialView/partialView.py", line 131, in merge
    self.remove_peer(can_be_replaced[i - how_many])
IndexError: list index out of range
[2017-12-13,17:22:00.424 - ERROR] exception.py:118 | handle_uncaught_exception - Internal Server Error: /exchange-view
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/src/cyclon_project/api/views.py", line 102, in exchange_view
    cyclon.partialView.merge(to_send, received_partial_view)
  File "/usr/src/partialView/partialView.py", line 131, in merge
    self.remove_peer(can_be_replaced[i - how_many])
IndexError: list index out of range

Since Minikube v0.23.0 supports Kubernetes v1.8, Pod UID should be exposed by the downward API

As I can read from this closed issue, Pod UID is supported since v1.8.0-alpha.2.

Cyclon exchange_view() should not return back to the source a partialView containing the source's ip

In cyclon/app.py the following piece of code is wrong.

source = message.get('source')
size = received_partial_view.size
to_avoid = None
if cyclon.partialView.contains_ip(source):
    to_avoid = cyclon.partialView.get_peer_by_ip(source)
to_send = cyclon.partialView.select_neighbors(to_avoid, size)

In fact, source is in the form http://<source_ip>:<port> while both cyclon.partialView.contains_ip and cyclon.partialView.get_peer_by_ip expect an IP. Hence, to_avoid value was alway None.

Introduce Elasticsearch to collect and analyze container logs

Rather than Splunk (Kubernetes Logging with Splunk) use Kubernetes Logging into Elasticsearch

Investigate Flask Deployment Options

As stated here:

While lightweight and easy to use, Flask’s built-in server is not suitable for production as it doesn’t scale well and by default serves only one request at a time.

We need to handle more than one request at a time.

Cyclon get_k_view should return a list of IPs

Right now get_k_view() route in cyclon/app.py returns a list of PodDescriptor. However, EpTO does not need to know what is a PodDescriptor, neither will use the concept of "age" for each peer. Hence, get_k_view() should return a list of IPs so that EpTO can simply go through the list and send messages to each entry of the list.

robzenn92 / eptodocker Goto Github PK

eptodocker's People

Watchers

eptodocker's Issues

Recommend Projects

Recommend Topics

Recommend Org