PART 1.1

install python,django,celery,uwsgi,nginx on ubuntu 16.04 LTS

install python, pip

sudo apt-get update 
sudo apt-get install python-pip 
sudo -H pip install --upgrade pip

install virtualenv

sudo -H pip install virtualenv virtualenvwrapper 
echo "export WORKON_HOME=~/Env" >> ~/.bashrc 
echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc 
source ~/.bashrc

install django and create first site

mkvirtualenv firstsite  
pip install django 
pip install celery  
cd ~ 
django-admin.py startproject firstsite 
cd ~/firstsite 
~/firstsite/manage.py migrate 
~/firstsite/manage.py createsuperuser 
	username: admin 
	password:Pfix0insa

nano ~/firstsite/firstsite/settings.py

ALLOWED_HOSTS = ['your-ip-address', 'localhost'] and 
add STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

~/firstsite/manage.py collectstatic

test running:

~/firstsite/manage.py runserver 0.0.0.0:8080

install uwsgi

sudo -H pip install uwsgi	
sudo mkdir -p /etc/uwsgi/sites 
sudo nano /etc/uwsgi/sites/firstsite.ini 

	[uwsgi]
	project = firstsite
	uid = $USER
	base = /home/%(uid)
	chdir = %(base)/%(project)
	home = %(base)/Env/%(project)
	module = %(project).wsgi:application

	master = true
	processes = 5

	socket = /run/uwsgi/%(project).sock
	chown-socket = %(uid):www-data
	chmod-socket = 660
	vacuum = true

Note! replace $USER with username

sudo nano /etc/systemd/system/uwsgi.service 

	[Unit]
	Description=uWSGI Emperor service
	[Service]
	ExecStartPre=/bin/bash -c 'mkdir -p /run/uwsgi; chown $USER:www-data /run/uwsgi'
	ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
	Restart=always
	KillSignal=SIGQUIT
	Type=notify
	NotifyAccess=all

	[Install]
	WantedBy=multi-user.target

Note! replace $USER with username

start uwsgi service <br/>
sudo service uwsgi start <br/>

install nginx server

sudo apt update <br/>
sudo apt install nginx <br/>
sudo nano /etc/nginx/sites-available/firstsite <br/>

	server {
	    listen 80;
	    server_name localhost;

	    location = /favicon.ico { access_log off; log_not_found off; }
	    location /static/ {
		root /home/pc/firstsite;
	    }

	    location / {
		include         uwsgi_params;
		uwsgi_pass      unix:/run/uwsgi/firstsite.sock;
	    }
	}

sudo ln -s /etc/nginx/sites-available/firstsite /etc/nginx/sites-enabled

check configuration with

sudo nginx -t

restart nginx

sudo systemctl restart nginx

nginx firewall allow

sudo ufw allow 'Nginx Full'

to start automatically at boot by typing:

sudo systemctl enable nginx 
sudo systemctl enable uwsgi

delete the default page on /etc/nginx/sites-enable/default

sudo rm /etc/nginx/sites-enabled/default

type url for first django site using your ip address, in my case it's

http://35.224.171.124/

to access admin page use

http://35.224.171.124/admin 
username: admin 
password: Pfix0insa

install docker on machine

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - 
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update <br/>

sudo apt install -y docker-ce <br/>

sudo usermod -aG docker ${user} <br/>

replace ${user} with your username
logout and login to affect the above cmd

install rabbitmq using docker

docker run -d --hostname ${hostname} --name my-rabit -p 8080:15672 rabbitmq

replace ${hostname} with your hostname

docker exec -it my-rabit /bin/bash

to enable plugins type

rabbitmq-plugins enable rabbitmq_management

then type exit

now rabbitmq is running and working good u can test with the ff url:

http://35.224.171.124:8080/ 
username: guest 
password:guest

install mysql on server

sudo apt-get update 
sudo apt-get install mysql-server 
sudo systemctl enable mysql

PART 1.2 Create a simple task in django that will be run be run by celery

 git clone https://github.com/demis-svenska/Django-Celery-Example.git 
 cd Django-Celery-Example
 sudo pip install django 
 sudo pip install celery 
 sudo pip install numpy 
 sudo pip install scipy 
 celery -A celery_try worker -l info 
 python manage.py migrate 
 python manage.py runserver 0.0.0.0:8090

Then visit http://35.224.171.124:8090/index/.
I have incountered an error on this while using rabbit docker container and executing the ff code.

celery -A celery_try worker -l info

It displays the following error

[2018-08-23 11:10:03,145: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused. Trying again in 6.00 seconds...

it may take some time to fix it

PART 1.3 Running two apps on same server and using two apps on same broker

Running 2 apps on same server

using VirtualEnv and VirtualEnvWrapper

install virtualenv, which can create Python virtual environments, and virtualenvwrapper, which adds some usability improvements to the virtualenv work flow.

sudo -H pip install virtualenv virtualenvwrapper
echo "export WORKON_HOME=~/Env" >> ~/.bashrc	
echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc	
source ~/.bashrc

to create virtual environment use mkvirtualenv cmd

mkvirtualenv secondsite

This will create a virtual environment, install Python and pip within it, and activate the environment.
i have already created first site on PART 1.1
our prompt will change to indicate that you are now operating within your new virtual environment. It will look something like this: (secondsite)user@hostname:~$

to move out of the virtual environment use:

deactivate cmd

to reactivate the virtual environment use:

workon secondsite

Using 2 apps on same broker

using rabbitmq virtual host

since am using rabbitmq docker machine, i need to login to container and execute the ff cmds.

docker exec -it my-rabit /bin/bash

Create virtual host for first site using cmd

rabbitmqctl add_vhost firstsite

create virtual host for second site using cmd

rabbitmqctl add_vhost secondsite

U can now see the list of vhosts

http://35.224.171.124:8080/#/vhosts

PART 2 Scale

Database

I recommend using ‘Amazon Route 53 Weighted record sets’ to distribute requests across the read replicas.

Create/Find DNS Endpoint for replicas
Create a Route 53 hosted zone 
Create a Record Set

Application server

option1

Nginx server works as reverse proxy as well as balancer.
and recommend that at least one more application sever to be added so that the whole system continues functioning incases of failure in the application servers.

option 2

if possible use Amazon Elastic Container Service for Kubernetes (Amazon EKS), makes it easy to deploy, manage, and scale containerized applications using Kubernetes on AWS.

Async queue

use performance tunning best practice

Configure to support priorities by setting the x-max-priority argument

CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1

tunning Rabbitmq

use RabbitMQ Best Practice for High Performance source: https://www.cloudamqp.com/blog/2018-01-08-part2-rabbitmq-best-practice-for-high-performance.html

Make sure your queues stay short
Set a queue max-length if needed
Remove the policy for lazy queues
Use transit messages
Use multiple queues and consumers
Split your queues over different cores
	RabbitMQ sharding 
Disable manual acks and publish confirms
Disable plugins you are not using

and observe the effect

I need more time to work on celery

demis-svenska / challenge30 Goto Github PK

challenge30's Introduction