jcalazan / ansible-django-stack Goto Github PK

I think, we need python3 support.

may be something like that (not tested)

vars:

#python3
python_version="3"
#python2
python_version="" 

task (install):

- python{{python_version}}
- python{{python_version}}-pip
- python{{python_version}}-pycurl
- python{{python_version}}-psycopg2
etc

and create virtualenv with

virtualenv -p python{{python_version}}

In the env_vars files there is a variable called ssl_key_password but I can't figure out what it does.
I've been looking through the rest of the files(grepping for ssl_key_password) and I can't find it in use anywhere.
Is it actually required?

Hi, I've used your playbook (awesome, btw) for deploying an app successfully but then I added a new dependency to requirements.txt, and imported this new python module on my settings. It seems that if you run the setup_supervisor task before the setup_django_app task, the supervisor restart gets broken and the following message ends up on logs:

Traceback (most recent call last):
File "/webapps/dispatcher/bin/gunicorn", line 11, in
sys.exit(run())
File "/webapps/dispatcher/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/webapps/dispatcher/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 185, in run
super(Application, self).run()
File "/webapps/dispatcher/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 71, in run
Arbiter(self).run()
File "/webapps/dispatcher/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 196, in run
self.halt(reason=inst.reason, exit_status=inst.exit_status)
File "/webapps/dispatcher/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 292, in halt
self.stop()
File "/webapps/dispatcher/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 343, in stop
time.sleep(0.1)
File "/webapps/dispatcher/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 209, in handle_chld
self.reap_workers()
File "/webapps/dispatcher/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 459, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

I've changed the order of the setup_django_app and setup_supervisor (so that new requirements are installed first) and it worked.

I ran into the following when I attempted to use this playbook. The repository pointed to doesn't exist and I am not able to find a suitable replacement.

TASK: [web | Add a custom repository for ffmpeg] ****************************** 
failed: [default] => {"failed": true, "parsed": false}
BECOME-SUCCESS-kipofcbjoqsbuaicrsiwoqujiigbueas
Traceback (most recent call last):
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1434537365.12-129649650050245/apt_repository", line 2524, in <module>
    main()
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1434537365.12-129649650050245/apt_repository", line 436, in main
    cache.update()
  File "/usr/lib/python2.7/dist-packages/apt/deprecation.py", line 98, in deprecated_function
    return func(*args, **kwds)
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 418, in update
    raise FetchFailedException(e)
apt.cache.FetchFailedException: W:Failed to fetch http://ppa.launchpad.net/jon-severinsson/ffmpeg/ubuntu/dists/precise/main/binary-amd64/Packages  404  Not Found
, W:Failed to fetch http://ppa.launchpad.net/jon-severinsson/ffmpeg/ubuntu/dists/precise/main/binary-i386/Packages  404  Not Found
, E:Some index files failed to download. They have been ignored, or old ones used instead.

Hi, just wondering if you have thoughts on best way to get this to work if you're pulling from a private repo. My initial thought is:

1. create a non-passphased ssh key pair just for github
2. add it to the github admin
3. within roles/web/tasks/setup_git_repo.yml have ansible add private key to the user account on the target machine.

Does that sound like an ok plan?

Hi

I've added SECRET_KEY variable in web/template/virtualenv_postactivate.j2 file. But I am still getting an error.

virtualenv_postactivate.j2 file

#!/bin/sh
export SECRET_KEY='xxxxxxxx'

{% for variable_name, value in django_environment.iteritems() %}
export {{ variable_name }}="{{ value }}"
{% endfor %}

Error

    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

Can you please help?
Thanks

Mainly the Playbooks:

• rename production.yml to site.yml
• always use inventory files
• to run the playbook against production servers for example, we can do ansible-playbook -i production site.yml

Where site.yml has includes to group-specific playbooks (such as webservers.yml and dbservers.yml with the host: option in these playbooks set to the proper group).

Currently installing the latest version from the default distro apt repo.

Add a task to include the official PostgreSQL apt repo before the install.

What's new in 9.5? https://www.postgresql.org/about/news/1636/

Just like the memcached role uses a handler to restart the service in case the config file changes, it would be good practice for the nginx role to do the same. Also, since both packages support it, they should just use reload instead of restart.

Hi there,

Thank you for making this thorough playbook available. It's been an incredible asset as I've figured out how to deploy a Django app.

I am, however, having an issue: the "Run Django collectstatic" task in the playbook continues to fail with the error:

You're using the staticfiles app without having set the STATIC_ROOT setting to a filesystem path.

It seems like an intuitive enough error message, but I can't seem to diagnose the problem:

• My STATIC_ROOT variable is defined exactly the way it is in your repo:

  STATIC_ROOT: "{{ nginx_static_dir }}"

• I added a task as follows to sanity check that the env variable is set:

   - name: Sanity check env vars
   shell: echo $STATIC_ROOT
   environment: django_environment

That task correctly outputs the path "/webapps/appname/static/"

• I even modified manage.py to output the env variable so it definitely knows that it's set and what it's set to. This (again) returns the correct path as expected.
• Finally, I tried setting the variable in my Django settings file instead. If I do that, then the collectstatic command works correctly and my static resources are loaded correctly.

At this point, I'm totally stumped on why I get the error message when I use the playbook configuration as opposed to setting this env variable in the Django settings directly; I can only assume the error message Django is returning is incorrect, but I don't know why.

Any ideas?

http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

Should probably add a var to check as well before including it in the config: enable_nginx_limit_req

Could you add a license for this project, please?

I would be in favor of MIT, but it's your choice.

I "stacked" both the db and web roles on an Ubuntu 14.04 installation. On browsing to the IP address of the board, I noticed that Apache served the page instead of nginx. I killed the Apache service and restarted the nginx server and all was OK (well, mostly - more on that in another issue).

I don't know if a running Apache instance is the default on all Ubuntu installations or if it was just my particular instance, but checking for any running services that may conflict with our installation (i.e. ports 80, 443) sounds like a good thing.

I have got issue
ansible/ansible#9556

(ansible 1.7.2 from Debian Jessie)

may be, better use files as source of certificates?

This is useful during those times when you deleted a package and the remaining .pyc files could break the app.

A good spot to do this is right after pulling the latest code from the git repo.

find . -name “*.pyc” -delete

Your ssl private key is visible In this file: https://github.com/jcalazan/ansible-django-stack/blob/master/env_vars/base.yml

Assuming this isn't what you use on your site, correct? What's the rational for having that in your public repo?

Use the new keywords, follow updated best practices, etc.

In ansible-django-stack/roles/web/templates/gunicorn_start.j2, according to the README directory structure, shouldn't the last line: {{ application_name }}.wsgi be {{ project_name }}.wsgi? (since wsgi is in project path, not apps dir)

@jcalazan have you considered adding backup and restore scripts? In particular one that also cleans the backup directory so that it doesn't become too large would be ideal. Let me know if helpful to add a PR? I'm not sure what your perspective on having too many roles in this ansible playbook is. I'm interested in adding various tasks that I use on my personal projects to have community code review and make sure I'm not doing anything silly :).

I tried hard to make the playbook working with Ubuntu 16.04, but Gunicorn wasn't cooperating and I couldn't figure it the problem from log. It will be nice if someone can give it a shot. One tiny advantage is that 16.04 is LTS and it comes with Python 3.5 by default.

Hi, manage.py run_gunicorn is already deprecated and the Django recommended way of running gunicorn is

gunicorn myproject.wsgi:application

https://docs.djangoproject.com/en/1.7/howto/deployment/wsgi/gunicorn/

So you can run it with gunicorn v19.

Great playbooks BTW!
Mabuhay.. 👍

Add RabbitMQ role with the web management plugin enabled.

This seems to be a pretty popular alternative to RabbitMQ as a message broker for Celery as it's simpler and can be used for other things.

I think, "Install Supervisor" task must not to be in celery role.
I don't use celery, and got error when config gunicorn with supervisor, because supervisor is not installed.

may be it must place in base or seprate role?

Usually I use a slightly modified nginx.conf like the last example in this stackoverflow answer. This way nginx redirects to https and the non-www version of my url.

Do you do something different, i.e., maybe using a CNAME DNS record? I'm not sure what the best approach is - would love to hear your opinion. I'd also be happy to submit a PR for a modified nginx.conf that includes the redirect to non-www (or perhaps either way with a flag) if that's helpful.

https://github.com/jcalazan/ansible-django-stack/blob/master/roles/web/vars/main.yml#L31

DATABASE_USER: "{{ db_user }}"
DATABASE_PASSWORD: "{{ db_password }}"

may be, we need to add

DATABASE_NAME: "{{ db_name }}"

?

Any ideas what might be causing it?

==> default: TASK: [web | Create the virtualenv postactivate script to set environment variables] ***
==> default: fatal: [localhost] => {'msg': "One or more undefined variables: 'omit' is undefined", 'failed': True}
==> default: fatal: [localhost] => {'msg': "One or more undefined variables: 'omit' is undefined", 'failed': True}
==> default:
==> default: FATAL: all hosts have already failed -- aborting
==> default:

Tanx

Hi,

When provisioning for second time, the task 'Setup the git repo' is failing with the following error:

failed: [default] => {"cmd": "/usr/bin/git ls-remote '' -h refs/heads/HEAD", "failed": true, "rc": 128}
stderr: Warning: Permanently added the RSA host key for IP address '131.103.20.168' to the list of known hosts.
conq: repository does not exist.
fatal: The remote end hung up unexpectedly

If I go on the VM to /webapps/ and rm -rf myapp/ then it works.

Any idea why this might be?

Regards

Really nice to have when you have multiple projects to manage.

For digital nomads, it's also a good idea to run your deployment scripts from a reliable VPS instead of running it from your laptop when you're working somewhere with unreliable connection.

Ansible Tower is too expensive :(

Currently user of nginx is default:

f.e. debian nginx.conf:
user = www-data

So, when nginx try to access 500.html

2016/01/18 13:02:09 [error] 10841#0: *12 open() "/srv/myapp/app-project/myapp/templates/500.html" failed (13: Permission denied)

May be we must change nginx user to
user = {{gunicorn_user}}

?

http://dpaste.com/3354G9X

This, essentially.

I'm concerned that I don't understand the names (project_name, application_name) and that somehow I'm creating a conflict there.

Provisioning always errors on the Restart Supervisor task.

http://dpaste.com/2M6BD66

Any help? What else can I provide? My repository is here:
https://github.com/mekhami/npdhub

That includes my modified provisioning playbooks.

Hi, great work whit this! Can HA be configured for PostgreSQL? Like replication? Hot Standby? What about for other services?

Thank you very much!

I haven't used the playbook directly, but it's been incredibly useful to learn from.

I think there's an inconsistency with naming...

The nginx_site_config.j2 template refers to {{ virtualenv_path }}/maintenance_on.html.

But the setup_virtualenv task moves the maintenance_off.html template to {{ virtualenv_path }}/maintenance_off.html.

Should the latter task use maintenance_ON rather than OFF?

Or have I misunderstood?

Add a var with a default value of python2.7 and can be replaced with python3.

# roles/web/defaults.yml
virtualenv_python_version: python2.7

In the task the create the virutalenv:

virtualenv -p {{ virtualenv_python_version }} {{ virtualenv_path }} --no-site-packages

Add role to run celery workers and celery camera in the background managed my supervisor.

Hello,

Impossible to deploy my models with migrate command.

application_name
|- manage.py
|- project
|--- settings.py
|--- models
|----- init.py
|----- object1.py
|----- object2.py

- name: Run Django database migrations
  django_manage:
    command: migrate
    app_path: "{{ project_path }}"
    virtualenv: "{{ virtualenv_path }}"
  environment:
    PATH: "{{ virtualenv_path }}/bin:{{ lookup('env', 'PATH') }}"
  when: run_django_db_migrations is defined and run_django_db_migrations
  tags: django.migrate

Same problem when i want migrate directly on the server.
After activate virtualenv, i try ./manage.py makemigrations project and get Permission denied error message to create migrations repository.

I have tried to change permissions on all files but nothing change.
Finally i have success to process makemigrations command like that
sudo /virtualenv_path/bin/python ./manage.py makemigrations project
But this not the goal to do this like that.

Any help will be appreciated.
Regards

pip freeze inside the virtualenv :
amqp==1.4.9
anyjson==0.3.3
billiard==3.3.0.23
celery==3.1.23
Django==1.9.8
django-push-notifications==1.4.1
djangorestframework==3.4.0
djoser==0.5.0
gunicorn==19.6.0
kombu==3.0.35
psycopg2==2.6.2
pytz==2016.6.1
Unidecode==0.4.19

This is more of a question on how to use this project rather than a specific issue.

This playbook works great on a local Vagrant VM but I'm not sure how to use it on a cloud provider[1]. Here is what I've tried:

• I setup four servers: webserver-1, webserver-2, dbserver, and memcachedserver (mirroring the server inventory in local)
• I created an inventory file called development which pointed to these 4 servers under the appropriate sections
• I ran: ansible-playbook -i development -v development.yml

As it's running, I notice all roles being applied to all servers (and this seems to be reflected in deployment.yml that lists hosts as all and mentions all roles). How do I go about ensuring that only the base + appropriate role is applied to the relevant server? Or have I misunderstood this entirely?

And thank you for this excellent project, I've learnt quite a bit from it as I've been learning Ansible and using it for my own Django apps.

[1] I'm attempting this with Google Cloud, but that's not relevant to the question.

Hello

Playbook runs without errors and creates postactivate file, but it doesn't actually export environmental variables. I notice this when I try to create superuser manually and always get a error that it can't connect to database because no credentials supplied.

Task for postactivate

- name: Create the virtualenv postactivate script to set environment variables
  template: src=virtualenv_postactivate.j2
            dest={{ virtualenv_path }}/bin/postactivate
            owner={{ gunicorn_user }}
            group={{ gunicorn_group }}
            mode=0640
            backup=yes
  tags: deploy

Environmental variables

django_environment:
  DJANGO_SETTINGS_MODULE: "{{ django_settings_file }}"
  DJANGO_SECRET_KEY: "{{ django_secret_key }}"
  MEDIA_ROOT: "{{ nginx_media_dir }}"
  STATIC_ROOT: "{{ nginx_static_dir }}"
  DATABASE_NAME: "{{ db_name }}"
  DATABASE_USER: "{{ db_user }}"
  DATABASE_PASSWORD: "{{ db_pass }}"

Database settings

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': os.getenv('DATABASE_NAME'),
        'USER': os.getenv('DATABASE_USER'),
        'PASSWORD': os.getenv('DATABASE_PASSWORD'),
        'HOST': os.getenv('DATABASE_HOST'),
        # 'PORT': os.getenv('DATABASE_PORT', '5432'),
    }
}

How can I ensure that django variables exported?

The web role has:

# Set this to 0 for unlimited requests. During development, you might want to
# set this to 1 to automatically restart the process on each request (i.e. your
# code will be reloaded on every request).
MAX_REQUESTS={{ gunicorn_max_requests }}

Using this current scheme:

- name: Run the Django syncdb command
  django_manage:
    command: syncdb
    app_path: '{{ application_path }}'
    virtualenv: '{{ virtualenv_path }}'
    settings: '{{ django_settings_file }}'
  environment: django_environment
  when: run_django_syncdb
  tags: django

- name: Run Django South migrations
  django_manage:
    command: migrate
    app_path: '{{ application_path }}'
    virtualenv: '{{ virtualenv_path }}'
    settings: '{{ django_settings_file }}'
  environment: django_environment
  when: run_django_south_migration
  tags: django

- name: Run Django collectstatic
  django_manage:
    command: collectstatic
    app_path: '{{ application_path }}'
    virtualenv: '{{ virtualenv_path }}'
    settings: '{{ django_settings_file }}'
  environment: django_environment
  when: run_django_collectstatic
  tags: django

Is there any way one could run just one of these tasks from the command line? The following command runs all tasks with true values for the when clause as determined by the environment:

$ ansible-playbook development.yml --tags "django"

For example, it would be nice to migrate the app without running django's syncdb:

$ ansible-playbook development.yml --tags "django_migrate"

If I'm missing a way to do this without modifying the tags, please let me know. I can see the logic of being able to group tasks.. I just want to conditionally run it via the command line, and was hoping there was a way around verbosely supplying lots of variable overrides as migrations are much more common than syncdb, etc.

Hi there,

Has someone encountered this error while provisioning?:

On the task:

- name: Ensure database is created
  sudo_user: postgres
  postgresql_db: name={{ db_name }}
                 encoding='UTF-8'
                 lc_collate='en_US.UTF-8'
                 lc_ctype='en_US.UTF-8'
                 template='template0'
                 state=present

Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql>/.s.PGSQL.5432"

While doing some tests on the local machine I was able to overcome this temporarly by logging in, doing apt-get remove --purge postgres and then installing postgres-9.1 in the task Install PostgreSQL, notice the version instead.

However this is not reliable, Have you guys encountered this error?

Cheers!

Ran into an issue recently where pip-installing a package used up all 512MB of RAM on my VPS (lxml module).

• in base/roles
• look for var create_swap_file is true before running the task
• other vars
  • swap_file_path (default /swapfile)
  • swap_file_size_kb (default 1024, note that block size is 1024, so this would create a 1GB swap)

Hi - great playbook, thanks for building!

Have you thought about incorporating some basic server security tasks from something like http://www.codelitt.com/blog/my-first-10-minutes-on-a-server-primer-for-securing-ubuntu/

I'd be happy to make a PR if that's helpful. I'd also be interested to hear if there's another fork that has already done this or if there's a reason it hasn't been included.

I am using ansible 2.0.0.2, and I am receiving this error during provisioning:

TASK [web : Run Django database migrations] ************************************
[DEPRECATION WARNING]: Using bare variables for environment is deprecated.
Update your playbooks so that the environment value uses the full variable
syntax ('{{foo}}'). This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.
fatal: [default]: FAILED! => {"failed": true, "msg": "ERROR! environment must be a dictionary, received django_environment (<class 'ansible.parsing.yaml.objects.AnsibleUnicode'>)"}

Use this PPA: http://askubuntu.com/questions/682869/how-do-i-install-newer-python-versions-using-apt-get/682875

By default, only v3.4 is installed on Ubuntu 14.04

Vagrant provision fails because of newlines in the SSL certs. Removing them on the VM works, but I'm not sure how to get ansible to not have them.

Also add an option to include a cron job to auto-renew the cert.

Currently thinking of doing something like:

(before Nginx installation)

1. Install certbot
2. Run standalone command, pull fqdn from an env var.
3. In the Nginx template, if option nginx_use_lets_encrypt is true, simply reference the cert and key to the default location created by certbot.
4. Cron job to auto-renew monthly (cert is good for 3 months).

If someone has done this before, would love to hear your solution (or open a pull request :D)!

Hi,

I got an issue on:

TASK [web : Install packages required by the Django app inside virtualenv] ***** task path: /Users/username/vagrant/ansible-django-stack/roles/web/tasks/setup_django_app.yml:3 fatal: [default]: FAILED! => {"changed": false, "cmd": "/webapps/testdjango/bin/pip install -r /webapps/testdjango/test-django/requirements.txt", "failed": true, "msg": "\n:stderr: Could not open requirements file: [Errno 2] No such file or directory: '/webapps/testdjango/test-django/requirements.txt'\n"}

by the way thank you for providing the set up.

After resolving #40, I was able to load the page. However, I noticed that no CSS files had been loaded. This was because I had defined STATIC_ROOT in my settings.py file and the value differed from the location used in configuring nginx. It would be good to provide a little guidance about which variables should/shouldn't be declared in settings.

An alternative would be to parse the value in the provided settings file, but this would obviously involve a lot more work.