Giter Club home page Giter Club logo

What is Difio

"If an application works, any change in its libraries or the versions of those libraries can break the application." - virtualenv

Difio keeps track of packages and tells you when they change. It provides multiple change analytics so you can make an informed decision when or what to upgrade. Difio seeks to minimize the risk of upgrading upstream packages inside your applications.

Difio is a Django based application. For development queries please head to the difio-devel group.

Supported programming languages and package sources

The following programming languages and package sources are supported:

Copyright

Difio was initially developed by Alexander Todorov and Svetlozar Argirov for www.dif.io.

Difio, except where otherwise noted, is released under the Apache License 2.0. See the LICENSE file located in this directory.

How to install

Difio depends on several external components which need to be installed and pre-configured on the server before installing difio/.

  • Django - new or existing project;

  • ClamAV - for anti virus scanning;

  • Web server for JSON content (Apache, Nginx, etc);

  • Database;

  • Messaging layer (RabbitMQ, Amazon SQS, etc);

  • Periodic task scheduler (cron, celerybeat, etc);

  • First install the web application:

      $ django-admin.py startproject mysite
      $ cd mysite/
      $ git clone https://github.com/difio/difio
      $ pip install -r difio/requirements.txt
    

Configure urls.py:

    url(r'^difio/', include('difio.urls')),

Configure settings.py. See the comments inline.

# loads Celery 
import djcelery
djcelery.setup_loader()


# configure the database
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}



# First define apps part of difio/ then the rest of your site
INSTALLED_APPS = (
# NB: ordered first b/c they override admin templates
    'difio',
    'djcelery',
# other apps go below
)



# email configuration for sending notifications
DEFAULT_FROM_EMAIL = '[email protected]'
# EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'



##### Default protocol and domain name settings
FQDN="http://example.com" # optional



#### JSON storage
# this setting is used to write the JSON files containing
# analytics data, which are then loaded by the web page using AJAX
DEFAULT_FILE_STORAGE = 'difio.filestorage.OverwriteFileSystemStorage'
MEDIA_ROOT = "/tmp/example.com/files"    # where to store these files
# Difio needs CORS enabled, for Apache add to your config:
#    Header set Access-Control-Allow-Origin      "*"
#    Header set Access-Control-Allow-Headers     "content-type, x-requested-with"
#    Header set Access-Control-Allow-Methods     "GET"
# See http://enable-cors.org/server_apache.html
MEDIA_URL = "http://example.com/files/"  # where to serve them from



##### STATIC FILES SETTINGS
# STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_ROOT = '/tmp/example.com/static'
STATIC_URL  = '/static/'

# List of finder classes that know how to find static files in various locations.
STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)



TEMPLATE_LOADERS = (
    'django.template.loaders.app_directories.Loader',
)



##### User profiles settings - replace with your own implementation
# with get_email_delay(), is_subscribed() and get_subscription_plan_name()
# methods. See the actual implementation for more details.
AUTH_PROFILE_MODULE = "difio.MockProfile"



##### GitHub auth tokens
# used by github.py for authenticated requests
# unauthenticated requests are severely limited.
# See http://developer.github.com/v3/#rate-limiting
# github.py uses OAuth2 Key/Secret authentication , see
# http://developer.github.com/v3/#authentication
# You will have to register an application with GitHub
# in order to be issued these tokens

# GITHUB_APP_ID                = '00000000000000000000'
# GITHUB_API_SECRET            = '77777777777777777777'



##### RubyGems.org API key
# used for importing packages via RubyGems.org web hooks
# http://guides.rubygems.org/rubygems-org-api/#webhook_methods
# If this is defined importing from RSS will be disabled automatically

# RUBYGEMS_API_KEY = '00000000000000000000000000000000'



##### CELERY MESSAGING SETTINGS
CELERY_DEFAULT_QUEUE = 'difio'
CELERY_QUEUES = {
    CELERY_DEFAULT_QUEUE: {
        'exchange': CELERY_DEFAULT_QUEUE,
        'binding_key': CELERY_DEFAULT_QUEUE,
    }
}

BROKER_USE_SSL = True
BROKER_URL = "amqp://"

CELERY_IGNORE_RESULT = True
CELERY_DISABLE_RATE_LIMITS = True
# using pickle b/c we pass date-time and callback parameters
# make sure to secure access to your message broker
CELERY_ACCEPT_CONTENT = ['pickle']
CELERY_TASK_SERIALIZER = 'pickle'

# ONLY FOR LOCAL DEVELOPMENT UNTILL THERE'S A UNIX SOCKET BROKER
# http://docs.celeryproject.org/en/latest/configuration.html#celery-always-eager
# CELERY_ALWAYS_EAGER = True
# CELERY_EAGER_PROPAGATES_EXCEPTIONS = True



##### CACHE SETTINGS
# NB: change FileBasedCache to what you use or leave unmodified otherwise
# DO NOT use LocMemCache because **cross-process caching is NOT possible**.

CACHES = {
# Used for temporary objects like email hashes
    'default': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION' : '/tmp/example.com/cache/default',
        'TIMEOUT' : 60*60*24*30, # 1 month timeout
    },
# Used to pass larger objects to tasks to avoid hitting message bus size limit
    'taskq': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION' : '/tmp/example.com/cache/taskq',
        'TIMEOUT' : 60*60*24, # 1 day timeout
    },
}
  • Initialize the database schema:

      $ python manage.py syncdb
    
  • Optionally obtain API keys from http://rubygems.org and register a web hook to import new gems:

      $ curl -H 'Authorization:00000000000000000000' \
             -F 'gem_name=*' -F 'url=http://example.com/difio/hook/rubygems/' \
             https://rubygems.org/api/v1/web_hooks
    
  • Configrue a periodic task scheduler to execute some of the maintenance tasks at regular intervals. Following is a list of tasks and execution frequencies

      # daily tasks
      difio.tasks.cron_notify_app_owners_1            # send daily email notifications
    
      # flexible interval tasks:
      # - depend on how often do you want to query upstream;
      # - depend on available computing resources (less frequent execution, less resources needed)
      difio.tasks.cron_delete_pending_apps            # deletes apps which were not approved
      difio.tasks.cron_import_new_versions_from_rss   # imports new versions from upstream RSS feeds
      difio.tasks.cron_find_new_versions              # alternatively query upstream for the latest version
      difio.tasks.cron_generate_advisory_files        # generate analytics report (aka Advisory)
      difio.tasks.cron_move_advisories_to_live        # everything in state PUSHED_LIVE becomes LIVE
    

The following script may be used as a cron helper:

#!/bin/bash
#
# Copyright (c) 2012-2013, Alexander Todorov <[email protected]>
#
# SYNOPSIS
#
# ./run_task module.tasks.task_name
#
# OR
#
# ln -s run_task module.tasks.task_name
#

TASK_NAME=$1
[ -z "$TASK_NAME" ] && TASK_NAME=$(basename $0)
MODULE_NAME=`echo "$TASK_NAME" | rev | cut -f2- -d. | rev`

source ~/.virtualenvs/$CHANGE_ME/bin/activate
APP_DIR="~/$CHANGE_ME/app"

echo "import $MODULE_NAME; $TASK_NAME.delay()" | $APP_DIR/manage.py shell

In case you decide to use Celerybeat see the Periodic Tasks document.

  • After the web application is installed Difio will start recording information from upstream sources as well as user input. It will compare currently listed package versions in the DB with the latest available and schedule analytics actions. This is done in long-running background tasks using Celery. These are called workers in Celery terms. A worker consists of the same application code, including same settings.py and configuration to start the worker daemon. Please see Workers Guide and Running the worker as a daemon for more information.

Maintenance

  • Most of Difio operations are automated but analytics content needs to be verified by person before it is published. This is to eliminate possible errors and account for cases which were not automated. See Content Administration Guide for more details;

Warnings

In case you see problems with pycurl and SSL support check these resources:

Building the sources locally (with --libs instead of --static-libs) and installing may be used in some cases!

difio's Projects

difio icon difio

Open source software change analyzer

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.