Giter Club home page Giter Club logo

puppet-catalog-diff-viewer's Introduction

Puppet Catalog Diff Viewer

License CI Donated by Camptocamp

A viewer for json reports produced by the puppet-catalog-diff tool

The interface can be tried online at http://voxpupuli.org/puppet-catalog-diff-viewer.

Acking diffs

This interface allows to ack differences in order to ease the report review. Differences are acked on all nodes that have the same resource with the same diff.

A global acking button is available for sections which allows to ack all diffs in the section at once.

Starring diffs

When using the global acking button, you might want to exclude some diffs from the global acking. Starring diffs does just that. Just as for acks, stars are cross-nodes.

Using with Docker

$ docker run -it --rm \
  -p 8080:80 \
  ghcr.io/voxpupuli/puppet-catalog-diff-viewer:development

will let you access the catalog diff viewer at http://localhost:8080.

Server Side storage

The will automatically populate the drop-down list of available reports, if they can be read from reportlist.json. This file contains a record of the json files in data. Assuming you have

data/
  file1.json
  file2.json

the reportlist.json should have the format

{
  "First Report": "file1",
  "Second Report": "file2"
}

The python script generate_reportlist.py will autopopulate it with all data/*.json using the filename also as key.

With the docker image, you can put everything in /data:

$ docker run -ti \
   -v ./data:/data \
   -p 8080:8080 \
   ghcr.io/voxpupuli/puppet-catalog-diff-viewer

S3 storage

The viewer can automatically retrieve catalogs from an S3 bucket. In order to use this feature, create a s3_credentials.js file with the following variables:

const s3_bucketName = 'your-bucket-name';
const s3_access_key = 'your-access-key';
const s3_secret_key = 'your-secret-key';
// something like this for standard s3 and the corresponding region
const s3_endpoint   = 's3.us-east-2.amazonaws.com';
// for self hosted s3
const s3_endpoint   = 'http://your.endpoint.example.com:9000';

// if you use a path within the bucket
const s3_bucketPathPrefix = 'your-prefix';
const s3_ForcePathStyle = true;

With the docker image, you can use:

$ docker run -it --rm \
   -v ./s3_credentials.js:/data/s3_credentials.js:ro \
   -p 8080:8080 \
   ghcr.io/voxpupuli/puppet-catalog-diff-viewer:development

or using environment variables:

$ docker run -it --rm \
   -e S3_BUCKET=your-bucket-name \
   -e S3_ACCESS_KEY=your-access-key \
   -e S3_SECRET_KEY=your-secret-key \
   -p 8080:8080 \
   ghcr.io/voxpupuli/puppet-catalog-diff-viewer:development

Make sure the access key belongs to a user that can perform actions s3:GetObject and s3:ListBucket on the bucket. Here is an example bucket policy you can use to upload files from the catalog-diff machine and retrieve them in the viewer:

{
  "Version": "2012-10-17",
  "Id": "Policy1451988974568",
  "Statement": [
    {
      "Sid": "Upload",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:user/uploader"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::bucket-id/my-site/*"
    },
    {
      "Sid": "ViewerList",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:user/viewer"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::bucket-id"
    },
    {
      "Sid": "ViewerGet",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:user/viewer"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::bucket-id/*"
    }
  ]
}

Contributing

Build a release

see RELEASE.md

Transfer Notice

This project was originally authored by Camptocamp. The maintainer preferred that Vox Pupuli take ownership of the project for future improvement and maintenance. Existing pull requests and issues were transferred over, please fork and continue to contribute here instead of Camptocamp.

puppet-catalog-diff-viewer's People

Contributors

alexjfisher avatar bastelfreak avatar claytono avatar davidsandilands avatar dependabot[bot] avatar msalway avatar raphink avatar rwaffen avatar saimonn avatar schildwaechter avatar simonhoenscheid avatar smortex avatar tuxmea avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppet-catalog-diff-viewer's Issues

Port to AngularJS

Expected gain:

  • Simpler code, using templates
  • Maybe a perf gain (from templates)
  • Routing, allowing for shareable URLs, such as /#/my_report/changes/foo.example.com/diff/dfere4358df

Routes

Here are the routes that need to be implemented:

  • /:report: load a report
  • /:report/failed: display failed nodes
  • /:report/failed/:node: display one failed node
  • /:report/changes: display nodes with changes
  • /:report/changes/:node: display one node with changes
  • /:report/changes/:node/diff: jump to diff panel
  • /:report/changes/:node/diff/:resource: jump to a resource in the diff panel
  • /:report/changes/:node/in-old: jump to in-old panel
  • /:report/changes/:node/in-old/:resource: jump to a resource in the in-old panel
  • /:report/changes/:node/in-new: jump to in-new panel
  • /:report/changes/:node/in-new/:resource: jump to a resource in the in-new panel
  • /starred (see #14)
  • /acked (see #14)

Resource IDs

Resources will need to be identified uniquely within a section (diff, in-old, in-new). There resource full name (type[title]) is sure to be unique, but not very easy to pass in a hash (it can contain spaces and special characters), so we might want to hash it:

  • URL encoding might do it, but it won't look very good in the URL
  • Base64 it:
> btoa("filesystem[/dev/mapper/vg0-pg_backups]")
"ZmlsZXN5c3RlbVsvZGV2L21hcHBlci92ZzAtcGdfYmFja3Vwc10="
  • using a hashing function, such as hashCode() or murmur (or https://github.com/garycourt/murmurhash-js, faster but only supports 32 bit hashes), might be nicer if it returns unique IDs. Several problems with hashCode():
    • it can return negative numbers => apply Math.abs() to it. This doubles the chances of collisions, but they are still quite unlikely
    • it returns only numbers => apply .toString(36) to it.

In short, for example:

> murmurhash3_32_gc("file[/etc/bind/dynamic/compute.camptocamp.com.conf]").toString(36)
"xagm9b"

If we have a nice, safe hashing function, we might actually want to use it internally for resource referrers, instead of trying to escape the full resource names.

Templates

Several templates are required:

  • chart.html: report chart template, generating the d3js markup
  • nodes_list.html: nodes list template, with links to node routes
  • node.html: node details template

Caching

As the global variables will be reimplemented in the $scope, it would be good to consider caching all the reports in there, and having a button to refresh reports. This way, reports and acks/stars will be kept in memory for all reports during the time of the session.

Redirect to http://localhost for http://localhost:8080 docker usage

I want to try your software, but when running

docker run -ti -p 8080:80 camptocamp/puppet-catalog-diff-viewer

I'll just get a redirect to http://localhost, where of nothing is listening of course.

 $ docker run -ti -p 8080:80 camptocamp/puppet-catalog-diff-viewer
172.17.0.1 - - [28/Sep/2016:08:48:05 +0000] "HEAD /catalog_diff HTTP/1.1" 301 0 "-" "curl/7.43.0" "-"

$ curl -I http://localhost:8080/catalog_diff
HTTP/1.1 301 Moved Permanently
Server: nginx/1.11.0
Date: Wed, 28 Sep 2016 08:48:05 GMT
Content-Type: text/html
Content-Length: 185
Location: http://localhost/catalog_diff/
Connection: keep-alive

What am I doing wrong and where to I need to put the local catalogs? I cannot find it in the README.

Viewer shows only changed nodes

From the folllowing json, only the changed node shows up in the viewer, the unchanged node is suppressed:

{
    "box1": {
        "added_and_removed_resources": "0 / 0",
        "catalag_percentage_added": "0.00",
        "catalog_percentage_changed": "0.00",
        "catalog_percentage_removed": "0.00",
        "content_differences": {},
        "differences_as_diff": {},
        "differences_in_new": {},
        "differences_in_old": {},
        "new_version": 1452012864,
        "node_differences": 0,
        "node_percentage": 0.0,
        "old_version": 1452012838,
        "only_in_new": [],
        "only_in_old": [],
        "params_in_new": {},
        "params_in_old": {},
        "total_resources_in_new": 4,
        "total_resources_in_old": 4
    },
    "box2": {
        "added_and_removed_resources": "+1 / 0",
        "catalag_percentage_added": "20.00",
        "catalog_percentage_changed": "0.00",
        "catalog_percentage_removed": "0.00",
        "content_differences": {},
        "differences_as_diff": {},
        "differences_in_new": {},
        "differences_in_old": {},
        "new_version": 1452012858,
        "node_differences": 1,
        "node_percentage": 20.0,
        "old_version": 1452012846,
        "only_in_new": [
            "package[git]"
        ],
        "only_in_old": [],
        "params_in_new": {},
        "params_in_old": {},
        "total_resources_in_new": 5,
        "total_resources_in_old": 4
    },
    "date": "2016-01-05 17:56:02 +0100",
    "most_changed": [
        {
            "box2": 20.0
        }
    ],
    "most_differences": [
        {
            "box2": 1
        }
    ],
    "total_nodes": 2,
    "total_percentage": 10.0,
    "with_changes": 1
}

They catalogs and diff were created with puppet 3.8.4 on Ubuntu 14.04 using latest acidprime/puppet-catalog-diff.

Also, the date format is not supported by the viewer.

Update the Docker image

Hi guys,

Could you kindly consider doing whatever is necessary to publish the latest version of the Docker image

The current one is 2 years old and is missing some features added.

Thanks in advance!

Clear host list when selecting empty category of hosts

Using the minimal example from #19, select either the changed or unchanged hosts.
Than select failed.
The list of hosts should be empty now, but the previous hosts remain, allowing to navigate to e.g. /failed/box2 which is wrong.

Add setup documentation to README

Can you when you have a chance add some basic installation installation information , my guess if you are generating on schedule via cron, so peut-être the example puppet cron code or just assumptions like file paths and such. :)

S3 reports cannot be read

even if s3_credentials.js is supplied, the reports are not read from s3 bucket.
in the logs you see it searches still for reportlist.json instead of listing files from the bucket.

improve layout

Use a layout with:

  • fixed header and footer (position: absolute)
  • two columns 4/8, with:
    • left column with two fixed div, the bottom one with overflow auto
    • right column fixed with overflow auto

Share marks between reports

Acks and stars are already shared between nodes of a report. It might be interesting to share them between reports also (i.e. make them global).

This way, when reviewing e.g. module updates reports, we could see resources that change on several infrastructures/puppet masters using stars.

reportlist.json is not generated

how should reportlist.json be generated in the container?
it is somehow generated with generate_reportlist.py, but this script is never run in the container? and there is no python in the container. how should this be triggered if someone uploads a new report?

Escape double quotes in keys when acking diffs

Otherwise, they might not be ackable.

Example:

Uncaught Error: Syntax error, unrecognized expression: [id="diff:postgresql_psql[ALTER ROLE "deploy" NOCREATEDB]"]
jquery-1.11.3.min.js:2 Uncaught Error: Syntax error, unrecognized expression: [id="diff:postgresql_psql[ALTER ROLE "deploy" NOCREATEROLE]"]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.