Giter Club home page Giter Club logo

cog's Introduction

Cog

Build Status Ebert

Cog brings the power of the command line to the place you collaborate with your team all the time -- your chat window.

Powerful access control means you can collaborate around even the most sensitive tasks with confidence. A focus on extensibility and adaptability means that you can respond quickly to the unexpected, without your team losing visibility.

  • Installation Guide - You want Cog, now go here to start installing it.
  • Development Guide
  • Cog Book - This is Cog's Documentation! Our new book that includes the above mentioned Installation Guide and a chapter on Command Bundle Writing.
  • Bundle Warehouse - Install one of these bundles on Cog and/or create your own command bundle and upload it here.

Status

Cog is production ready! ๐ŸŽ‰

More Links

cog's People

Contributors

christophermaier avatar davejlong avatar drapergeek avatar hershi avatar imbriaco avatar jsteiner avatar justinkinney avatar ktheory avatar lhaskins avatar matusf avatar mpeck avatar nmohoric avatar ohaiwalt avatar operable-eng avatar shelton avatar technicalpickles avatar tecnobrat avatar vanstee avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cog's Issues

JSON aware filter

Example:

filter --field="username" --return="username,first_name,last_name" /^kev/

Create chat handles via cogctl

In dev setup, we create users and then create chat handles for those users. With cogctl we can create users but we don't have a way to add chat handles for those users. Let's add that.

Bundle uninstall via relayctl

This is an obsolete design. See comments below for more recent info.

Trigger uninstalling a bundle from Cog and all Relays.

Examples:

cogctl bundle rm --bundle=BFC92057-B4AB-43B0-9E4C-D044AEDDCD78

Cache HipChat user and room information, allowing use of rooms with uppercase names

NOTE: this is based on Hedwig 0.3.0, but a new version is coming out soon with a markedly different codebase

Currently when we receive a message from the Hipchat adapter, it looks something like this:

%Hedwig.Stanzas.Message{body: "@marvin help",
                        client: #PID<0.775.0>,
                        delayed?: false,
                        from: %Hedwig.JID{resource: "Christopher Maier",
                                          server: "conf.hipchat.com",
                                          user: "123456_bot-testing"},
                        html: nil,
                        id: "",
                        matches: nil,
                        payload: [%{name: "body",
                                    payload: ["@marvin help"]}],
                        to: %Hedwig.JID{resource: "<big long proxy string>",
                                        server: "chat.hipchat.com",
                                        user: "123456_1234567"},
                        type: "groupchat"}     

Currently, we've been slicing up the from.user string to extract a room name (above, this would be "bot-testing"). We then resolve the ID of this room by using the Hipchat V2 API call /v2/room/bot-testing (see https://www.hipchat.com/docs/apiv2/method/get_room)

However, if the room name begins with an uppercase letter (e.g. "Operable"), the sliced-out version of the name will be "operable" (note lower-case). Making an API call with this name will return a 404.

Instead what we'll want to do is to start caching both room and user information, just like we do in the Slack adapter. Among the data returned from the Hipchat API for a room is the xmppp_jid, which for our room above would be "[email protected]" (that is,"#{from.user}@#{from.server}", or Hedwig.JID.bare(jid) (see
https://github.com/hedwig-im/hedwig/blob/e7292ecab53ce119dd251f8378fdcf7facf037a3/lib/hedwig/jid.ex#L50-L62)

We can use this "bare JID" to query our room cache to retrieve a room record, and from there obtain the room ID. This will allow the bot to post back to rooms with upppercase names :)

DM redirects are buggy

DM redirects work if the bot and the target user have previously communicated via DM. If not, then output is not sent and the pipeline executor will timeout. Some users have reported executors locking up and/or crashing but these have been hard to repro. @christophermaier has a fix.

cogctl writes error messages to stderr

We need to make a pass thru cogctl's code to ensure all error messages are sent to stderr and not stdout.

tl;dr Do thisIO.puts(:stderr, "Oh noes!") not this IO.puts("Oh noes!").

Document log handling behavior

Need to fully document TolerantFile behavior:

  • Describe file lifecycle and stat() calls
  • Explain behavior when log file is moved
  • Describe recommended approach for log rotation (ie: just move the file, cog will make a new one)

Building Cog & Relay instructions

  • Development tool prerequisites:
    • Erlang R18B01 or later
    • Elixir 1.2 or later
    • Postgres 9.4 or later
  • Getting the source for Cog and Relay
  • Building Cog and Relay
  • Starting it up for the first time
  • Pointers to:
    • Design docs
    • "How to contribute" docs

Structured logging

Main Idea: Log output for Loop, Spanner, Relay, and Carrier should be JSON formatted with standardized documented field names.

Does `-` denote flag or negative number?

Currently, the - is parsed as a flag indicator, but it can also mean negative number. We need to be able to parse our commands such that they will know the difference.

In other words, if a command does not have an option defined as -<some number>, then treat it as a negative number.

Example:
sum 2 -9 should be interpreted as 2 + -9 We currently have to say sum 2 "-9" in order for -9 to not be interpreted as a flag.

Hosting docs

We need to determine where we're going to host docs which span multiple components, such as the design guide. Do we want to centralize them in Cog's wiki, keep them as markdown files in Cog's repo, or set up a separate site?

Injecting command configuration

Main idea: gen_command "automagically" injects command config on each request. Config is automatically reloaded whenever it changes. This frees command authors from having to think about how to manage dynamic config. Note: gen_command is shorthand for "command hosting infrastructure of which commands are blissfully unaware". This feature will require changes to both Spanner and Relay.

Implementation Notes: Command config should be file driven and updateable by external entities like configuration management or manually scp-ing files onto a Relay host. Creating a directory hierarchy based on command names is an acceptable approach for launch but the design should accommodate other command identifiers such as UUIDs, name/version pairs, etc. For example, Relay could load command config from $RELAY_ROOT/command_configs/$bundle_name/$command/config.json for an initial implementation. This wouldn't preclude a different naming scheme in the future like $RELAY_ROOT/command_configs/$bundle_name-$bundle_vsn/$command/config.json

How to contribute

  • How to communicate w/team
  • Ideal contribution
    • Self-contained
    • Modules and public functions should have exdoc
    • Tests covering both positive and negative results
    • Descriptive PR

Bundle activation/deactivation via cogctl

Activate/deactivate bundle on one or more relays. Deactivation is a non-destructive act resulting in a bundle which is unavailable but files are still installed on all Relay hosts. Think of it as taking a bundle out of rotation or pressing "pause" on a bundle.

Example:

# One relay
cogctl bundle deactivate --relay=BFC92057-B4AB-43B0-9E4C-D044AEDDCD78 --bundle=BFC92057-B4AB-43B0-9E4C-D044ABDDCD32

# All relays
cogctl bundle deactivate --relay=all --bundle=BFC92057-B4AB-43B0-9E4C-D044ABDDCD32

Installation instructions

Step-by-step installation guide including the following:

  • Downloading Cog
  • Initial set up (including environmental dependencies like Postgres)
  • Configuration
  • Startup and bootstrapping users with cogctl
  • Installing commands

Return bundle details along with permissions

When granting permissions you need to use the fully qualified <bundle-name>:<permission-name> which is fine, but the index action for permissions GET /permissions only includes the permission id and name. We should return the permission id, permission name, bundle id and bundle name to make granting and revoking possible.

Make carrier credential storage backend pluggable

The Carrier.CredentialStore in operable/carrier currently stores message signature related credentials to a local DETS database. This storage interface needs to be abstracted so that operable/cog can store credentials in the database while retaining the DETS option for operable/relay.

This would remove the only piece of local state that has to be persisted with cog and simplify hosting options.

Newlines are not displayed correctly

When executing sort with single digits, the newlines that are inserted are not being honored. When observing what response is being sent to emqtt, the following is seen "1\n3\n4\n6\n8\n"

Ex. sort 8 4 6 3 1
is displayed as

34
68

When observing what response is being sent to emqtt, the following is sent "1\n3\n4\n6\n8\n"

Design documentation

High level document aimed at technical end users. Should cover:

  • Permissions and rules
    • What they are
    • Lifecycle
    • How they're applied
  • Commands & bundles
    • Definition of concepts
    • Layout of a bundle file
    • High level installation process description
    • Pointer to operable/hello-cog repo
  • Pipeline Execution & Redirection
  • Relay and MQTT

Batch & Stream command types

We need to extend command bundle metadata to include the notion of a command "type" with two possible values: stream and batch.

stream commands are called once per input item. The executor is responsible for iterating over command inputs, preparing the command invocation, checking permissions, and then invoking the command. In effect, the executor "maps" command invocations over the collected inputs. A pipeline execution will fail if any command invocation fails a permission check.

batch commands are called once with a list of input items. The executor is responsible for accumulating command inputs, performing a permission check per item, and then invoking the command with accumulated inputs. In this model, the executor performs permission checks as per the stream case. The difference is that the checked data is accumulated and sent to the command once instead of item-by-item. The command is responsible for iterating over its inputs and generating a single aggregate response. A pipeline execution will fail if any permission check fails. This means calling a batch command could fail due to lacking permissions before the executor invokes the command.

github commands crash when Github returns non-200 responses

Example:

{%Poison.EncodeError{message: nil,
   value: {404,
    %{"documentation_url" => "https://developer.github.com/v3",
      "message" => "Not Found"}}},
  [{Poison.Encoder.Any, :encode, 2, [file: 'lib/poison/encoder.ex', line: 339]},
   {Poison.Encoder.Map, :"-encode/3-fun-1-", 4,
    [file: 'lib/poison/encoder.ex', line: 213]},
   {Poison.Encoder.Map, :"-encode/3-lists^foldl/2-0-", 3,
    [file: 'lib/poison/encoder.ex', line: 214]},
   {Poison.Encoder.Map, :encode, 3, [file: 'lib/poison/encoder.ex', line: 214]},
   {Poison, :encode!, 2, [file: 'lib/poison.ex', line: 41]},
   {Carrier.Signature.Carrier.Credentials, :mangle!, 1,
    [file: 'lib/carrier/credential_signature.ex', line: 30]},
   {Carrier.Signature.Carrier.Credentials, :sign, 3,
    [file: 'lib/carrier/credential_signature.ex', line: 39]},
   {Carrier.Messaging.Connection, :publish, 3,
    [file: 'lib/carrier/messaging/connection.ex', line: 97]}]}}

Automatically create new users on first invocation

Cog should allow admins to configure it to create a new user the first time someone speaks to it either via a direct message or a command invocation attempt. This needs to be scoped further, but conceptually I'm thinking of configuration that might look like this:

config :cog, Cog.Models.User,
  auto_create: true, # Should Cog automatically create new users? Default false.
  auto_group: "cog:users" # Initial group for automatically created users. Default nil.

Audit logging

Main Idea: Comprehensive log of bot <-> user interactions. We should log to a file and format all log statements as JSON.

Implementation notes: Logging mechanism must tolerate having files moved out from under it by external agents such as logrotate.

Events to log: pipeline start/finish, command invocation start/finish, REST API accesses.

What to log: All entries should include a timestamp and the calling user's userid. Pipeline and command entries should also include the complete pipeline/command invocation in the start message and the final disposition of the pipeline/command (returned {"insert": {"some": {"json": ["here"]}}}, failed w/permission error, failed due to other error, etc). REST API events should also include route, parameters, HTTP verb, and final disposition.

Version bundles not commands

I just noticed Cog versions commands instead of bundles. This strikes me as unnecessarily complicated since bundled commands all live within the same bundle directory structure. Versioning bundles makes more sense to me and seems as thought it'd be easier to implement.

cog_env special key for primitive commands

Commands defined as primitive receive additional input via the cog_env key on the Command.Request. cog_env is populated with the execution scope of the command. Specifically it's the values key from the scope in the current state of the executor. It is necessary because certain primitive commands, filter or sort for example, need access to the entire map returned by a previous command. We don't want that information available to non primitive commands because it makes permissions difficult to lock down. Since primitives have no permissions, it doesn't matter.

For example
Given the command @cog stackoverflow vim | echo $title

The stackoverflow command produces output similar to the following:
For simplicity we'll assume stackoverflow only returns one result.

[{ "title": "Vim an editor",
   "link": "http://stackoverflow.com/vim-an-editor",
   "likes": 40,
   "some_other_key": "some other value"
  }]

The executor then uses the results of stackoverflow to create a scope for the execution of echo. In this case $title is replace with "Vim an editor". And so the user receives "Vim an editor" as the result of the operation. This is basically how all commands work, but primitives also receive an addition key on the request, cog_env.

Which in this case would be populated with:

{ "title": "Vim an editor",
   "link": "http://stackoverflow.com/vim-an-editor",
   "likes": 40,
   "some_other_key": "some other value"
}

Now in the case of echo this is irrelevant, but for commands like filter it becomes more important.
Given the command @cog stackoverflow vim | filter --matches="^Vim" --key="title" --returns="title,link"

Stackoverflow might return input similar to the following:

[{ "title": "Vim an editor",
   "link": "http://stackoverflow.com/vim-an-editor",
   "likes": 40,
   "some_other_key": "some other value"
  },
  { "title": "Emacs vs Vim",
   "link": "http://stackoverflow.com/emacs-vs-vim",
   "likes": 23,
   "some_other_key": "a new some other value"
  },
  { "title": "VimGolf, what's up with that?",
   "link": "http://stackoverflow.com/vimgolf",
   "likes": 86,
   "some_other_key": "another new some other value"
  }]

Assuming filter uses the multi execution type, I'll explain single in a bit, it would be executed once per entry in the results list using each result as the execution scope for the command. In this case there is no variable replacement like in echo, instead filter pulls the data it needs directly from the map. It can then check the title field to see if it matches with "^Vim". If it does it returns a new map consisting of title and link. If it doesn't it just returns an empty response.

The user would then receive a response like this:

[{ "title": "Vim an editor",
   "link": "http://stackoverflow.com/vim-an-editor"
  },
  { "title": "VimGolf, what's up with that?",
   "link": "http://stackoverflow.com/vimgolf"
  }]

Now if we throw a single execution command into the mix.
Given the command @cog stackoverflow vim | filter --matches="^Vim" --field="title" | sort --field="likes" --desc | filter --return="title,link"

We can assume stackoverflow returns the same output as the previous example.

filter would then output something like this:

[{ "title": "Vim an editor",
   "link": "http://stackoverflow.com/vim-an-editor",
   "likes": 40,
   "some_other_key": "some other value"
  },
  { "title": "VimGolf, what's up with that?",
   "link": "http://stackoverflow.com/vimgolf",
   "likes": 86,
   "some_other_key": "another new some other value"
  }]

The execution of sort is where we can see the difference between multi and single. sort is of the single execution type. This means that the command is only executed one time no matter how many items are in the input list. It get's the entire list. Since sort is a primitive, cog_env would match the output of filter.

[{ "title": "Vim an editor",
   "link": "http://stackoverflow.com/vim-an-editor",
   "likes": 40,
   "some_other_key": "some other value"
  },
  { "title": "VimGolf, what's up with that?",
   "link": "http://stackoverflow.com/vimgolf",
   "likes": 86,
   "some_other_key": "another new some other value"
  }]

Sort is responsible for iterating over the list an processing accordingly.

[{ "title": "VimGolf, what's up with that?",
   "link": "http://stackoverflow.com/vimgolf",
   "likes": 86,
   "some_other_key": "another new some other value"
  },
{ "title": "Vim an editor",
   "link": "http://stackoverflow.com/vim-an-editor",
   "likes": 40,
   "some_other_key": "some other value"
  }]

Then filter runs again and the final output looks like this:

[{ "title": "VimGolf, what's up with that?",
   "link": "http://stackoverflow.com/vimgolf"
  },
{ "title": "Vim an editor",
   "link": "http://stackoverflow.com/vim-an-editor"
  }]

Verify Hipchat adapter is stable and ready to ship

Verification should include the following:

  • Executing single commands work
  • Executing command pipelines work
  • DM redirects work
  • Room redirects work
  • Multi-redirects to rooms, DMs, and combinations thereof work
  • Templates render reasonable output

Flaky room redirect behavior

Room redirects work if the bot is already present in the room. If not, then unstable behavior similar to the DM case above occurs.

After some team discussion we agreed to the following high level points to address room redirects:

  1. We can use channels.invite and the SLACK_HTTP_API_TOKEN to invite the bot into target rooms.
  2. The bot should join, post the message, and leave to address user concerns about the bot possibly seeing too much privileged communication. We need to take care that we're not joining and leaving too frequently as this will rapidly annoy users.
  3. The bot should have a notion of "home base" rooms where it is always present.

A complete fix, which is explicitly out of scope before 2/1/2016, would also include verifying bot access to redirect targets and flagging issues before pipeline execution begins.

Example Cog command repo

We need a reference Elixir project developers can use to come up to speed on how to build Cog commands. Suggested repo name is cog-clock. It should be well documented and illustrate the basic concepts of:

  • Defining new permissions
  • Installing default rules
  • Help documentation
  • Writing a basic request/response oriented command

The command will return the current time UTC and optionally convert it to a specified time zone. The command should also be able to format time in 12 or 24 hour format

user: cog:clock --tz=EST
cog: 4:25 PM

or

user: cog:clock --tz=EST --zulu
cog: 16:25

math subcommand crashes

(FunctionClauseError) no function clause matching in Float.parse_unsigned/1
(elixir) lib/float.ex:48: Float.parse_unsigned(4)
(loop) lib/loop/commands/math.ex:47: Loop.Commands.Math.accumulate_args/2โ€

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.