Giter Club home page Giter Club logo

explainshell's Introduction

explainshell.com - match command-line arguments to their help text

explainshell is a tool (with a web interface) capable of parsing man pages, extracting options and explaining a given command-line by matching each argument to the relevant help text in the man page.

How?

explainshell is built from the following components:

  1. man page reader which converts a given man page from raw format to html (manpage.py)
  2. classifier which goes through every paragraph in the man page and classifies it as contains options or not (algo/classifier.py)
  3. an options extractor that scans classified paragraphs and looks for options (options.py)
  4. a storage backend that saves processed man pages to mongodb (store.py)
  5. a matcher that walks the command's AST (parsed by bashlex) and contextually matches each node to the relevant help text (matcher.py)

When querying explainshell, it:

  1. parses the query into an AST
  2. visits interesting nodes in the AST, such as:
    • command nodes - these nodes represent a simple command
    • shell related nodes - these nodes represent something the shell interprets such as '|', '&&'
  3. for every command node we check if we know how to explain the current program, and then go through the rest of the tokens, trying to match each one to the list of known options
  4. returns a list of matches that are rendered with Flask

Missing man pages

Right now explainshell.com contains the entire archive of Ubuntu. It's not possible to directly add a missing man page to the live site (it might be in the future).

Running explainshell locally

Setup a working environment that lets you run the web interface locally using docker:

# download db dump
$ curl -L -o /tmp/dump.gz https://github.com/idank/explainshell/releases/download/db-dump/dump.gz

# Clone Repository
$ git clone https://github.com/idank/explainshell.git

# start containers, load man pages from dump
$ docker-compose build
$ docker-compose up

$ docker-compose exec -T db mongorestore --archive --gzip < /tmp/dump.gz

# run tests
$ docker-compose exec -T web make tests
..SSSSSSSSS.....................................................................
----------------------------------------------------------------------
Ran 80 tests in 0.041s

OK (SKIP=9)
# open http://localhost:5000 to view the ui

Processing a man page

Use the manager to parse and save a gzipped man page in raw format:

$ docker-compose exec -T web bash -c "PYTHONPATH=. python explainshell/manager.py --log info /usr/share/man/man1/echo.1.gz"
INFO:explainshell.store:creating store, db = 'explainshell_tests', host = 'mongodb://localhost'
INFO:explainshell.algo.classifier:train on 994 instances
INFO:explainshell.manager:handling manpage echo (from /tmp/es/manpages/1/echo.1.gz)
INFO:explainshell.store:looking up manpage in mapping with src 'echo'
INFO:explainshell.manpage:executing '/tmp/es/tools/w3mman2html.cgi local=%2Ftmp%2Fes%2Fmanpages%2F1%2Fecho.1.gz'
INFO:explainshell.algo.classifier:classified <paragraph 3, DESCRIPTION: '-n     do not output the trailing newlin'> (0.991381) as an option paragraph
INFO:explainshell.algo.classifier:classified <paragraph 4, DESCRIPTION: '-e     enable interpretation of backslash escape'> (0.996904) as an option paragraph
INFO:explainshell.algo.classifier:classified <paragraph 5, DESCRIPTION: '-E     disable interpretation of backslash escapes (default'> (0.998640) as an option paragraph
INFO:explainshell.algo.classifier:classified <paragraph 6, DESCRIPTION: '--help display this help and exi'> (0.999215) as an option paragraph
INFO:explainshell.algo.classifier:classified <paragraph 7, DESCRIPTION: '--version'> (0.999993) as an option paragraph
INFO:explainshell.store:inserting mapping (alias) echo -> echo (52207a1fa9b52e42fb59df36) with score 10
successfully added echo

Note that if you've setup using the docker instructions above, echo will already be in the database.

explainshell's People

Contributors

abitrolly avatar anestv avatar artoria2e5 avatar automaticgiant avatar avindra avatar bgreenlee avatar chrismwendt avatar dependabot[bot] avatar dmuix avatar ericandrewlewis avatar idank avatar joseph-onsip avatar k3ck3c avatar mgalgs avatar moebrowne avatar rohit-nair avatar rugk avatar sammcj avatar sondove avatar stoivo avatar theworstcomrade avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

explainshell's Issues

Feature request: split args to show only relevant arguments

Hi,

it would be cool if you could turn this into a kind of "Microsoft Office Assistant".
So that it could show you only what the args of "gcc -Wall -ansi -pedantic -g -o" mean, excluding the whole manpage, but not when the manpage is very short.

A step further: "explain !!"

Explain inline shell

The app don't explain nested shells (sub-shells) command or $(command)

like this command:

ls | while read file; do echo "$(echo "$file" | sed -e "s/foo/bar/")"; done

feature reqest: Make the explanation boxes collapsible

The explanation boxes should be shrinkable / collapsible / able to hide to a size where only the title or short explanation can be read.
This way the user could hide the things too big or already known to him.
example search:

cat foo.txt | grep "function"

the description of the pipeline takes up so much space I have to scroll down for every other information on my screen.

ssh isn't parsed well

"ssh machine command arguments"

isn't parsed properly, it seems to think "machine command arguments" is all just the machine

Quoted values joined with option name not recognized

It failes to parse correctly the quoted argument values for single-dashed options when they are joined together with the option e.g.

cut -f1-2 -d' '

Instead it lists the first quote ' as the argument value and the remaining part is flagged as unknown. More over when this is followed by pipe, the quotation mark is shown as the pipe and the pipe itself is concatenated together with the following command name.

For example this command is perfectly valid, but not recognized correctly:

echo "a b c d" | cut -f1-2 -d' ' | grep -e'a b'

bash for loop wasn't recognized

I imagine this kind of syntax should be reconizable

for myvar in $mylist; do echo $myvar; done 

screen shot 2013-12-02 at 12 53 40 pm

BTW, VERY COOL. The UI with the arrows is awesome!

Alignmnt issues in html

if i view my design in google chrome and edit all my design according to it
thn if i view it in mozila its alignmnt is affected
cn any1 help

keyboard shortcuts

keyboard shortcuts to navigate the explanations would be really cool.
great site!

wrong alignment of search field

After we search for a command when we click on the search bar again it expands and comes over the github logo. The github logo should expand as well. I will try to check and correct it but I am not that good with css so it might take some time.

Add delay to path redraw/highlight on argument hover change

Currently all paths/highlighting is updated when the mouse enters or leave an argument box. If you move the cursor up and down between arguments you get the following behaviour:

  • Path to explanation highlighted
  • All paths highlighted
  • Path to new explanation highlighted

I think it would be easier to follow/less distracting if there was a delay between transitions so that it behaved something similar to:

  • Path to explanation highlighted
  • Path to new explanation highlighted

Some conceptual inspiration can be found at http://bjk5.com/post/44698559168/breaking-down-amazons-mega-dropdown

mysql parsing not quite right.

For some reason I've grown up typing this to start the mysql shell:

mysql -u -p

I'm not putting the password on the command line (that would be inadvisable, especially if I'm pasting that into explainshell). I'm saying "ask me for my password, and this is the database " which is exactly how the command is designed to work.

Explainshell parses that command incorrectly. It groups -p together as though it was -p even though you can't have a space after -p if you want what follows to be the password.

P.S. Thanks for explainshell. It's great.

Add support for process substitution

process substitution <(command) is supported by all major modern shells (ksh/bash/zsh). a practical example might be diff <(ls /archives/$(pwd)) <(ls).

Other variations: $(), ``, >().

pacman missing

Hi, pacman which is the packet manager of ArchLinux was missing.

Thanks in advance,

honky

missing man pages

To submit a missing man page please reply here with a link to it in raw format (nroff). Even better: I created a repo with all the source man pages I used here, clone it and send me a pull request.

Assignment isn't covered

When running the command 'hey="hey"', the application complains that "no man page for hey="hey" " or something to that effect.

Escaping of quotes gives me a 500

rsync -avplHtxS --compress --rsh=''ssh -o "Compression no"'' --progress --stats

This is actually a valid (and used) way to pass parameters to the rsync underlying ssh transport to turn off compression.

Incomplete backtick support

I tried it out with a random line from my .bashrc:

lremvd=`svn diff $@ | grep "^-" | grep -v "^---" | wc -l`

And everything until the first pipeline was picked up as a single command. I tried simplifying it a bit by removing the assignment:

echo `svn diff $@ | grep "^-" | grep -v "^---" | wc -l`

However, it that made it think I was echoing the text "`svn diff $@" through the pipeline. I tried getting rid of the echo:

`svn diff $@ | grep "^-" | grep -v "^---" | wc -l`

And it couldn't find a man page for "`svn".

The conclusion is that there is no code to detect code inside backticks. This is a great tool otherwise, though.

split arguments on =

Breaks with commands such as 'dd if=/dev/null', whereas a simple 'dd if' works (despite being an invalid use of dd, of course)

License info

What license is the core of explainshell licensed under?

Add search plugin to website

(I didn't find any separate project for the website; apologies if I am posting this to the wrong place.)

An autodiscoverable OpenSearch plugin would be a nice addition to the site. Sample plugin code:

<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/" xmlns:moz="http://www.mozilla.org/2006/browser/search/">
    <ShortName>explainshell</ShortName>
    <Description>match command-line arguments to their help text</Description>
    <InputEncoding>UTF-8</InputEncoding>
    <Image width="16" height="16" type="image/x-icon">data:image/x-icon;base64,AAABAAEAEBAAAAAAAABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoAAAD+AAAAFAAAALkAAADsAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/AAAAAAAAAPAAAADcAAAAAAAAAHYAAAAKAAAA5wAAANUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASwAAAP8AAAD/AAAAAAAAAP8AAADtAAAAAAAAAP4AAADmAAAAlwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP8AAADeAAAAAAAAAP8AAAD9AAAAAAAAADUAAAD+AAAA+gAAAAAAAADnAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACOAAAAAAAAAP4AAAB9AAAAAAAAAK0AAAD/AAAAYgAAAFEAAAAAAAAAAAAAANwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPMAAAD/AAAAAAAAAN4AAAD+AAAA0gAAAOEAAADuAAAAAAAAAAAAAAASAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAUAAAAAAAAAD+AAAACgAAAPUAAAAAAAAA/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAO8AAAAAAAAA/gAAALYAAAC2AAAABAAAAP8AAAAAAAAAAAAAAAAAAAD+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/wAAAP8AAACMAAAA6gAAAAAAAAD/AAAAAgAAAAAAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA0AAAAjAAAAM0AAAD+AAAA+QAAAMMAAAD9AAAAjQAAAAYAAAANAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPIAAAAOAAAAAAAAAAAAAAAAAAAAAQAAAP8AAABQAAAAIwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAqwAAAP4AAAD8AAAA+wAAAPEAAAArAAAA/gAAAC4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAIAAAA/QAAAP8AAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPkAAAD+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACFAAAAAQAAAAAAAAAA//8AANP/AACnPwAAyR8AAJMvAACs9wAAyD8AAPq/AADouwAA8L8AAPwHAAD99wAA/gsAAP/zAAD/8wAA//cAAA==</Image>
    <Url type="text/html" method="GET" template="http://explainshell.com/explain">
        <Param name="cmd" value="{searchTerms}"/>
    </Url>
    <Url type="application/opensearchdescription+xml" rel="self" template="http://http://explainshell.com/opensearch.xml" />
    <moz:SearchForm>http://explainshell.com/</moz:SearchForm>
</OpenSearchDescription>

Sample code for making the plugin autodiscoverable:

<link rel="search" type="application/opensearchdescription+xml" title="explainshell" href="http://explainshell.com/opensearch.xml" />

Command line version

I would appreciate command line version, so I could write explain tar xzvf archive.tar.gz or even explain !!.

PGP encrypting

Some of the pgp commands are not getting the explanation.

Stick To Top Shell Command

Probably the shell command should stick to the top so that easier to refer the description when scrolling down

Ubuntu: Couldn't get a file descriptor referring to the console

After installing with gem I get the following error on my Ubuntu machine:

$ explain tar
Couldn't get a file descriptor referring to the console

The reason for this is that Ubuntu's /bin/open is a symlink to openvt which is not exactly what we want here. Replacing open by xdg-open solves the issue. Maybe that's possible for other platforms as well?

add support for history expansions

It should be fairly easy to make the parser aware of history expansions so the matcher could give more accurate help for things like !!.

Original bug report by VolMi:

It appears as if the BASH man page is not included…?!?
At least the commandlinefu.com #1 command

sudo !!

is not explained correctly.
But on my debian, the "!!" syntax is explained in man bash, like here:

!!

Refer to the previous command. This is a synonym for '!-1'. 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.