nteract / commuter Goto Github PK

View Code? Open in Web Editor NEW

485.0 40.0 65.0 2.69 MB

🚎 Notebook sharing hub

License: BSD 3-Clause "New" or "Revised" License

JavaScript 89.69% CSS 8.67% Dockerfile 1.65%

notebook nteract jupyter-notebook s3 sharing collaboration

commuter's People

Stargazers

Watchers

commuter's Issues

Handle inline maths in Markdown

Notebooks with embedded maths in Markdown aren't rendering on commuter.

Top level way to hide all and show all code

User story: want to link to a notebook document as a standalone report -- an interactive data memo. To facilitate this, we want a way to hide and show code quickly.

Users also want:

availability as a URL param ("so I can share links that open a clean notebook without code cells")
ability to toggle it per cell (in the live view)

Limit large files

We can only render files of a certain size with reliability. Some users end up having large CSVs that they shouldn't try to render on here (yet). For now, there should be a cap. Related: #93.

Turn off the search/replace from the Editor

The CodeMirror find/replace is currently hijacking the entire Find view if a user clicks inside the cell. Oops!

Large files proxied may need additional handling

When opening nyc_taxi.ipynb (from the demo) now, this error message is logged to the console:

[0] ::ffff:127.0.0.1 - - [19/Feb/2017:08:26:06 +0000] "GET /api/contents/ HTTP/1.1" 304 -
[0] /Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:31
[0]             throw err;
[0]             ^
[0]
[0] SyntaxError: Unexpected end of JSON input
[0]     at Object.parse (native)
[0]     at Response.s3.getObject (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/src/services/s3.js:70:30)
[0]     at Request.<anonymous> (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:358:18)
[0]     at Request.callListeners (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
[0]     at Request.emit (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
[0]     at Request.emit (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:671:14)
[0]     at Request.transition (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:22:10)
[0]     at AcceptorStateMachine.runTo (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/state_machine.js:14:12)
[0]     at /Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/state_machine.js:26:10
[0]     at Request.<anonymous> (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:38:9)
[1] Proxy error: Could not proxy request /api/contents//nyc_taxi.ipynb from 127.0.0.1:3000 to http://localhost:4000.
[1] See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNRESET).
[1]
[0]
[0] npm
[0]  ERR! Darwin 16.4.0
[0] npm ERR! argv "/Users/kylek/.nvm/versions/node/v6.9.2/bin/node" "/Users/kylek/.nvm/versions/node/v6.9.2/bin/npm" "run" "server"
[0] npm ERR! node v6.9.2
[0] npm ERR! npm  v4.0.5
[0] npm ERR! code ELIFECYCLE
[0] npm ERR! @nteract/[email protected] server: `./src/index.js server`
[0] npm ERR! Exit status 1
[0] npm ERR!
[0] npm ERR! Failed at the @nteract/[email protected] server script './src/index.js server'.
[0] npm ERR! Make sure you have the latest version of node.js and npm installed.
[0] npm ERR! If you do, this is most likely a problem with the @nteract/commuter-cli package,
[0] npm ERR! not with npm itself.
[0] npm ERR!
[0]  Tell the author that this fails on your system:
[0] npm ERR!     ./src/index.js server
[0] npm ERR! You can get information on how to open an issue for this project with:
[0] npm ERR!     npm bugs @nteract/commuter-cli
[0] npm ERR! Or if that isn't available, you can get their info via:
[0] npm ERR!     npm owner ls @nteract/commuter-cli
[0] npm ERR! There is likely additional logging output above.
[0]
[0] npm ERR! Please include the following file with any support request:
[0] npm ERR!     /Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-cli/npm-debug.log
[0] npm run server exited with code 1

Other notebooks load just fine. Since I know that the nyc_taxi notebook is huge (due to it embedding BokehJS + data, quadruple encoded in many places), it's likely that S3 is expecting us to get the data in chunks (this is my hunch at least). As an aside to this, we'll want to prevent fully exiting/crashing during errors like these (unless we're going to set up some forever for this handling...)

Ability to run commuter on a url prefix

e.g https://host.tld/commuter/<path_to_ipynb>

Prefix in this case will be /commuter/

Bring in the plotly renderer

Otherwise commuter tries to render the embedded plotly HTML which tries to use requirejs.

Run flow and prettier eslint on Travis

It would be great to see flow and prettier get checked on travis.

Note: only the commuter-server has flow setup.

Heroku deployment failing

The Heroku pipeline has been failing since Lerna 2.0.0-rc.1 was brought in.

       Lerna v2.0.0-rc.1
       Independent Versioning Mode
       This is not a git repository, did you already run `git init` or `lerna init`?
       
       npm ERR! Linux 3.13.0-112-generic
       npm ERR! argv "/tmp/build_6b2b2babbbb3cc8f0a897e4e530ce41d/nteract-commuter-61cfbaf/.heroku/node/bin/node" "/tmp/build_6b2b2babbbb3cc8f0a897e4e530ce41d/nteract-commuter-61cfbaf/.heroku/node/bin/npm" "run" "lerna"
       npm ERR! node v6.9.5
       npm ERR! npm  v3.10.10
       npm ERR! code ELIFECYCLE
       npm ERR! @nteract/[email protected] lerna: `lerna bootstrap`
       npm ERR! Exit status 1
       npm ERR!
       npm ERR! Failed at the @nteract/[email protected] lerna script 'lerna bootstrap'.
       npm ERR! Make sure you have the latest version of node.js and npm installed.
       npm ERR! If you do, this is most likely a problem with the @nteract/commuter-client package,
       npm ERR! not with npm itself.
       npm ERR! Tell the author that this fails on your system:

It looks like lerna is not pleased with the fact that it isn't a git repo by the time it reaches Heroku.

Error Handling

Bits for cleanup:

Show loading state / progress
When there is no discovery backend, API should possibly return

{
  results: [],
  error: "Message about the discovery backend not being connected"
}

Handle S3 timeouts, errors, etc.
Any other miscellaneous errors need to be logged as well as displayed for users

Allow deploying commuter on a basepath

In order to deploy commuter as a service for JupyterHub (and generally on the configurable HTTP proxy), we need the ability to run commuter on a prefix.

/cc @minrk @cabhishek

Render source code with a syntax highlighted view

For .js, .py, and all the others, use CodeMirror or something lightweight for syntax highlighting.

npm run build:frontend takes forever*

Ok, not forever. Long enough that it's well past the timeout on Heroku, and, at least locally, I haven't seen finish directly.

Password / Authentication ?

Is it possible to lock down the notebook via password

install on master not working

$ npm i

> @nteract/[email protected] install /Users/kylek/code/src/github.com/nteract/commuter
> npm run lerna && npm run build


> @nteract/[email protected] lerna /Users/kylek/code/src/github.com/nteract/commuter
> lerna bootstrap

Lerna v2.0.0-beta.36
Independent Versioning Mode
Bootstrapping 5 packages
Preinstalling packages
Installing external dependencies
NpmUtilities.installInDir     ("./packages/commuter-cli", ["copy-dir@^0.3.0"])
Command exited with status 1: npm install copy-dir@^0.3.0
npm ERR! Darwin 15.6.0
npm ERR! argv "/Users/kylek/.nvm/versions/node/v6.9.2/bin/node" "/Users/kylek/.nvm/versions/node/v6.9.2/bin/npm" "install" "copy-dir@^0.3.0"
npm ERR! node v6.9.2
npm ERR! npm  v4.0.5
npm ERR! code E404

npm ERR! 404 Not found : @nteract/commuter-directory-listing
npm ERR! 404
npm ERR! 404  '@nteract/commuter-directory-listing' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 It was specified as a dependency of '@nteract/commuter-client'
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-cli/npm-debug.log

Errored while running BootstrapCommand.execute
Command exited with status 1: npm install copy-dir@^0.3.0

npm run start script doesn't restart failed processes

If the server dies, the client stays running and you have to restart the entire thing.

[bug] Author field not getting populated on the Discover UI

Conformance Testing

Based on #30, we need to make sure our API is consistent and correct. At the very least we need to make sure our contents API is 100% compatible with the Jupyter Contents API.

Persist notebooks back to S3

Right now notebooks can only be listed from S3. Implement notebooks saving feature in a way that's compatible with notebooks content API using rx-jupyter

Niceties for Zeppelin notebook URLs

Make the URL be /YAZXC1425/note.json#Name-of-zeppelin-notebook
Within the directory listing show links for zeppelin/${ID}/note.json as the name of the Zeppelin notebook.

Organized collections

Something I've noted from others who are familiar with products like Tableau -- even if an overall organization is sharing notebooks all over, their suborganizations each want to share collections.

In an almost nonfictional example, imagine an org* is split across several product lines, disciplines or customer segments, each with their own staff of data scientists, statisticians, and data engineers:

Latin America
Asia
Europe
Marketing
Infrastructure Operations

* Not completely non-fictional, I've worked at several companies that were large enough to split across lines like these

Add tests for react components

And use jest

Render other content

HTML - do it in an iframe (#88)
Images
RMarkdown - convert to an in-memory notebook (?)

Ensure server is built before launching with `start`

If the server package hasn't had a fresh build, whatever old version in lib is the one that runs on npm run start.

Create a "file-based" discovery backend

Allow users to configure COMMUTER_DISCOVERY_FILE that is a JSON document used to respond to /v1/api/discovery. Alternatively, they could put it on S3 in their bucket, maybe at the base path?

frontend build error with octicons.js

➜  commuter git:(master) node --version
v8.9.0
➜  commuter git:(master) npm --version
5.5.1
➜  commuter git:(master) git rev-parse HEAD
484d26e83d5d1a25001c0d21566c39b8f49cea88

npm run dev

 DONE  Compiled successfully in 1998ms                                                                                16:08:11

[Wed Nov 01 2017 16:08:11 GMT+0800 (CST)] INFO Node env: development
[Wed Nov 01 2017 16:08:11 GMT+0800 (CST)] INFO Commuter server listening on port 4000
> Building page: /


 DONE  Compiled successfully in 1189ms                                                                                16:08:22

::1 - - [01/Nov/2017:08:08:22 +0000] "GET / HTTP/1.1" 302 -
> Building page: /view


 ERROR  Failed to compile with 1 errors                                                                               16:08:28                                                                          

 error  in ../frontend/icons/octicons.js

Module build failed: TypeError: /home/*/Repos/github/nteract/commuter/packages/frontend/icons/octicons.js: Cannot read property '0' of null
    at Buffer._append (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/buffer.js:95:25)
    at Buffer.append (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/buffer.js:72:10)
    at Generator._append (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:206:52)
    at Generator.word (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:131:10)
    at Generator.Identifier (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/generators/types.js:38:8)
    at /home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:298:23
    at Buffer.withSource (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/buffer.js:168:5)
    at Generator.withSource (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:189:15)
    at Generator.print (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:297:10)
    at Generator.MemberExpression (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/generators/expressions.js:216:8)

 @ ../frontend/components/contents/directory-listing.js 19:16-47
 @ ../frontend/components/contents/index.js
 @ ../frontend/pages/view.js?entry
 @ multi ../frontend/pages/view.js?entry

Display section numbers

As reported by @jeffwong, commuter doesn't display section numbering like the classic notebook does.

Admittedly, neither does the nteract app since we're using commonmark which has no opinionated stance on sections. We should probably do some mangling of the AST to take a stance on the header links.

/cc @mpacer who's done a bunch of thinking on this

Local Storage Backend

Right now we have an S3 backend. In order to support EFS and other NFS based storage options, we'll want another backend.

Designing the Search API

CC @parente and @mariusvniekerk who were interested in this.

Are there current search APIs we'd want borrow from for this?

Symlink Handling

When commuter comes across a symlink, whether it's a file or directory, it fails. It also fails hard, not returning any results on a directory listing. If a symlink exists and we have access to what it resolves to, we should support it.

Display last_modified and other metadata on file view

Especially for notebooks, it would be nice to know when the file was last modified.

Development flow broken at head

When I follow the developer instructions for commuter I get the error below. The error first appears with the Oct 2nd update, which leads me to believe this is a problem with updating packages.

I rolled back the Sept 11th update and everything works. I also confirmed that this is not only my computer.

Error:
error in ../frontend/icons/octicons.js
Module build failed: TypeError: .../octicons.js: Cannot read property '0' of null

Docker ?

I can see Commuter being used as a docker server, while running a commuter instance from within a docker container. If there isn't a plan in place already i'd be happy to contribute a Pull Request, that installs, and runs commuter via docker.

Please let me know your thoughts.

Make iframe for HTML viewer more seamless

In #88, I used absolute positioning and left: 10rem to put the iframe in place. I'd like to see if there's a better way to make the iframe take up the rest of the page while being positioned like our other renderers (ok, the one notebook renderer).

CSV Rendering

Render .csv files.
Provide data resource renderer from nteract/nteract
Render the zeppelin tsv format with the above CSV renderer

Custom CSS

There are a few levels of CSS that we should think about configuring:

At the notebook level, be able to set CSS for the notebook (in metadata), as opposed to outputting it in a cell:

metadata: {
  nteract: {
    css: {
      url: "someCustom.css"
    }
  }
}

Some users have voiced that they would like to see their own styling across the entire deployment of notebooks so it matches their personal style. Perhaps this means a toggle for now with our current themes, while allowing them to in the future set up a custom theme.

Ability to search for notebooks?

Greetings,

This may already be documented somewhere , but it it possible to search for a notebook.
As a project or instance grows, it may make it easier to share amongst a larger team if one could search for a notebook they're looking for.

Include nteract fonts on notebook view

Should be as easy as adding:

    <link href='https://fonts.googleapis.com/css?family=Source+Code+Pro:400,700,300,200,500,600,900' rel='stylesheet' type='text/css'>
    <link href='https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,200,200italic,300,300italic,400italic,600,600italic,700,700italic,900,900italic' rel='stylesheet' type='text/css'>

to the index.html

commuter screen shots

Support auth via aws_access_key_id & aws_secret_access_key

Implementation details here

Right now only IAM_ROLE auth works.

Return standard jupyter API response when "directory" not found

Right now when S3 "directories" are not found they return an empty content directory when they should return:

{
  "reason": null,
  "message": "No such file or directory: someName"
}

Configure Redux Devtools

Notebooks failing to load

commit used to reproduce the errors:

Directory listing works:

Notebook fails:

Hide certain files

In particular, don't show .ipynb_checkpoints, maybe other hidden files.

Create a default rendered view when the content is not known

Right now users end up at a blank page, we should give them at least a <pre>. Need some way of dealing with binary files though. 🤔

Non-notebook files should adhere to the contents API

Right now the API is trying to JSON.parse all files when it should only decode notebook content. Here are the expected responses for a few files:

/api/contents/trial.json

{
  "type": "file",
  "name": "trial.json",
  "format": "text",
  "created": "2017-03-07T21:36:34.692957Z",
  "path": "trial.json",
  "last_modified": "2017-03-07T21:36:34.692957Z",
  "content": "{\n    \"stuff\": \"cool\"\n}",
  "writable": true,
  "mimetype": "application/json"
}

/api/contents/test.html

{
  "type": "file",
  "name": "test.html",
  "format": "text",
  "created": "2017-03-07T21:19:44.792556Z",
  "path": "test.html",
  "last_modified": "2017-03-07T21:19:44.792556Z",
  "content": "<h1>Hey</h1>\n\n<p>Is this real life?</p>",
  "writable": true,
  "mimetype": "text/html"
}

nteract / commuter Goto Github PK

commuter's People

Stargazers

Watchers

Forkers

commuter's Issues

Recommend Projects

Recommend Topics

Recommend Org