Giter Club home page Giter Club logo

livebook's Introduction

Livebook

Website Latest Version

Livebook is a web application for writing interactive and collaborative code notebooks. It features:

  • Code notebooks with Markdown support and Code cells where Elixir code is evaluated on demand.

  • Rich code editor through CodeMirror: with support for autocompletion, inline documentation, code formatting, etc.

  • Interactive results via Kino: display Vega-Lite charts, tables, maps, and more.

  • Automation: use Smart cells to perform high-level tasks and write notebooks faster than ever. Query databases, plot charts, build maps, and more directly from Livebook's UI.

  • Reproducible: Livebook ensures your code runs in a predictable order, all the way down to package management. It also tracks your notebook state, annotating which parts are stale.

  • Collaboration: multiple users can work on the same notebook at once, no additional setup required.

  • Decentralized: Livebook is open-source and you can run it anywhere. The "Run in Livebook" badge makes it easy to import any Livebook into your own Livebook.

  • Versionable: notebooks are stored in the .livemd format, which is a subset of Markdown with support for diagrams via Mermaid and for mathematical formulas via KaTex. .livemd files can be shared and play well with version control.

  • Custom runtimes: when executing Elixir code, you can either start a fresh Elixir instance, connect to an existing node, or run it inside an existing Elixir project, with access to all of its modules and dependencies. This means Livebook can be a great tool to introspect and document existing projects too.

Getting started

Head out to the Install section of Livebook's website to get started. Once Livebook is up and running on your machine, visit the "Learn" section with introductory guides and documentation on several Livebook features. Here is a sneak peak of the "Welcome to Livebook" guide:

Screenshot

For screencasts and news, check out news.livebook.dev.

Installation

We provide several methods for running Livebook, pick the one that best fits your use case.

Desktop app

Docker

Running Livebook using Docker is another great option to run Livebook in case you don't have Elixir installed.

# Running with the default configuration
docker run -p 8080:8080 -p 8081:8081 --pull always ghcr.io/livebook-dev/livebook

# In order to access and save notebooks directly to your machine
# you can mount a local directory into the container.
# Make sure to specify the user with "-u $(id -u):$(id -g)"
# so that the created files have proper permissions
docker run -p 8080:8080 -p 8081:8081 --pull always -u $(id -u):$(id -g) -v $(pwd):/data ghcr.io/livebook-dev/livebook

# You can configure Livebook using environment variables,
# for all options see the dedicated "Environment variables" section below
docker run -p 8080:8080 -p 8081:8081 --pull always -e LIVEBOOK_PASSWORD="securesecret" ghcr.io/livebook-dev/livebook

# Or if you need to run on different ports:
docker run -p 8090:8090 -p 8091:8091 --pull always -e LIVEBOOK_PORT=8090 -e LIVEBOOK_IFRAME_PORT=8091 ghcr.io/livebook-dev/livebook

To deploy Livebook on your cloud platform, see our Docker Deployment guide.

For CUDA support, see images with the "cuda" tag.

To try out features from the main branch you can alternatively use the ghcr.io/livebook-dev/livebook:edge image. See Livebook images.

Embedded devices

If you want to run Livebook on embedded devices, such as Raspberry Pi, BeagleBone, etc., check out the Livebook firmware built with Nerves.

Direct installation with Elixir

You can run Livebook on your own machine using just Elixir. You will need Elixir v1.15.2 or later. Livebook also requires the following Erlang applications: inets, os_mon, runtime_tools, ssl and xmerl. Those applications come with most Erlang distributions but certain package managers may split them apart. For example, on Ubuntu, these Erlang applications can be installed as follows:

sudo apt install erlang-inets erlang-os-mon erlang-runtime-tools erlang-ssl erlang-xmerl erlang-dev erlang-parsetools

Note: The livebook package is meant to be used as a CLI tool. Livebook is not officially supported as a Mix/Hex dependency.

Escript

Running Livebook using Escript makes for a very convenient option for local usage and provides easy configuration via CLI options.

mix do local.rebar --force, local.hex --force
mix escript.install hex livebook

# Start the Livebook server
livebook server

# See all the configuration options
livebook server --help

After you install the escript, make sure you add the directory where Elixir keeps escripts to your $PATH. If you installed Elixir with asdf, you'll need to run asdf reshim elixir once the escript is built.

To try out features from the main branch you can alternatively install the escript directly from GitHub like this:

mix escript.install github livebook-dev/livebook

From source

You can run latest Livebook directly from source.

git clone https://github.com/livebook-dev/livebook.git
cd livebook
mix deps.get --only prod

# Run the Livebook server
MIX_ENV=prod mix phx.server

Security considerations

Livebook is built to document and execute code. Anyone with access to a Livebook instance will be able to access any file and execute any code in the machine Livebook is running.

For this reason, Livebook only binds to the 127.0.0.1, allowing access to happen only within the current machine. When running Livebook in the production environment - the recommended environment - we also generate a token on initialization and we only allow access to the Livebook if said token is supplied as part of the URL.

Environment variables

The following environment variables can be used to configure Livebook on boot:

  • LIVEBOOK_ALLOW_URI_SCHEMES - sets additional allowed hyperlink schemes to the Markdown content. Livebook sanitizes links in Markdown, allowing only a few standard schemes by default (such as http and https). Set it to a comma-separated list of schemes.

  • LIVEBOOK_APP_SERVICE_NAME - sets the application name used by the cloud provider to aid debugging.

  • LIVEBOOK_APP_SERVICE_URL - sets the application url to manage this Livebook instance within the cloud provider platform.

  • LIVEBOOK_APPS_PATH - the directory with app notebooks. When set, the apps are deployed on Livebook startup with the persisted settings. Password-protected notebooks will receive a random password, unless LIVEBOOK_APPS_PATH_PASSWORD is set. When deploying using Livebook's Docker image, consider using LIVEBOOK_APPS_PATH_WARMUP.

  • LIVEBOOK_APPS_PATH_HUB_ID - deploy only the notebooks in LIVEBOOK_APPS_PATH that belong to the given Hub ID

  • LIVEBOOK_APPS_PATH_PASSWORD - the password to use for all protected apps deployed from LIVEBOOK_APPS_PATH.

  • LIVEBOOK_APPS_PATH_WARMUP - sets the warmup mode for apps deployed from LIVEBOOK_APPS_PATH. Must be either "auto" (apps are warmed up on Livebook startup, right before app deployment) or "manual" (apps are warmed up when building the Docker image; to do so add "RUN /app/bin/warmup_apps" to your image). Defaults to "auto".

  • LIVEBOOK_AWS_CREDENTIALS - enable Livebook to read AWS Credentials from environment variables, AWS Credentials, EC2/ECS metadata when configuring S3 buckets.

  • LIVEBOOK_BASE_URL_PATH - sets the base url path the web application is served on. Useful when deploying behind a reverse proxy.

  • LIVEBOOK_CACERTFILE - path to a local file containing CA certificates. Those certificates are used during for server authentication when Livebook accesses files from external sources.

  • LIVEBOOK_CLUSTER - configures clustering strategy when running multiple instances of Livebook using either the Docker image or an Elixir release. See the "Clustering" section of our Docker Deployment guide for more information: https://hexdocs.pm/livebook/docker.html

  • LIVEBOOK_COOKIE - sets the cookie for running Livebook in a cluster. Defaults to a random string that is generated on boot.

  • LIVEBOOK_DATA_PATH - the directory to store Livebook's internal configuration. Defaults to "livebook" under the default user data directory.

  • LIVEBOOK_DEBUG - enables verbose logging, when set to "true". Disabled by default.

  • LIVEBOOK_DEFAULT_RUNTIME - sets the runtime type that is used by default when none is started explicitly for the given notebook. Must be either "standalone" (Elixir standalone), "attached:NODE:COOKIE" (Attached node) or "embedded" (Embedded). Defaults to "standalone".

  • LIVEBOOK_DISTRIBUTION - sets the node distribution for running Livebook in a cluster. Must be "name" (long names) or "sname" (short names). Note that this sets RELEASE_DISTRIBUTION if present when creating a release. Defaults to "sname".

  • LIVEBOOK_FIPS - if set to "true" will try to enable the FIPS mode on startup. See more details in the documentation.

  • LIVEBOOK_FORCE_SSL_HOST - sets a host to redirect to if the request is not over HTTPS. Note it does not apply when accessing Livebook via localhost. Defaults to nil.

  • LIVEBOOK_HOME - sets the home path for the Livebook instance. This is the default path used on file selection screens and others. Defaults to the user's operating system home.

  • LIVEBOOK_IDENTITY_PROVIDER - controls whether Zero Trust Authentication must be used as the identity provider. This is useful when deploying Livebook inside a cloud platform, such as Cloudflare and Google. Supported values are:

    • "basic_auth::"
    • "cloudflare:<your-team-name (domain)>"
    • "google_iap:<your-audience (aud)>"
    • "tailscale:"
    • "teleport:"
    • "custom:YourElixirModule"

    See our authentication docs for more information: https://hexdocs.pm/livebook/authentication.html

  • LIVEBOOK_IFRAME_PORT - sets the port that Livebook serves iframes at. This is relevant only when running Livebook without TLS. Defaults to 8081.

  • LIVEBOOK_IFRAME_URL - sets the URL that Livebook loads iframes from. By default iframes are loaded from local LIVEBOOK_IFRAME_PORT when accessing Livebook over http:// and from https://livebookusercontent.com when accessing over https://.

  • LIVEBOOK_IP - sets the ip address to start the web application on. Must be a valid IPv4 or IPv6 address.

  • LIVEBOOK_NODE - sets the node name for running Livebook in a cluster. Note that this sets RELEASE_NODE if present when creating a release.

  • LIVEBOOK_PASSWORD - sets a password that must be used to access Livebook. Must be at least 12 characters. Defaults to token authentication.

  • LIVEBOOK_PORT - sets the port Livebook runs on. If you want to run multiple instances on the same domain with the same credentials but on different ports, you also need to set LIVEBOOK_SECRET_KEY_BASE. Defaults to 8080. If set to 0, a random port will be picked.

  • LIVEBOOK_SECRET_KEY_BASE - sets a secret key that is used to sign and encrypt the session and other payloads used by Livebook. Must be at least 64 characters long and it can be generated by commands such as: openssl rand -base64 48. Defaults to a random secret on every boot.

  • LIVEBOOK_SHUTDOWN_ENABLED - controls if a shutdown button should be shown in the homepage. Set it to "true" to enable it.

  • LIVEBOOK_TOKEN_ENABLED - controls whether token authentication is enabled. Enabled by default unless LIVEBOOK_PASSWORD is set. Set it to "false" to disable it.

  • LIVEBOOK_UPDATE_INSTRUCTIONS_URL - sets the URL to direct the user to for updating Livebook when a new version becomes available.

  • LIVEBOOK_WITHIN_IFRAME - controls if the application is running inside an iframe. Set it to "true" to enable it. If you do enable it, then the application must run with HTTPS.

If running Livebook via the command line, run livebook server --help to see all CLI-specific options.

Livebook Desktop

When running Livebook Desktop, Livebook will invoke on boot a file named ~/.livebookdesktop.sh on macOS or %USERPROFILE%\.livebookdesktop.bat on Windows. This file can set environment variables used by Livebook, such as:

  • the PATH environment variable

  • set LIVEBOOK_DISTRIBUTION=name to enable notebooks to communicate with nodes in other machines

  • or to configure the Erlang VM, for instance, by setting ERL_AFLAGS="-proto_dist inet6_tcp" if you need Livebook to run over IPv6

Be careful when modifying boot files, Livebook may be unable to start if configured incorrectly.

Development

Livebook is primarily a Phoenix web application and can be setup as such:

git clone https://github.com/livebook-dev/livebook.git
cd livebook
mix setup

# Run the Livebook server
mix phx.server

# Run tests
mix test

Once you submit a pull request, Uffizzi will setup a preview environment where anyone can try out your changes and give feedback.

Desktop app builds

For macOS, run:

# Test macOS app locally
(cd rel/app/macos && ./run.sh)

# Build macOS installer
.github/scripts/app/build_macos.sh

For Windows, run:

# Test Windows app locally
(cd rel/app/windows && ./run.sh)

# Build Windows installer
.github/scripts/app/build_windows.sh

Platinum sponsors

Fly.io

Fly is a platform for running full stack apps and databases close to your users.

Sponsors

Hugging Face

The platform where the machine learning community
collaborates on models, datasets, and applications.


Tigris

Tigris is a globally distributed S3-compatible object storage
service that provides low latency anywhere in the world.

License

Copyright (C) 2021 Dashbit

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

livebook's People

Contributors

aar2dee2 avatar aledsz avatar awerment avatar basilenouvellet avatar benjamin-philip avatar benjreinhart avatar brooklinjazz avatar byeongukchoi avatar cocoa-xu avatar cristineguadelupe avatar fhunleth avatar gpopides avatar heavybr avatar hugobarauna avatar jacqueslorentz avatar jannikbecher avatar jeantux avatar jonatanklosko avatar josevalim avatar kianmeng avatar mgibowski avatar moogle19 avatar oo6 avatar paulo-valim avatar petejodo avatar qizot avatar qrede avatar wojtekmach avatar zachallaun avatar zolrath avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

livebook's Issues

Add drag and drop to reorder cells

This is pending a feature to be implemented upstream on LiveView. We should probably have a small part of the cell that is draggable.

Improve the workflow of starting a new notebook

Today, when starting a notebook, you end-up with this:

Screenshot 2021-03-19 at 09 33 30

Unless you know you have to click on the sidebar, you will get stuck. There are a couple options we can choose to improve this (perhaps even more than one):

  1. Have new notebooks already start with a section
  2. Start notebooks the sidebar open
  3. ???

Support read-only code cells

Those cells are not executed. It can be useful for when we want to only render some example code that may not yet be functional. We need a clean way to identify those cells.

Editor autocompletion

Show autocomplete suggestion relevant for the given cell. Probably something similar to IEx.Autocomplete. The editor has support for showing more info for each suggestion, so it can be nicely used to show some docs/specs.

Set the current directory for Elixir executable

I have been playing with Livebook to give my presentation on Nx and MNIST and I noticed a problem related to the current working directory. For the Elixir executable, we are running it on the Livebook server directory but in this case I think it makes the most sense for it to be the in the same directory as the Livebook file itself.

This will work fine for when you open up an existing Livebook, but what to do when you start with the Livebook in memory and then you save it? Should we change cwd of the existing evaluator? I would say so.

For the Mix evaluator, we will document the root is the Mix project root. For the distributed one, it is whatever the existing node is in.

Add tooltips

This is just a reminder so we don't forget to incorporate them. We need tooltips on sidebar icons and on the icons on the focused cells.

Kickoff meetings notes

LiveBook is meant to be the Elixir version of notebooks with a focus on:

  • Real-time collaboration
  • Live features - a cell can be alive and stream values as it runs

The biggest challenges we have ahead of us is to address the drawbacks from Jupyter notebooks:

  • Do not depend on evaluation order. We need to attach some ordering to cells, either explicitly or implicitly.
  • How to persist the notebooks? JSON is not executable, not friendly to version control software.

Other considerations:

  • Can the notebooks be executable or even deployed to prod?
  • Security: how can we sandbox the execution?

Fix quote/backtick completion on markdown cells

  • Inside markdown cells, if I press [, it correctly adds ] and puts the cursor in the middle. However, if I press ", it generates two quotes and does not put the text in the middle.
  • Pressing cmd+enter should default to a no-op (similar to Elixir)

Remove old deltas

We have to keep a list of deltas on the server to properly rebase the incoming ones, but we should drop the old ones. Ideally we would keep only the necessary deltas, but this would require idle clients to send additional acknowledgements of received deltas and may get overly complex. I think we should be fine limiting the list to N deltas.

Support more ANSI escape codes

Extend LiveBookWeb.Helpers.ansi_string_to_html to support more advanced ANSI codes like those generated with IO.ANSI.color/3 (more details here). Additionally it would be great to support multiple modifiers at once (e.g. foreground + background + italic).

Rename project to livebook?

Since we are going to ship with an executable, we should probably rename the app from live_book to livebook. This is a matter of replacing live_book -> livebook in the code and renaming a couple files. We will likely still style it as "LiveBook". I can rename this repo accordingly once decided.

Support branching sections

Use case: all notebooks will need to execute some code to prepare for the sections/cells that come next. For example, code that runs Mix.install or some imports/requires. We need to support this somehow.

One idea is to mark a section (or multiple) sections as header sections. Those sections are always executed before any other section and their bindings are passed forward to the other ones. Alternatively, we can support header cells but I think marking a whole section is probably closer in intent to how we want folks to design the notebooks.

How are we going to ship livebook?

I am thinking we can start by using escripts but we should also consider shipping a release or using bakeware in the future.

Escripts are the most convenient for Elixir developers, so we should definitely consider it. The only downside of escripts is that it can't serve assets from the "priv" directory, so we will have to embed them like we did in the live dashboard.

Make the location of modal button consistent

On Notebooks settings, the "Save" button is on the right side for the File tab but on the left side for the Runtime tab.

My suggestion is to move it to the left. For the file button, we can show the filename that will be saved close to the save button (and I think having them close together is a good idea in the first place). :)

Add anchor links

Each section and cell should have an anchor icon that changes the URL to that cell, so we can copy and paste it and share it.

Cannot scroll output while it is being printed

Evaluating the following code takes a long time:

for i <- 1..1000 do
  IO.puts i
end

I assume this is because the browser is busy handling each line and doing virtualization, we may need to revisit the approach here. Should we also automatically scroll to the bottom, like a terminal would?

Tab navigation is unclear

Today pressing tab consecutively behaves weirdly, as we bounce around elements without a visual indication, and we eventually end-up inside the editor, where pressing becomes 2 spaces. perhaps we should make it jump from cell to cell, with an outline, but without actually entering them?

Clicking reevaluate flashes the screen

That's because the output is temporarily discarded and then added back. I wonder if we can do something to address this? Perhaps we can avoid removing the output until the first stream comes? Or is that too tricky? ๐Ÿค”

Support starting a standalone remote within a Mix project

You can choose the root of a Mix project and we will call "mix deps.get" and start something with elixir --sname -S mix run automatically for you.

Because compiling the project can fail in many ways, we will need to report the output if executing the commands above fail.

Add notebook creation button and file persistence

Once we are done with the markdown importer/exporter, we should add file persistence to the notebook.

Each notebook may be associated to a file, to where we write to periodically (every 5s?).

Creating a new notebook can be done by clicking a button. You will be able to choose between three options:

  1. New: creates a new notebook with no file attached to it - you will have to pick one in order to start auto-save
  2. Open: opens up a notebook from a file and autosaves to the file
  3. Import: opens up a notebook from a file but have no file attached to it

Once you select between New / Open / Import, you will select the execution mode: "elixir" (standalone), "mix" (like standalone but running mix deps.get + mix run inside a Mix project), or "distributed".

Fix inserting cells using keybindings

Currently when we have a cell focused and use keybindings to insert another cell of the same type (n, Esc, n) it usually breaks. This is a LV bug reported here, but opening this just to remember to update LV version once this is fixed.

Disable HTML nodes in markdown

Both in the UI and when importing notebooks, we should disable any HTML nodes in any markdown source. We should also disable the IAL extension as that would allow people to add random onclick events and similar. Alternatively we can keep the extension but explicitly list which attributes we allow (I guess class and not much more).

Improve collaborative features

We should allow names to be assigned to users and give each of them a cursor color.

  • Implement user profiles UI (profiles should be stored on cookies that will be shared across notebooks).
  • Profiles should appear on all pages. Changes to profile are broadcast.
  • Show user cursors based on the user color. Mousing over the cursor should show the user name.
  • Add a users button to the sidebar. Clicking on the user should take to their cursor if they are on insert mode anywhere.

Then there is a v2 of collaborative features, which is comments and allow read-only users, but we are not going to discuss those for now. We will open up issue once this one is closed.

Undo section/cell deletion

We already store the deleted sections/cells, we should have like a modal dialog that shows these and allows to insert them back somewhere. I think this is much better than a global undo/redo, because if the user wants to get back some old code, they doesn't have to undo all the work they did since then.

Add buttons to export / download / view source for the whole notebook

Since we have a markdown source, we should make it easier to download the markdown source if we want. The big question is where we are going to have those buttons. I believe we have two options right now:

  1. Put it under notebook settings in the sidebar

  2. Put those as buttons on the notebook title, similar to what we do to sections, see the red circle here:

Screenshot 2021-03-22 at 08 55 32

Another notebook-wide button that we may want to make accessible is the fork button.

Cancellation UI

For a running/queued cells the run icon should be changed to a cancellation icon. Also a keybinding for that.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.