withoutboats / notty Goto Github PK

A new kind of terminal

License: GNU Affero General Public License v3.0

Shell 0.09% Ruby 0.05% Rust 99.87%

notty's Introduction

notty - not a typewriter

notty is a virtual terminal like xterm, gnome-vte, sh, or rxvt. Unlike these programs, notty is not intended to emulate a DEC VT-series physical video terminal, or any other physical device. Instead, notty is an experimental project to bring new features to the command-line which would not have been possible for the physical terminals other terminals emulate.

Current command-line tools have stagnated (if you prefer, 'stabilized') around the ECMA-48/ISO 6429/ANSI X3.64 escape code protocol, which is defined on the basis of the capabilities of 1980s video terminal devices. Essentially all terminals in use today are virtual terminals being run on devices which are significantly more capable than these ancient machines, and yet the terminal environment has not kept pace with these developments. The last revision of the ANSI escape code protocol was released in 1991.

notty will attempt to remain true to the text-oriented, command-line user interface of the terminal while extending it to include new and more powerful interface features, such as:

Full support for rich text formatting, including 24-bits of color.
Full and correct support for all of Unicode.
Lossless keyboard input.
Inline media content, including raster graphics and structured data.
Dropdown menus, tooltips, and other features which do not strictly reside in the character grid.
Local echo and retained off-screen character grid state to reduce the need for the tty to transmit data back to the terminal.
Subdividing the character grid to enable more complex interface layouts without repeatedly reimplementing that logic in the controlling process.
And more! If you know any features you wish the terminal had, please open an issue and let's talk about this.

Many of these features are not yet implemented.

To achieve these ends, notty will implement a new and more consistent escape protocol than ANSI escape codes. This protocol will be comparatively easy to extend with new features as the project grows. Once a feature set has been stabilized, I will write a framework for creating terminal applications that use notty's features, written against this protocol (allowing other terminals to implement the protocol and support these features as well). This framework will include a polyfill for approximating these features as well as possible in terminals that don't implement notty codes.

This repository is a library which defines an engine for translating both ANSI and notty escape codes into state changes in the terminal's abstract state. This library does not directly implement any means of drawing the state to the screen and is agnostic about the interface it uses to communicate with the controlling process, so hopefully it will be reusable for writing terminals in different graphical environments, for writing screen/tmux-like server-side multi-terminal managers, and for writing SSH clients in non-UNIX environments. Because it implements (most) ANSI escape codes, this terminal is backwards compatible with existing command line tools.

A subdirectory, named scaffolding, contains a minimal graphical terminal using GTK/pango/cairo, intended for testing notty's features interactively. This terminal is buggy and feature poor and not intended for general use.

A major difference between notty and other projects in the same space is that this is just a virtual terminal, and is fully backwards compatible with the existing shell/terminal setup. It does not implement any features of a shell, and it is not attempting to totally supplant any existing paradigms. Graphical terminals based on this library should be useable as drop-in replacements for other terminals, but with new features that can be used to implement better interfaces for command line programs such as shells, text editors, and other utilities.

That said, terminals as they exist today are a pile of ugly kludges. From the kernel's tty/pty subsystem, to the termios ioctl calls which control it, to the terminfo and termcap databases, to the ANSI escape codes they describe, to the ancient codebases of the terminal emulators themselves, this is a universe of arcane and poorly documented old growth code, much of which is no longer actively useful to people in the 21st century - your system ships with a terminfo db page for more than 2500 different terminal devices, nearly all of them extinct, and every new console you open has a baud rate set in the kernel, even though it exchanges data in memory. More advanced features of notty will certainly requiring sidestepping this system to a certain extent: the current plan is to implement a command to "switch" notty to an extended mode; in such a mode, only notty escape codes would be used and the tty's flags would all be unset except for CREAD and ISIG (and maybe not even ISIG).

This implementation is written in Rust, an exciting new systems language from Mozilla.

Much thanks to Thomas E. Dickey, the maintainer of xterm, whose website hosts excellent documentation regarding xterm's behavior, and to Paul Flo Williams, who maintains vt100.net, which hosts manuals for the actual DEC VT-series terminals. Credit also to Gary Bernhardt, whose talk A Whole New World influenced me to pursue this project seriously.

License

notty is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see http://www.gnu.org/licenses/.

Conduct

The notty project is committed to upholding the Rust Code of Conduct. Please see CONDUCT.md for more information.

notty's People

Contributors

Stargazers

Watchers

notty's Issues

Contributing to notty

It doesn't seem reasonable to write a CONTRIBUTING.md for a repo to which I am the only major contributor, but since there are 45 people watching this repo I am optimistically positing that at least one person is interested, at least in the abstract, in contributing to notty, and may read this.

If you've thought about contributing to this project, but were put off - yeah I expect you would be. The core notty crate is changing in deep ways with each commit I make following a design that is only inside my head. It isn't realistic to imagine anyone else could make a large to this section of the code at this point.

But what would be very helpful to me would be if someone were interested in taking over management of either the notty-cairo crate, which draws the screen state of notty to a Cairo context, or the scaffolding crate, which uses GTK+ to create a window with a cairo context and coordinates the interaction between the notty state machine and that context. Hopefully between these crates an actually useful terminal emulator will grow out of this.

If anyone is interested in getting involved with this by taking over one or both those wings of the project, let me know. The interface notty presents to these libraries is fairly stable, and most changes come from improvement to those libraries rather than changes to notty, and I'm happy to answer any questions about how the system works right now.

Build error on OSX 10.12: error: Could not compile `glib`

Hey i'm really hyped to try out notty, but i haven't been able to compile it.

The compilation always crashes because of glib v0.0.8 - i've pasted the relevant errors below.

I'm already using the nightly build of rust, so it should compile fine – since there are other OSX users here that have managed to compile it without errors.

I've tried using a newer package of glib, but unfortunately that didn't help. Also searched on Google and stackoverflow and found some similar problems with glib. But the solutions provided there also didn't help. Like reinstalling the glib located at /usr/local/opt with brew. Didn't help either... Then cargo updateand re-build – still failing at the same point.

What could be wrong with my installation / settings?
Do i need a specific glib version or some extra flags to get past the errors?

Here are the generated errors:

sudo sh build_test.sh
   Compiling notty v0.1.0 (file:///Users/david/github/build/notty)
(...)
test terminal::screen::split::tests::into_6_6::max_right ... ok
test terminal::screen::split::tests::into_6_6::percent ... ok
test terminal::styles::tests::styles_update ... ok

test result: ok. 74 passed; 0 failed; 0 ignored; 0 measured

   Doc-tests notty

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

   Compiling either v1.1.0
   Compiling cairo-rs v0.1.2
   Compiling gdk-pixbuf v0.1.2
   Compiling libc v0.2.21
   Compiling gdk v0.5.2
   Compiling c_vec v1.2.0
   Compiling gdk-sys v0.3.3
   Compiling gio-sys v0.3.3
   Compiling itertools v0.5.10
   Compiling gdk-pixbuf-sys v0.3.3
   Compiling bitflags v0.5.0
   Compiling bitflags v0.4.0
   Compiling pango-sys v0.3.3
   Compiling cairo-sys-rs v0.3.3
   Compiling gio v0.1.2
   Compiling glib-sys v0.3.3
   Compiling gobject-sys v0.3.3
   Compiling glib v0.1.2
   Compiling pango v0.1.2
   Compiling glib v0.0.8
error[E0591]: `extern "C" fn(&std::cell::RefCell<std::boxed::Box<std::ops::FnMut() -> source::Continue + 'static>>) -> i32 {source::trampoline}` is zero-sized and can't be transmuted to `std::option::Option<unsafe extern "C" fn(*mut libc::c_void) -> i32>`
  --> /Users/david/.cargo/registry/src/github.com-1ecc6299db9ec823/glib-0.0.8/src/source.rs:73:70
   |
73 |         glib_ffi::g_idle_add_full(glib_ffi::G_PRIORITY_DEFAULT_IDLE, transmute(trampoline),
   |                                                                      ^^^^^^^^^
   |
note: cast with `as` to a pointer instead
  --> /Users/david/.cargo/registry/src/github.com-1ecc6299db9ec823/glib-0.0.8/src/source.rs:73:70
   |
73 |         glib_ffi::g_idle_add_full(glib_ffi::G_PRIORITY_DEFAULT_IDLE, transmute(trampoline),
   |                                                                      ^^^^^^^^^

error[E0591]: `extern "C" fn(&std::cell::RefCell<std::boxed::Box<std::ops::FnMut() -> source::Continue + 'static>>) -> i32 {source::trampoline}` is zero-sized and can't be transmuted to `std::option::Option<unsafe extern "C" fn(*mut libc::c_void) -> i32>`
  --> /Users/david/.cargo/registry/src/github.com-1ecc6299db9ec823/glib-0.0.8/src/source.rs:91:78
   |
91 |         glib_ffi::g_timeout_add_full(glib_ffi::G_PRIORITY_DEFAULT, interval, transmute(trampoline),
   |                                                                              ^^^^^^^^^
   |
note: cast with `as` to a pointer instead
  --> /Users/david/.cargo/registry/src/github.com-1ecc6299db9ec823/glib-0.0.8/src/source.rs:91:78
   |
91 |         glib_ffi::g_timeout_add_full(glib_ffi::G_PRIORITY_DEFAULT, interval, transmute(trampoline),
   |                                                                              ^^^^^^^^^

error[E0591]: `extern "C" fn(&std::cell::RefCell<std::boxed::Box<std::ops::FnMut() -> source::Continue + 'static>>) -> i32 {source::trampoline}` is zero-sized and can't be transmuted to `std::option::Option<unsafe extern "C" fn(*mut libc::c_void) -> i32>`
   --> /Users/david/.cargo/registry/src/github.com-1ecc6299db9ec823/glib-0.0.8/src/source.rs:109:13
    |
109 |             transmute(trampoline), into_raw(func), Some(destroy_closure))
    |             ^^^^^^^^^
    |
note: cast with `as` to a pointer instead
   --> /Users/david/.cargo/registry/src/github.com-1ecc6299db9ec823/glib-0.0.8/src/source.rs:109:13
    |
109 |             transmute(trampoline), into_raw(func), Some(destroy_closure))
    |             ^^^^^^^^^

error: aborting due to 3 previous errors

error: Could not compile `glib`.
Build failed, waiting for other jobs to finish...
error: build failed

tmux integration

what iterm2 does is amazing, so let’s do that!

Can't build master branch

Joyful compiler errors

error[E0038]: the trait `cmds::EscCode` cannot be made into an object
 --> /home/seunlanlege/.cargo/git/checkouts/notty-encoding-26bc718f9ba2caca/ba5daba/src/client.rs:6:5
  |
6 |     fn write(&mut self, &EscCode) -> io::Result<()>;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `cmds::EscCode` cannot be made into an object
  |
  = note: the trait cannot contain associated consts like `OPCODE`

error[E0038]: the trait `cmds::EscCode` cannot be made into an object
  --> /home/seunlanlege/.cargo/git/checkouts/notty-encoding-26bc718f9ba2caca/ba5daba/src/client.rs:12:5
   |
12 |     fn write(&mut self, code: &EscCode) -> io::Result<()> {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `cmds::EscCode` cannot be made into an object
   |
   = note: the trait cannot contain associated consts like `OPCODE`

error: aborting due to 2 previous errors

error: Could not compile `notty-encoding`.
warning: build failed, waiting for other jobs to finish...
error: build failed

rustc version
rustc 1.22.0-nightly (f1b5225e8 2017-10-01)

[SOLVED] Cannot build notty

Hello @withoutboats ,
I am trying to build notty but get this error :

cargo build --verbose                                                                                                                                                                                                                                       
       Fresh base64 v0.1.1
       Fresh num-traits v0.1.36
       Fresh uuid v0.3.1
       Fresh log v0.3.6
       Fresh unicode-width v0.1.4
   Compiling notty-encoding v0.1.0 (https://github.com/withoutboats/notty-encoding#ba5daba3)
       Fresh num-integer v0.1.32
     Running `rustc /home/julien/.cargo/git/checkouts/notty-encoding-26bc718f9ba2caca/ba5daba3cea3972cd17a3a1b2e4a610385816dcd/src/lib.rs --crate-name notty_encoding --crate-type lib -g -C metadata=52feb8dc22e8ee7a -C extra-filename=-52feb8dc22e8ee7a --out-dir /home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps --emit=dep-info,link -L dependency=/home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps --extern base64=/home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps/libbase64-cbe986a1390b04a9.rlib --cap-lints allow`
   Compiling num-iter v0.1.32
     Running `rustc /home/julien/.cargo/registry/src/github.com-1ecc6299db9ec823/num-iter-0.1.32/src/lib.rs --crate-name num_iter --crate-type lib -g -C metadata=50df698bc905252c -C extra-filename=-50df698bc905252c --out-dir /home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps --emit=dep-info,link -L dependency=/home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps --extern num_integer=/home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps/libnum_integer-52fdddf28cd8e924.rlib --extern num_traits=/home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps/libnum_traits-92bb90166cd1857c.rlib --cap-lints allow`
error[E0554]: #[feature] may not be used on the stable release channel
 --> /home/julien/.cargo/git/checkouts/notty-encoding-26bc718f9ba2caca/ba5daba3cea3972cd17a3a1b2e4a610385816dcd/src/lib.rs:1:1
  |
1 | #![feature(associated_consts)]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

Build failed, waiting for other jobs to finish...
error: Could not compile `notty-encoding`.

Caused by:
  process didn't exit successfully: `rustc /home/julien/.cargo/git/checkouts/notty-encoding-26bc718f9ba2caca/ba5daba3cea3972cd17a3a1b2e4a610385816dcd/src/lib.rs --crate-name notty_encoding --crate-type lib -g -C metadata=52feb8dc22e8ee7a -C extra-filename=-52feb8dc22e8ee7a --out-dir /home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps --emit=dep-info,link -L dependency=/home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps --extern base64=/home/julien/Code/Sources/Perso/notty/scaffolding/target/debug/deps/libbase64-cbe986a1390b04a9.rlib --cap-lints allow` (exit code: 101)

my cargo version :

cargo --version                                                                                                                                                                                                                                            
cargo 0.15.0-nightly (298a012 2016-12-20)

My rustc version :

rustc --version                                                                                                                                                                                                                                            
rustc 1.14.0 (e8a012324 2016-12-16)

How could i solve this problem ?

scaffolding/notty_cairo/notty incorrect scrolling behavior

When the terminal contents exceed the size of the scaffolding window they appear to continue downward in the undisplayed portion of the drawing area rather than pushing old terminal contents above the displayed portion of the drawing area to make room for new stuff as I have come to expect of terminal emulators.

Related work

There might be fundamental differences between notty's approach and iTerm2's image handling, but either way, given there are at least two terminals with image support, there may be a chance for collaboration?

I do a lot of command-line image work, and it would be cool if I could support notty and iTerm both with the same protocol, although this is not a strong need.

Integrating with system clipboard

Currently you can't copy the output of the terminal or paste input into it. This needs to be addressed.

There's also the issue of bracketed paste mode, but that's not as widely used or expected as normal copy and paste.

Wishlist : Magic Wormhole

I have a little idea but don't know how it could be done or if it's even doable or not :

The magic wormhole is a fifo between multiple terminal. Let's take an example.

You open a terminal. You connect to ssh. You then enter a container with "lxc-attach -n foobar".
You open another terminal. You start another ssh session on another machine which can't reach the lxc container.
You start/enter/trigger the magic wormhole to bind the two terminal to get data from one term to the other. That way you can transfer files or even pipe commands between the two terminals.

For now you can mimic this with tmux by using buffer copy and all but it's quite complex and not very reliable.

I discussed a bit of this on alacritty's irc channel and was redirected here. What do you think of this ?
Is it out of scope ? technically not feasible ?

Compilation issue

fukushima ~/s/f/notty & ./build_test.sh 
   Compiling notty-encoding v0.1.0 (https://github.com/withoutboats/notty-encoding#db5c95fd)
   Compiling num-integer v0.1.32
/home/mw/.cargo/git/checkouts/notty-encoding-f90d2d7580660013/master/src/lib.rs:1:1: 1:31 error: #[feature] may not be used on the stable release channel
/home/mw/.cargo/git/checkouts/notty-encoding-f90d2d7580660013/master/src/lib.rs:1 #![feature(associated_consts)]
                                                                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: aborting due to previous error
Build failed, waiting for other jobs to finish...
Could not compile `notty-encoding`.

To learn more, run the command again with --verbose.

Any ideas how to fix this?

Apply styles to images

Images currently ignore styles. While (most) styles cannot be applied to images, this is not 100% correct. Specifically:

The opacity style in the cell containing the image data should be applied to the image as a whole.
The background color of every cell the image covers should be set, even though they will probably not be visible.

Add to Gentoo repository

Typo in readme

There's a typo in Readme.md "not have been possible for the physical terminals other terminals emulate." You probably wanted to remove "other terminals" and replace it with "to"

notty-encoding fails to build on OSX 10.11.6

Hi,

I'm trying to compile notty and have run into an error building the encoding package. Here is the output from my build:

➜  notty git:(master) cargo build --verbose
       Fresh uuid v0.3.1
       Fresh base64 v0.1.1
       Fresh num-traits v0.1.37
       Fresh unicode-width v0.1.4
       Fresh log v0.3.7
   Compiling notty-encoding v0.1.0 (https://github.com/withoutboats/notty-encoding#ba5daba3)
       Fresh num-integer v0.1.34
     Running `rustc --crate-name notty_encoding /Users/mgrunder/.cargo/git/checkouts/notty-encoding-26bc718f9ba2caca/ba5daba/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=46ffb3a5a218e748 -C extra-filename=-46ffb3a5a218e748 --out-dir /Users/mgrunder/dev/rust-foss-projects/notty/scaffolding/target/debug/deps -L dependency=/Users/mgrunder/dev/rust-foss-projects/notty/scaffolding/target/debug/deps --extern base64=/Users/mgrunder/dev/rust-foss-projects/notty/scaffolding/target/debug/deps/libbase64-21fea01bef6f0feb.rlib --cap-lints allow`
       Fresh num-iter v0.1.33
       Fresh num v0.1.37
       Fresh serde v0.6.15
       Fresh mime v0.1.3
error[E0038]: the trait `cmds::EscCode` cannot be made into an object
 --> /Users/mgrunder/.cargo/git/checkouts/notty-encoding-26bc718f9ba2caca/ba5daba/src/client.rs:6:5
  |
6 |     fn write(&mut self, &EscCode) -> io::Result<()>;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `cmds::EscCode` cannot be made into an object
  |
  = note: the trait cannot contain associated consts like `OPCODE`

error[E0038]: the trait `cmds::EscCode` cannot be made into an object
  --> /Users/mgrunder/.cargo/git/checkouts/notty-encoding-26bc718f9ba2caca/ba5daba/src/client.rs:12:5
   |
12 |     fn write(&mut self, code: &EscCode) -> io::Result<()> {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `cmds::EscCode` cannot be made into an object
   |
   = note: the trait cannot contain associated consts like `OPCODE`

error: aborting due to 2 previous errors

error: Could not compile `notty-encoding`.

Caused by:
  process didn't exit successfully: `rustc --crate-name notty_encoding /Users/mgrunder/.cargo/git/checkouts/notty-encoding-26bc718f9ba2caca/ba5daba/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=46ffb3a5a218e748 -C extra-filename=-46ffb3a5a218e748 --out-dir /Users/mgrunder/dev/rust-foss-projects/notty/scaffolding/target/debug/deps -L dependency=/Users/mgrunder/dev/rust-foss-projects/notty/scaffolding/target/debug/deps --extern base64=/Users/mgrunder/dev/rust-foss-projects/notty/scaffolding/target/debug/deps/libbase64-21fea01bef6f0feb.rlib --cap-lints allow` (exit code: 101)

rustc:

➜  notty git:(master) rustc --version
rustc 1.19.0-nightly (06fb4d256 2017-04-30)

OS Information:

➜  notty git:(master) system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: OS X 10.11.6 (15G1217)
      Kernel Version: Darwin 15.6.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Computer Name: MG-MBP
      User Name: Michael Grunder (mgrunder)
      Secure Virtual Memory: Enabled
      System Integrity Protection: Disabled
      Time since boot: 2 days 13 minutes

I was able to find a few places on the net mentioning this error but unfortunately I don't know nearly enough about rust to figure out a solution. Hopefully I'm not missing something totally obvious here 😃

Edit: I get the same error in Ubuntu.

Installation?

Looks like a really interesting project, it would be great to see some installation instructions.

Will `cat /dev/urandom` corrupt my display?

Hi, one thing that was always bothering me about unix terminals is that cat /dev/urandom and similiar could corrupt the display.

It seems to me that "untyped stream of bytes" was never the right abstraction for communication between processes and interactive terminals.

I have not give it too much thought yet, and I'm looking for opinions and resources, but it seems to me that if every stream of bytes was typed (with some default), like eg. prefixed with mime type (defaulting to binary data stream) or simper-http-like protocol, then both communication between processes and between process and terminal could be much robust.

Eg. legacy process like cat, streaming bunch of bytes would output bunch of binary adta and notty would know that all data sent contains no escape codes, displaying eg. hex dump of it, and responding with no "type header" on cat's stdin.

New-world interactive processes, when outputting anything would start with a \0notty/v1\0 header, or something like that, to which terminal would respond with \0notty/v1\0 on stdin, confirming that process is attached to tthe terminal.

The scheme could be completely different.

How do I build and run notty?

I cloned the repo and ran cargo build, but I see:

➜  notty git:(master) cargo build
    Updating registry `https://github.com/rust-lang/crates.io-index`
    Updating git repository `https://github.com/withoutboats/notty-encoding`
 Downloading unicode-width v0.1.3
 Downloading mime v0.1.3
 Downloading base64 v0.1.1
 Downloading serde v0.6.15
 Downloading libc v0.2.7
 Downloading log v0.3.5
 Downloading num v0.1.31
   Compiling unicode-width v0.1.3
   Compiling libc v0.2.7
   Compiling base64 v0.1.1
   Compiling num v0.1.31
   Compiling notty-encoding v0.1.0 (https://github.com/withoutboats/notty-encoding#671ce83f)
   Compiling log v0.3.5
/home/patrick/.cargo/git/checkouts/notty-encoding-f90d2d7580660013/master/src/args/movement.rs:16:9: 16:44 error: expected type, found `/// Direction of the tab character.`
/home/patrick/.cargo/git/checkouts/notty-encoding-f90d2d7580660013/master/src/args/movement.rs:16         /// Direction of the tab character.
                                                                                                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Build failed, waiting for other jobs to finish...
Could not compile `notty-encoding`.

To learn more, run the command again with --verbose.

Is this a regression introduced by withoutboats/notty-encoding@91d7e8c, or is there something wrong with my environment? I've never used Rust before so I may be missing something obvious.

Graphical rendering

Hi,

I am the developer of a new terminal emulator, https://github.com/kovidgoyal/kitty and I am considering adding graphics rendering capabilities to it. To that end, I have posted some thoughts here kovidgoyal/kitty#33

Since you also seem to be trying to develop standards in this space, I thought I would invite you to join the discussion.

compiling on 32 bit system

I was able to build and run the demo terminal by changing line 25 in image_renderer.rs from
'as i64' to 'as i32'.

I know very little about Rust... possible to automatically infer that type?

Wondering if line-based data like Markdown and RTF which allow for higher lines and Headings will be first-class in this project

I love using markdown, but it just never looks right in the terminal I think something like this could really fix that.

build failure

running 2 tests
test cfg::toml::tests::test_default ... FAILED
test cfg::toml::tests::test_update_from_file ... FAILED

failures:

---- cfg::toml::tests::test_default stdout ----
thread 'cfg::toml::tests::test_default' panicked at 'assertion failed: (left == right) (left: "Monospace 10", right: "Inconsolata 10")', src/cfg/toml.rs:244
note: Run with RUST_BACKTRACE=1 for a backtrace.

---- cfg::toml::tests::test_update_from_file stdout ----
thread 'cfg::toml::tests::test_update_from_file' panicked at 'assertion failed: (left == right) (left: "Monospace 10", right: "Inconsolata 10")', src/cfg/toml.rs:244

failures:
cfg::toml::tests::test_default
cfg::toml::tests::test_update_from_file

implement or document text reflow

a common feature request is to reflow all text when the window is reszed.

that’s not trivial and involves a bit of guesswork, but very useful if it works

notty crashes when running "ls"

System Info

+ set -x
+ rustc --version
rustc 1.8.0-dev (dcf8ef272 2016-02-05)
+ cargo --version
cargo 0.9.0 (3e285ce 2016-02-05)
+ git show-ref HEAD
f1cbbec6bdff79f6e4a96e3a9d22815cfaa9c7ec refs/remotes/origin/HEAD
+ cat /etc/debian_version
stretch/sid
+ set +x

Reproduce:

Run ls in the scaffolding terminal.

Road Map to Version 0.2

I would like to start versioning these crates instead of just pegging them at 0.1 at some point soon, incrementing them all to 0.2 to distinguish from the phase up until now. After this, any breaking change to a crate's public API would require a minor version increment (because we're still pre-1.0), and any visible change to any command recognized by notty would also require a minor version increment of the notty library. Between 0.2 and 1.0, at least, I don't think its valuable to keep the versions in sync between different crates associated with the project.

At version 0.2, we'll put notty, notty-encoding, and the GTK terminal (under a better name) on crates.io. Any subcrates of these should be bundled up into these three crates by whatever means is most appropriate (thinking of pangocairo and tty).

Here's the roadmap of features I'd like to see before we increment the minor version:

notty

Multi-grids. This is essential to getting scrolling working properly, and is a major differentiating feature of notty. To elaborate on this, the feature is that you can divide the screen into multiple 'grids', each of which can retain off-screen state as you move around them. You can either divide the screen into two rectangular grids, or you can stack a grid completely over another (so no non-rectangular subdivisions of the screen can be created through this process). Things that can wait: support for adding scrollbars to the subgrids.
Local echo. I'd like to get line-oriented local echo working properly. This is basically implementing a subset of GNU readline, but on the terminal side of the tty. So left and right arrow keys, home/end, and all of the tty special chars (Ctrl-W, Ctrl-U etc). Things that can wait: setting your own control behaviors in local echo, screen-oriented local echo.
Full input. notty should have a mode which sends all keyboard input: key releases, pressing modifier keys, etc. This needs to be revised, I don't think its right at the moment, but I really don't remember.

scaffolding

Basic usability things. Actually get the grid dimensions right, for example. Don't have any known crashes.
A way to scroll the active grid; this will probably be a keyboard shortcut right now and not visible scrollbars.
Non-compile time configurations.
Tabbed terminals. This is the only really big change to scaffolding. Personally, I need this to be willing to start dogfooding the terminal, and I think it would put good organizational pressure on the scaffolding crate, which currently is very poorly organized.
A better name than scaffolding. I'm currently inclined to use the name anterminal after Gary Bernhardt's presentation.

Reach goals

Get notty off of nightly. We're mainly on notty because we use the io::Chars iterator for parsing output.
A build.rs in scaffolding that reliably installs C dependencies like GTK, or at least tells users that they need to install them.

Both of these goals are aimed at getting scaffolding/anterminal's build instructions down to:

Download Rust from rust-lang.org or your package manager.
cargo install anterminal

Is this a dead project?

I realize this project had very lofty goals from the start, and that it would take a tremendous amount of time and effort. That being said, the latest commit was nine months ago. I'm not trying to complain about this as I realize people have busy lives, but I am wondering if you plan on picking this back up or if it is abandoned.

Running `TERM=xterm vi` crashes scaffolding terminal

I wanted to see how compatible is notty with xterm. I understand that it might not work, but probably shouldn't crash notty.
I use Ubuntu 16.04. RUST_BACKTRACE=1 ./target/debug/scaffolding-terminal

stack backtrace:
   1:     0x5567a3415eac - std::sys::imp::backtrace::tracing::imp::write::h9c41d2f69e5caabf
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:42
   2:     0x5567a3418bae - std::panicking::default_hook::{{closure}}::h1f61f3c769fffe7a
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:351
   3:     0x5567a34187b4 - std::panicking::default_hook::hd5bda4e453dfb4be
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:367
   4:     0x5567a341904b - std::panicking::rust_panic_with_hook::hffbc74969c7b5d87
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:555
   5:     0x5567a3418ee4 - std::panicking::begin_panic::h3bbdbb2a23440035
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:517
   6:     0x5567a3418e09 - std::panicking::begin_panic_fmt::h34f5b320b0f94559
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:501
   7:     0x5567a3418d97 - rust_begin_unwind
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:477
   8:     0x5567a344121d - core::panicking::panic_fmt::h1016b85b51d1931f
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/panicking.rs:69
   9:     0x5567a32d3583 - core::result::unwrap_failed::h8e1322835f67bad8
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/macros.rs:29
  10:     0x5567a32c5734 - <core::result::Result<T, E>>::unwrap::h490137277e8cf964
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/result.rs:745
  11:     0x5567a32b99e8 - <notty::output::Output<R>>::csi_code::haf11f6348d6b03e0
                        at /home/michal/misc/notty/src/output/mod.rs:136
  12:     0x5567a32ec4e7 - <notty::output::Output<R> as core::iter::iterator::Iterator>::next::hec2ad9aae17cc842
                        at /home/michal/misc/notty/src/output/mod.rs:258
  13:     0x5567a32f76c1 - scaffolding_terminal::main::{{closure}}::h365f32399a302701
                        at /home/michal/misc/notty/scaffolding/src/main.rs:80
  14:     0x5567a32f12c1 - <std::panic::AssertUnwindSafe<F> as core::ops::FnOnce<()>>::call_once::hbd20d33fdc386e6b
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panic.rs:296
  15:     0x5567a32c02b1 - std::panicking::try::do_call::h9ed515c166360879
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:460
  16:     0x5567a341ff1a - __rust_maybe_catch_panic
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libpanic_unwind/lib.rs:98
  17:     0x5567a32bffbb - std::panicking::try::h7b55dadf400a88ba
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:436
  18:     0x5567a32bf061 - std::panic::catch_unwind::h68fe3a79110fe04a
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panic.rs:361
  19:     0x5567a32bfc44 - std::thread::Builder::spawn::{{closure}}::haf0b8bbea4de2314
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/thread/mod.rs:357
  20:     0x5567a32d79f3 - <F as alloc::boxed::FnBox<A>>::call_box::h9d58fbb764766bfa
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/liballoc/boxed.rs:605
  21:     0x5567a3418004 - std::sys::imp::thread::Thread::new::thread_start::h76badbf9b0ecaf58
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/liballoc/boxed.rs:615
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/sys_common/thread.rs:21
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/sys/unix/thread.rs:84
  22:     0x7f1e755af6f9 - start_thread
  23:     0x7f1e750cfb5c - clone
  24:                0x0 - <unknown>

Wishlist: `dialog` support

Some apps like debconf support dialog for drawing various widgets in (traditional) terminals.

If notty can operate as a drop-in replacement for dialog (or whiptale), then it may in that way find some customers!

Ctrl-D causes a panic

Trying to quit the shell with Ctrl-D out of habit, I got the following stack trace:

➜  scaffolding git:(master) RUST_BACKTRACE=1 cargo run
     Running `target/debug/scaffolding-terminal`
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os { code: 5, message: "Input/output error" } }', ../src/libcore/result.rs:746
stack backtrace:
   1:     0x7ffb5339f240 - sys::backtrace::tracing::imp::write::h7af3c3cfd3e65a783bv
   2:     0x7ffb533a1b2b - panicking::default_handler::_$u7b$$u7b$closure$u7d$$u7d$::closure.44497
   3:     0x7ffb533a1798 - panicking::default_handler::h75ebcc0c8bcd1085rRz
   4:     0x7ffb53397eac - sys_common::unwind::begin_unwind_inner::hcec45e84262f1855m0t
   5:     0x7ffb533982f8 - sys_common::unwind::begin_unwind_fmt::h62c174e37e54c321sZt
   6:     0x7ffb5339e6b1 - rust_begin_unwind
   7:     0x7ffb533d0e7f - panicking::panic_fmt::h5f920bbc722cd548lcM
   8:     0x7ffb532e106b - result::unwrap_failed::h14713097509445665626
                        at ../src/libcore/macros.rs:29
   9:     0x7ffb532ee0cb - result::Result<T, E>::unwrap::h4578224278132733853
                        at ../src/libcore/result.rs:687
  10:     0x7ffb532e288a - main::_$u7b$$u7b$closure$u7d$$u7d$::closure.10320
                        at src/main.rs:67
  11:     0x7ffb532e21c0 - std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::closure.10309
                        at ../src/libstd/thread/mod.rs:277
  12:     0x7ffb532e2169 - sys_common::unwind::try::try_fn::h14296686403988533152
                        at ../src/libstd/sys/common/unwind/mod.rs:127
  13:     0x7ffb5339e63b - __rust_try
  14:     0x7ffb5339c20d - sys_common::unwind::inner_try::h27915955daa5bb0coXt
  15:     0x7ffb532e20ce - sys_common::unwind::try::h12233384044057444796
                        at ../src/libstd/sys/common/unwind/mod.rs:123
  16:     0x7ffb532e1f40 - std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::closure.10284
                        at ../src/libstd/thread/mod.rs:277
  17:     0x7ffb532e2c0c - boxed::F.FnBox<A>::call_box::h7394299706424739120
                        at ../src/liballoc/boxed.rs:541
  18:     0x7ffb533a06a9 - sys::thread::Thread::new::thread_start::h47cad50136a0a353LOy
  19:     0x7ffb50770181 - start_thread
  20:     0x7ffb50f9247c - __clone
  21:                0x0 - <unknown>

It looks like the issue is in scaffolding/src/main.rs at:

    thread::spawn(move || {
        let output = Output::new(BufReader::new(tty_r));
        for cmd in output {
            tx_out.send(cmd.unwrap()).unwrap();
        }
    });

Because I suppose in src/output/mod.rs, Output::next is returning None.

Scaffolding does not draw the cursor to the screen

Currently, the cursor is not drawn to the screen at all. A good first step would be changing the background color of the cursor cell, so that at least we can see the cursor.

In the long term, the cursor can currently be styled in any way that text can. Its not exactly clear what this means. For example, if you apply the bold style to the cursor, does that mean the character under the cursor should be bold, or does it mean nothing at all? Are there only some styles that actually are applied to the cursor, and trying to apply others is just ignored? How do the styles applied to the cursor interact with the styles applied to the cell the cursor is over? This needs to be determined and implemented.

ANSI escape codes not yet implemented

There are a large number of ANSI escape sequences which are not currently implemented, but most of these are almost never actually used. This is a tracking issue for any ANSI escape sequences seen in the wild by anyone testing or using a terminal based on this library.

The escape codes that are most important to implement are those that are called out in the terminfo sheets for xterm and its offspring, because this is the protocol, and not the full features of an ANSI terminal, that well-written programs are written against. The only major gap here is the terminfo item csr, which is called in VT manuals DECSTBM, or Set Top and Bottom Margins.

Page margins are a very complicated feature to implement, and not very efficient. More importantly, though, their interaction with other commands that move the cursor or scroll the screen is very underspecified in the documentation of VT series terminals and their emulators. I have read these manuals with interest and still have no idea how exactly this is supposed to work.

vim uses this feature, and seems a bit slower without it. It would be a good thing to add for legacy. However, a more powerful, cleaner, but similar feature for future programs would be just to allow multiple visible grids on the screen, which have their own boundaries and cursor, and scroll separately. Logically this is what this feature is being used to implement anyway.

Store Config info on the Terminal struct

As of writing this issue, config information is stored on a global constant called CONFIG, which is imported into several scopes, both in notty and in notty-cairo. Because configuration info is stored on a constant, the only way to reconfigure notty is to recompile it. This is not a great user experience.

There are several steps that need to be taken to make notty configs convenient to change, this issue is only the first step. Here is a broad roadmap of the whole process to making notty configured dynamically:

The Terminal holds a Config struct, from which all the configurable values for that terminal are accessed. At first, all terminals can be instantiated with the CONFIG constant for this. This is what this issue is about.
This Config struct is made into an argument, and the config info in src/cfg.rs is moved into the scaffolding terminal (though the Config type definition will stay inside of notty).
Add a way for scaffolding to create a Config struct at runtime from some source (probably a TOML file). This way, users can configure the scaffolding terminal without recompiling notty.

This is a great task for someone to pick up to learn the notty code base better (and probably also Rust). It doesn't interact much with the core logic of notty, but it does require you to explore how it is structured. I am glad to provide guidance and answer any questions about this task.

To sort of restate the issue, once this is done, cfg::CONFIG should be invoked in only one location, stored somewhere on the Terminal struct, and everywhere it is invoked now should instead invoke the value stored on the Terminal.

Please drop a note here if you start to work on this and I will assign the issue to you.

Bug when using Command-T in vim

I haven't even tried to figure out what esc codes are actually involved, but the vim plugin command-t interacts very poorly with notty right now. Arrow keys don't work at all, and it is quirky in other ways as well.

Arrow key issues may have to do with the difference between "application mode" and "normal mode," an inexplicable "feature" of ANSI terminals.

Here is a script log of me opening vim and using command t, attempting to press the down arrow:

log file

Traffic?

I notice this repo has had a bit of a traffic spike today, but in the info GitHub provides me, I can't find a novel source of traffic that could explain it. If anyone comes across this and knows what's up, I'd like to know. 😄 I love answering questions about this project and its goals, and I know its not in a state which speaks for itself just yet.

Local echo & line discipline

I ❤️ issue blogging.

ANSI terminals have no concept of "local echo" - nothing entered at the keyboard of a terminal is written to the screen unless it is written to the output side of the tty connection. However, the tty subsystem of a UNIX kernel does implement an echo feature, in which it echos characters it receives back to the terminal. A number of options exist to modify this feature (such as determining how ctrl characters and backspace ought to be echoed), and it is often combined with the tty's "canonical line discipline," which buffers input until a newline character is received.

These features of the tty make implementing simple programs much easier, but they have two limitations:

The maximal interface features the tty can provide are still quite minimal (cat with no arguments demonstrates them). You can enter keys and press backspace, and you have access to some shortcuts like ^W. Arrow key navigation doesn't work in any capacity.
This buffering is performed on the same system as the controlling process. If that is not the localhost, each key must perform a roundtrip before being displayed on the terminal, introducing interface latency.

The simplest solution to the second issue is to implement echo features inside the terminal. The solution to the first is for this echo to be more featureful than the tty's echo.

These are the MVP features of local echo. Each needs to be independently togglable:

Echo character input (excluding ctrl chars).
Erase left on backspace, erase right on delete.
Line buffering.
Word and line erase characters.
Horizontal arrow keys.
Vertical arrow keys (when not in line buffering mode).

Other than the last two points, these are features supported by the tty line discipline. The tty also does a few other things. It generates signals from input; this is basically the only feature that makes the most sense to keep in the tty, because it needs to be on the remote machine but it can't be in the controlling process. Input can be turned off with Ctrl+S, this feature does not seem worthwhile to re-implement.

Beyond this, there are a number of really radical extensions that could be implemented by notty, but may not be good ideas:

History. Store history of input lines when in line buffering mode and allow it to be accessed and searched through. This is logic that is re-implemented, or not implemented at all, in different shells, repls, and other line-oriented program interfaces.
Syntax highlighting. Commands which carry some sort of regular expression which will be matched on what's written to the screen later, in order to style it.
Arbitrary commands: Associate key presses to "echo" arbitrary sequences of ansi codes; if this were powerful enough (meaning among other things that it can map key sequences, and not only keys), this could be used to implement vim/emacs-like interfaces. Of course, only those commands which deal with modifying and navigating the local buffer could be implemented purely locally.
Scripting language. (I think this is probably a bad idea). Receive scripts that drive terminal behavior, can be registered to trigger on receiving certain input, etc. Technically, this could be used to implement all of the above features. However, scripts cannot be allowed to send arbitrary data to the controlling process, or a program run while in ssh could exit the ssh session and execute arbitrary code from the user's local shell. There are certainly subtler security issues with this, and I am not currently in favor of going in this direction.

Figure out how font extents are supposed to work

We need to know the dimensions of the visible area of the terminal in terms of characters for a variety of use cases. Currently, we attempt to derive this information from the FontExtents struct of the cairo library, which has several fields.

However, the information presented by this struct is not the information we actually want, and it results in weird dimensions in several ways - opening vim in the scaffolding terminal is a good demonstration of the issue.

If someone could figure out how to derive from a cairo canvas or a font the actual dimensions of a character in that font, that would be really helpful. Assuming a monospace font is fine.