observablehq / framework Goto Github PK

We should have an intrapage table of contents sidebar, like VitePress, automatically derived from Markdown headings.

This would give more flexibility, and let us turn off the spurious console output when running tests.

Generators.input, Generators.observe, and Generators.queue should be rewritten as async generators, now that async generators are widely supported by browsers. Ref. observablehq/stdlib#126

Like #30, but for fetch. We only check for ./ currently which doesn’t account for these other types of paths.

https://github.com/observablehq/cli/blob/62d025157ca5e07e002ee0104705032848307f0e/src/javascript/features.js#L68

to reproduce:

cd cli/
yarn dev

open http://127.0.0.1:3000/markdown

mv docs/markdown.md ./

navigate to http://127.0.0.1:3000/javascript by clicking on "JavaScript reference" in the side bar

↑ {
  type: 'hello',
  path: '/markdown',
  hash: '13f37e067d6239136c9165ffd158da4161de47b99b8c55d7441d387ffd4b1e04'
}
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: ENOENT: no such file or directory, open 'docs/markdown.md'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'docs/markdown.md'
}

We should have the equivalent of Observable’s data table display for a nice visual summary of tabular data.

It’d be nice if code blocks had convenient copy buttons, like they do on GitHub and VitePress.

It’d be nice if you could use file attachments in local ES modules. I’m imagining it would work like this:

import {FileAttachment} from "npm:@observablehq/stdlib";

const foo = await FileAttachment("foo.csv").json();

In this case, the path foo.csv would be relative to the current ES module.

The dual meaning of importing the Observable standard library would be a little odd, though. I’m not sure how that would work… and maybe this is a pipe dream.

As an alternative to including a custom stylesheet such as dashboard.css, we should allow named themes to be applied via YAML frontmatter, and provide built-in stylesheets for these themes. The current theme could be moved to default.css.

We currently clear the display when a cell goes into the pending state. It would be better if the old display persisted temporarily until the new variable resolves or rejects, or until the new variable calls display the first time.

Depends on DatabaseClient #46. A client-side alternative to a data loader #38, presumably. Depends on #8 to implement tagged template literal code blocks, plus some additional directives to indicate which database you’re querying, and what name the result set should be exposed as.

The CLI should offer a way to sign-in via Observable, saving a personal access token, and thereby allowing queries to be proxied against the Observable-hosted data connector, and other platform features.

It’d be nice if we could give some visual indication that “something is happening” when code is running. But it’s not immediately clear how we would do this, or if we could do this, for both inline expressions and fenced code blocks. And also some code blocks don’t display anything, so those probably shouldn’t display a loading indicator. 🤔

If the socket closes, the client never tries to reopen and reconnect.

Rather than simply enumerating all pages in the project, we should allow (or require) the user to specify the sidebar navigation structure explicitly, including sections, titles, and paths.

Ref. #41 (comment)

Currently the footer always says “© 2023 Observable, Inc.” but that doesn’t make sense for user content. We should allow it to be configured, and probably default to something like “Built with Observable” with a link to observablehq.com or some such.

E.g.,

We don’t currently document that you can use the show directive to echo the contents of a live JavaScript code block.

```js show
1 + 2
```

We might also want to call it “echo” for parity with Quarto.

As I was looking at the options, I think I'd like us to reserve the -h shorthand for --help (and -v for --version). if we want to keep it short, we could follow live-server and use --host instead of --hostname, maybe?

EDIT: -h is an option for a subcommand, and we can still support observable -h to mean help (though that might be confusing?)

We currently detect link elements with href attributes and promote them to file attachments, but we should do the same with other types of elements that commonly reference static files.

https://github.com/observablehq/cli/blob/0081db4f685ef635df42d0725ec2bbac5e87fa6c/src/markdown.ts#L64

It’s missing two things:

• The stylesheet isn’t included in the npm distribution so we have to load it from GitHub.
• The stylesheet and the ESM source use __ns__ as the namespace rather than oi-3a86ea.

This is really a bug in the Inputs repo, but I’m filing it here because this is the first time we’re consuming it as an ES module.

Symbol.for("observablehq.display")? Ref. https://docs.deno.com/runtime/manual/tools/jupyter#rich-content-output, but in our case it should return an HTML element/node for insertion, rather than a string, since we’re running in the client and we don’t need serializable values.

Related to DatabaseClient #46, how might we support Secret? Presumably secrets would need to be baked-in to the static site during the build step. (You could avoid this by only using secrets server-side from data loaders #38.)

When the preview server encounters a syntax error, it logs the message to the console:

SyntaxError: Assignment to external variable 'x' at line 4, column 1

It would be nicer if this message included the file with the syntax error, and a snippet of code to point to the error in context.

If a project is hosted on GitHub (or other hosting service), it’d be nice if we could automatically generate an “edit me” link in the footer.

Currently our file watcher only handles change events. This means that a command such as git checkout docs/index.md which generates a rename event is ignored, and then the file watcher becomes detached from the file and no longer reports any changes.

https://github.com/observablehq/cli/blob/0081db4f685ef635df42d0725ec2bbac5e87fa6c/src/preview.ts#L142

Had this, and reordering the big numbers broke the update

Before:

<div class="grid grid-cols-4" style="grid-auto-rows: 86px;">
  ${BigNumber(1, {title: "title a"})}
  ${BigNumber(2, {title: "title b"})}
  ${BigNumber(3, {title: "title c"})}
  ${BigNumber(4, {title: "title d"})}
  ${BigNumber(5, {title: "title e"})}
</div>

After:

<div class="grid grid-cols-4" style="grid-auto-rows: 86px;">
  ${BigNumber(1, {title: "title a"})}
  ${BigNumber(4, {title: "title d"})}
  ${BigNumber(2, {title: "title b"})}
  ${BigNumber(3, {title: "title c"})}
  ${BigNumber(5, {title: "title e"})}
</div>

We currently rewrite static imports as dynamic imports, but we await each import individually. This isn’t urgent given that we also use modulepreload to preload modules in parallel, but it’d be nice if we used Promise.all to await all of a cell’s imports rather than serializing them.

The code currently assumes that the project will be served at the root of a domain and generates absolute paths starting with slash. We should support a serve root option (different from the current source root option) that allows the site to be served from a subpath.

Related to custom inspectors #51, we could automatically detect tabular data and give a better default display using the summary table #23.

As noted, the rewriteImports function currently assumes that the ES module is in the root folder, but it should be allowed to be in a subfolder.

Our syntax highlighting colors aren’t very good, especially in dark mode. We should improve them.

There seems to be a race condition in how the display is reset resulting in occasional duplicate errors:

To reproduce this, I created the following Markdown file:

```js
const data = FileAttachment("data.json").json();
```
```js
data
```

Then I wrote a script to generate the data.json:

import {getRandomValues} from "crypto";

process.stdout.write("[");
for (let i = 0; i < 1000; ++i) {
  await new Promise((resolve) => setTimeout(resolve, 150));
  if (i) process.stdout.write(",");
  process.stdout.write(`\n{"value": "${randomHex(640)}"}`);
}
process.stdout.write("\n]\n");

function randomHex(n) {
  return Array.from(getRandomValues(new Uint8Array(n)), (b) => b.toString(16).padStart(2, "0")).join("");
}

Which I ran as:

node --loader tsx/esm test.ts > docs/data.json

I believe it has something to do with how the old variable is deleted — this is equivalent to setting its value to undefined, which results in an asynchronous evaluation. Perhaps this prevents the variable from cleaning up after itself correctly?

https://github.com/observablehq/runtime/blob/5116a31ccd401137f26952e79cd5bb38e7bbc49c/src/variable.js#L204-L206

Import maps are static (see WICG/import-maps#92 for context), so when previewing a site, we can’t add a library to the import map after the page has loaded. This results in a broken experience:

TypeError: Failed to fetch dynamically imported module: npm:whatever

A workaround would be to detect in the diff when a library needs to be added to the import map. In this case we would reload the entire page rather than doing an incremental update. (A full reload isn’t ideal, but at least the page will continue working.)

A more involved workaround would be to avoid import maps entirely during live preview, and instead transpile the JavaScript directly to use URLs. This would avoid needing to reload the page. The only wrinkle is that we’d also need to transpile local ES modules, but I expect we’ll probably have to do that anyway if we e.g. want to support FileAttachment in ES modules, too. It’s also kind of a shame because I was excited to use import maps.

Starting with this trivial Markdown file:

hello world

If I edit the file to make “hello world” an h1:

# hello world

I end up getting two h1 elements:

Cc @wiltsecarpenter.

Rather than loading ES modules from jsDelivr, we could download and self-host them during the build step to remove external dependencies and isolate against new releases.

In a case like this…

<div class="grid grid-cols-4" style="grid-auto-rows: 86px;">
  ${modes.map(mode => BigNumber(events[0][mode], {title: mode}))}
</div>

we really want to avoid the wrapper SPAN element so that each interpolated big number is a direct descending of the grid container. Maybe we can do this using document fragments? And comment nodes to denote where the fragment starts and ends?

We currently copy-over the client.js as-is. But as the code grows, we should bundle and minify during build it so that it loads faster.

We should formalize the concept of a data loader: a script that runs during build (and as needed during preview) to materialize a file attachment.

As an initial sketch, it could be something like this:

• A data loader is an arbitrary executable. Most often we expect them to be written as Node.js programs (possibly with dependencies in package.json), but we want to support arbitrary executables for a polyglot workflow, e.g. writing a data loader in R, Python, Julia, Zig, etc.
• A data loader, when run, generates and outputs the contents of a single file to stdout.
• A data loader doesn’t take any arguments; it is invoked automatically by the CLI.
• A file generated by a data loader is accessed like any other file (e.g., via FileAttachment or fetch); nothing on the client indicates any difference between a “static” file and a file that is generated by a data loader.
• The name of the generated file corresponds to the name of the data loader (i.e., file-based routing). For example, if the generated file is named docs/foo.csv, the corresponding data loader would be named docs/foo.csv.ts or docs/foo.csv.py.
• The double extension (e.g., .json.ts) is used to distinguish data loaders from static files.
• [Optional] We could support SQL shorthand for data loaders, e.g., docs/foo.csv.sql. For this we’d need metadata to specify which database to use, and we’d infer the output format (such as CSV) from the file name.

During preview:

• The server automatically runs the data loader when the corresponding file it generates is requested, if needed.
• The server automatically re-runs the data loader when the data loader is edited, if a client is currently watching the corresponding output file.
• We leverage reactive updates: by simply editing and saving a change to a data loader, the data loader runs automatically and the browser updates to reflect the new data as an incremental update without reloading.
• The server keeps a cache of data loader outputs. It only runs a data loader if the desired output is missing from the cache, or if the output is older than the data loader source file.

During build:

• Any data loaders that correspond to referenced files are run, generating a corresponding output file.
• If a data loader is not referenced as a file attachment by any page in the project, it is not run.
• [Optional] The build step could respect the cache, allowing incremental builds.

The file-based routing approach described above isn’t a requirement. But it does have some nice properties that we should seek to maintain:

• You can add or remove a data loader just by adding or removing a file.
• You can rename a data loader just by moving a file.
• It’s (more) obvious that a file is generated, and where the source code is for a generated file.
• You don’t need to manage some external mapping between files and data loaders.
• Each file can be generated independently, and in parallel, on demand.
• The CLI can automatically manage the data loader cache.
• The CLI only needs to run data loaders that are referenced, both during preview and build.

The preview server could watch attached files, and when an attached file changes, it could automatically re-run referencing cells. This way you can update a file and the preview updates immediately without having to reload.

Note that we might want to debounce file watching in the case of streaming writes.

I’m not sure how we would fix this, but our current approach of eval’ing JavaScript source during live preview makes it incompatible with import.meta; however, you can use import.meta when the page loads (and statically when building the site). So, it’d be nice if we could make import.meta work consistently.

There are a number of Observable standard library features that we don’t want to support going forward (viz. observablehq/stdlib#32 observablehq/stdlib#31). Now would be a good time to remove these deprecated features.

In addition to data loaders #38 that run server-side, we’ll want the ability to run client-side database queries, too. When the site is built statically, presumably these queries would depend on an external server.

Authentication is still TBD, but we’ll probably want (optionally) a shared secret statically-baked into the built pages, in addition to the origin restrictions that the database proxy may enforce.

Currently inline expression ${…} are only supported for node content. So you can’t do this:

```js
const link = "https://example.com";
```
This is a <a href=${link}>link</a>.

You can workaround it using Hypertext Literal like so:

```js
const link = "https://example.com";
```
This is a ${htl.html`<a href=${link}>link</a>`}.

I’m not immediately sure how we would support this. It might involve writing our own Markdown parser? Or at least pre-processing the content to remove the live code expressions before passing it to markdown-it? Presumably we could repurpose parts of Hypertext Literal to parse the HTML.

We currently list any notebook without an h1 as "Untitled" in the sidebar.

An author already chooses a name for their notebook when they pick a file name. If possible we should fall back to displaying the file name instead of "Untitled".

We currently use standard fenced code blocks

```js
1 + 2
```

This can be surprising because it behaves differently from standard Markdown: the code runs instead of being displayed! And also it’s inconsistent with non-JavaScript code blocks, which do display as normal.

So instead, we could follow Quarto’s example by using curly braces:

```{js}
1 + 2
```

I also thought about using script tags, or some such…

<script type="observable">
1 + 2
</script>

But that seems weird. 🤔

In a JavaScript code block like this:

x = 1

We could implicitly add a top-level declaration like so, treating it like so:

let x;
x = 1;

This could even work with conditional assignment, i.e.:

if (condition) x = 1;

Could become:

let x;
if (condition) x = 1;

This might make it easier to port from today’s Observable notebooks, and is probably friendlier than the current error that we throw.

SyntaxError: Assignment to external variable 'x'

We could use localStorage or a cookie to remember whether the user wants the sidebar open.