Through my tinkering, I have noticed that accessing an uniform array (i.e provided from the program) in the fragment shader's for-loop is quite expensive (my computer's fan really kicks in). We should ideally figure out why this is happening and if it can be worked around as arrays are far more pleasant to look at/read than what we have now. This will also allow us to scale much better. Potential options that I have done no real digging into include:

• dynamically creating the shader code so we don't need arrays (I have my suspicions)
• fixing the potential underlying problem (a parameter we don't know of? a different way of structuring data?)
• creating an array of structs that have color, texture, and slider values(is this any better?)
• Punting, and accepting that we will have a fixed number of channels, which could be very high though - (repeated) code is cheap!
• something else entirely

Package may or may not work, but it would be good to clear up warnings, regardless:

$ npm run-script build-component

> @hubmap/[email protected] build-component /Users/chuck/github/hubmap/vitessce-image-viewer
> rm -r dist && rollup -c


src/index.js → dist/index.js, dist/index.es.js...
(!) Plugin node-resolve: preferring built-in module 'fs' over local alternative at 'fs', pass 'preferBuiltins: false' to disable this behavior or 'preferBuiltins: true' to disable this warning
(!) Plugin node-resolve: preferring built-in module 'util' over local alternative at '/Users/chuck/github/hubmap/vitessce-image-viewer/node_modules/util/util.js', pass 'preferBuiltins: false' to disable this behavior or 'preferBuiltins: true' to disable this warning
(!) Plugin node-resolve: preferring built-in module 'module' over local alternative at 'module', pass 'preferBuiltins: false' to disable this behavior or 'preferBuiltins: true' to disable this warning
(!) Plugin node-resolve: preferring built-in module 'path' over local alternative at 'path', pass 'preferBuiltins: false' to disable this behavior or 'preferBuiltins: true' to disable this warning
(!) Circular dependencies
node_modules/@loaders.gl/core/dist/esm/lib/parse.js -> node_modules/@loaders.gl/core/dist/esm/lib/loader-utils/parse-with-worker.js -> node_modules/@loaders.gl/core/dist/esm/lib/parse.js
node_modules/@luma.gl/gltools/dist/esm/state-tracker/unified-parameter-api.js -> node_modules/@luma.gl/gltools/dist/esm/state-tracker/track-context-state.js -> node_modules/@luma.gl/gltools/dist/esm/state-tracker/unified-parameter-api.js
node_modules/@luma.gl/webgl/dist/esm/webgl-utils/texture-utils.js -> node_modules/@luma.gl/webgl/dist/esm/classes/texture-2d.js -> node_modules/@luma.gl/webgl/dist/esm/classes/texture.js -> node_modules/@luma.gl/webgl/dist/esm/classes/resource.js -> node_modules/@luma.gl/webgl/dist/esm/webgl-utils/index.js -> node_modules/@luma.gl/webgl/dist/esm/webgl-utils/texture-utils.js
...and 5 more
(!) Unresolved dependencies
https://rollupjs.org/guide/en/#warning-treating-module-as-external-dependency
fs (imported by node_modules/@loaders.gl/core/dist/esm/node/read-file-sync.node.js, node_modules/@loaders.gl/core/dist/esm/node/write-file.node.js)
util (imported by node_modules/@loaders.gl/core/dist/esm/node/write-file.node.js)
module (imported by node_modules/@loaders.gl/loader-utils/dist/esm/lib/library-utils/require-utils.node.js)
path (imported by node_modules/@loaders.gl/loader-utils/dist/esm/lib/library-utils/require-utils.node.js)
(!) Import of non-existent export
node_modules/@math.gl/culling/dist/esm/lib/plane.js
1: import _classCallCheck from "@babel/runtime/helpers/esm/classCallCheck";
2: import _createClass from "@babel/runtime/helpers/esm/createClass";
3: import { Vector3, equals as _equals, assert, MathUtils } from 'math.gl';
                                                ^
4: var scratchPosition = new Vector3();
5: var scratchNormal = new Vector3();
created dist/index.js, dist/index.es.js in 6s

The image appears to wrap when using TIFF. My hunch (based on some investigation) is that the geotiff.js library is requesting too much data, which means we might need to make a PR into their repo.

User story
We need a way to map n colors to n separate 0 - (1 - 2^b) (where b is number of bits per channel) ranges and vice-versa to allow for many color channels. This will require some research into color mappings and glsl.

This will make panning look better as well as zooming until we change the cache to rely on rendering instead of loading the data.

Take scale from metadata and render it as a little yard-stick in the viewer

The vertex shader is essentially a clean pull from deckgl's BitmapLayer and I don't think all of it is necessary.

Check if we're using dark mode appropriately? Or just hardcode some css.

Originally posted by @mccalluc in #81

Currently all channels of the pyramid for the zarr example are stored as [4, 512, 512] "tiles", where the first dimension is the channel. This makes for fewer requests, but we want to add some logic for determining how to initialize the zarr connections if these are separate pyramids (similiar to the Tiff example).

We assume that the chunking is this way at the moment, but it should be simple to add some logic. If the urls in the config for the incoming sourceChannels are all the same, assume same pyramid and respect the ordering of the channels. If sourceChannels have different urls, initialize separate pyramids.

We need to add some anti-aliasing robustness to prevent this sort of thing from happening:

Follow up to #82

When the demo is pushed out, we need a little bit of information about what you're seeing on the page, perhaps above the controls? Something like:

vitessce-image-viewer ("Viv")

A viewer for high bit depth, high resolution, multi-channel images using DeckGL over the hood and WebGL under the hood. More information and documentation on github.

• We need to allow multiple layers overlapping to each have an individual opacity

• We want the names of channels to be determined by what is in source-info.js i.e the keys of the url.
• Also We want to be able to turn channels on/off (by passing down a [0,0,0] color to its color).

I'm lumping these together because they are conceptually similar.

When I swap the commenting in source-info.js, I just get an error from geotiff.bundle.min.js. Is this a regression, or is there some other change I need to make at the same time?

In preparation of a z-stack (and probably a good idea anyway) is to utilize the ability to make the cache size a function of memory as per visgl/deck.gl#4246

Eslint has worked for me... but if you find something newer and better, great. (Strongly prefer CI checks to just relying on pre-commit hooks, but they could be complementary.)

At the minimum, bundle with a non-tiling GeoTIFF accessor function. Optimally, we would get tiling working, but the leap to that isn't going to be much more.

• Makes it easy to identify where something comes from.
• Distinguish single-export files from utils.
• Typically, we'll be using default exports, so being out of sync isn't an issue... but it will be nice to know what we should name it on import.

@ilan-gold : Is this ok with you?

I understand that webpack is doing the demo and rollup is doing the packaging... but it feels like we should be able to use one build tool, even if two separate configs are necessary? I'm guessing webpack came from the CRA, but rollup is easier to configure for packaging?

My mental model is that both tools are pretty much in the same space? Scenarios I'd worry about: • Someone needs to fix the build, and they end up in the wrong config file, and none of their changes seem to make a difference.
• We need new language features, and we have to figure out how to get them working in both tools.

And maybe it should be rooted at dist, and not just include it?

$ npm publish --access public
npm notice
npm notice 📦  @hubmap/[email protected]
npm notice === Tarball Contents ===
npm notice 1.1kB LICENSE
npm notice 1.6MB dist/index.es.js
npm notice 1.6MB dist/index.js
npm notice 56B   src/data-utils/index.js
npm notice 214B  src/index.js
npm notice 38B   src/XRLayer/index.js
npm notice 1.6kB src/microscopy-viewer-layer.js
npm notice 2.2kB src/microscopy-viewer.js
npm notice 2.0kB src/tiling-utils.js
npm notice 2.3kB src/XRLayer/xr-layer-fragment.js
npm notice 493B  src/XRLayer/xr-layer-vertex.js
npm notice 5.1kB src/XRLayer/xr-layer.js
npm notice 1.6kB src/data-utils/zarr-utils.js
npm notice 2.1kB package.json
npm notice 3.5MB dist/index.es.js.map
npm notice 3.5MB dist/index.js.map
npm notice 3.2kB README.md
npm notice === Tarball Details ===
npm notice name:          @hubmap/vitessce-image-viewer
npm notice version:       0.0.1
npm notice package size:  1.8 MB
npm notice unpacked size: 10.1 MB
npm notice shasum:        c830bc13561b98a788939e51a0bcdd5646f656b6
npm notice integrity:     sha512-8YsHUvxnrF7bL[...]wZch5rTj9GnYQ==
npm notice total files:   17
npm notice
+ @hubmap/[email protected]

I'd do it myself, but I don't have privs:

• Only allow squash.
• delete branches after merge.

User story
Through my journey, I have noticed there is more than one way of bringing shaders into your project. This should be thoroughly investigated and a solution settled on that makes the most sense with our needs.
Preferred solution

Possible alternatives

• Right now the hsv converter assumes a black background
• We need to allow for white background (potentially discernable from within the tiff metadata)

Rather than a really high degree of coverage, I would suggest getting at least a few simple tests in at different levels. If bugs are fixed in the future, we don't want the bug-fixer to also have the responsibility for setting up the test infrastructure in the first place.

• unit tests for math
• cypress, maybe?
• something in between

And/or in the title/url section of the viv landing page on GH.

Both zarr.js and geotiff.js rely on a metadata-like object (fetched) to give the necessary information for fetching the underlying data. We should write something to cache this data as it only needs to really be fetched once.

The Vitessce approach of demo versions unconnected to NPM versions is not something to emulate. On the other hand, Alex with 4DN has put demos into the NPM release, and you can view it via unpkg, ie: https://unpkg.com/@hms-dbmi-bgm/[email protected]/index.html

Serving gh-pages from a docs directory, which in turn references unpkg and npm is another option.

It would be good if the same demo code that gets built and deployed could also be used for local development.

I'm expecting the picture to get brighter as the sliders go up (and the thick part of the slider gets longer). I'm also not sure that the numbers in the control are useful... but maybe they are during development?

mccalluc

Do we want to move @deck.gl/* into peer dependencies? Vitessce itself will have these, and if the versions do diverge, I don’t want to be bundling multiple versions of a big dependency.
but not a blocker now.

Ilan Gold

I don't think so. We want the package to be usable without a @deck.gl component already existing. I don't think people look at peerDependency warnings but I might have that wrong.
I suppose we could just make it clear in the docs that you absolutely need to install your own @deck.gl to use the standalone version.

See #74 for outstanding suggestions.

Try converting to 512x512 size tiles. Mapbox switched to 512x512 tiles a while back and said it improved performance.

Right now we support tiff and zarr as formats, but in the future others could have data in a different format and we should expose the ability to create custom getters for the data. I think we should standardize how these data utilities work and what they must implement to plugin with the viewer. We shouldn't be dumping everything in state in React if these aren't going to change for a layer.

PR #81 puts a foot forward in this direction but I would ultimately like to see this implemented as a class. I'm a bit tired so maybe I'm thinking about this wrong; any feedback is appreciated!

class ImageLoader {
  constructor(connections, minZoom, imageWidth, imageHeight, tileSize, channels) {
    // required state for zarr
    this.connections = connections;
    this.minZoom = minZoom;
    this.imageWidth = imageWidth;
    this.imageHeight = imageHeight;
    this.tileSize = tileSize;
    this.channels = channels;
  }

  async initilize() {
    // Perform work to get connections, minZoom, imageWidth, imageHeight, tileSize
    throw Error("Need to implement .initilize method");
  }

  async getTile() {
    // Perform logic to get tile from data-specific connections
    throw Error("Need to implement .getTile method");
  }
}

class ZarrLoader extends ImageLoader {
  async initilize({ channelSources }) {
    // carry out logic for creating array, getting connections, etc...

    // returns Promise for that loader
    // We just have React manage the Loader instance, rather than passing around these params.
    return new ZarrLoader(connections, minZoom, imageWidth, imageHeight, tileSize, channels)
  }
  async getTile({ x, y, z }) {
    // e.g
    const tiles = []
    for (let i = 0; i < this.channels.length; i++) {
      const tile = this.connections[z].getRawChunk([0, y, x]);
      tiles.push(tile);
    }
    const resolved = await Promise.all(tiles);
    return resolved;
  }
}

That way in microscopy-base-layer we can use:

ZarrLoader.intilize({...props}).then(loader => this.setState({loader}));

All pyramids in one tiff or across many? Can we leave metadata in?

To take the first one off the list:

npm WARN @deck.gl/[email protected] requires a peer of @deck.gl/core@^8.0.0 but none is installed. You must install peer dependencies yourself.

Why does this happen? It seems to be in place:

grep version 'node_modules/@deck.gl/core/package.json'
  "version": "8.1.0-alpha.1"

And in any event, it seems to be working?

See visgl/deck.gl#4238. We will need to implement our own to get OpenSeaDragon-like performance.

• Load the demo with the default zarr.
• The image has a greenish cast, with blue dots
• Toggle off DAPI (blue).

I would expect the green areas to stay the same, but instead it goes to all black. If I lower the right end of the green slider to 3500 I start to get back to something like the original, but the color scale is more abrupt: The darker areas are darker than in the original, while the brights are brighter.

• Release
• Use in Vitessce

We are currently on an alpha branch and will need to upgrade DeckGL off that once it is no longer needed.