uber / manifold Goto Github PK

Summary

After following the code snippet, how to locally start the example, an error is thrown and the website remains blank.

Expected Behavior

Example website pops up.

Current Behavior

Upon startup an error message is thrown

`$ npm run start

@ start /Users/jh186076/Documents/03_Privat/manifold/examples/manifold
webpack-dev-server --progress --hot --open

10% building 1/1 modules 0 activeℹ ｢wds｣: Project is running at http://localhost:8080/
ℹ ｢wds｣: webpack output is served from /
ℹ ｢wds｣: Content not from webpack is served from /Users/jh186076/Documents/03_Privat/manifold/examples/manifold
40% building 15/16 modules 1 active /Users/jh186076/Documents/03_Privat/manifold/examples/manifold/node_modules/events/events.jsℹ ｢wdm｣: wait until bundle finished: /
✖ ｢wdm｣: Hash: d8e9b3efe73932330b3a
Version: webpack 4.41.2
Time: 1141ms
Built at: 01/13/2020 9:47:44 AM
Asset Size Chunks Chunk Names
app.js 377 KiB app [emitted] app
app.js.map 427 KiB app [emitted] [dev] app
index.html 184 bytes [emitted]
Entrypoint app = app.js app.js.map
[0] multi (webpack)-dev-server/client?http://localhost:8080 (webpack)/hot/dev-server.js ./src/main.js 52 bytes {app} [built]
[./node_modules/strip-ansi/index.js] 161 bytes {app} [built]
[./node_modules/webpack-dev-server/client/index.js?http://localhost:8080] (webpack)-dev-server/client?http://localhost:8080 4.29 KiB {app} [built]
[./node_modules/webpack-dev-server/client/overlay.js] (webpack)-dev-server/client/overlay.js 3.51 KiB {app} [built]
[./node_modules/webpack-dev-server/client/socket.js] (webpack)-dev-server/client/socket.js 1.53 KiB {app} [built]
[./node_modules/webpack-dev-server/client/utils/createSocketUrl.js] (webpack)-dev-server/client/utils/createSocketUrl.js 2.89 KiB {app} [built]
[./node_modules/webpack-dev-server/client/utils/log.js] (webpack)-dev-server/client/utils/log.js 964 bytes {app} [built]
[./node_modules/webpack-dev-server/client/utils/reloadApp.js] (webpack)-dev-server/client/utils/reloadApp.js 1.59 KiB {app} [built]
[./node_modules/webpack-dev-server/client/utils/sendMessage.js] (webpack)-dev-server/client/utils/sendMessage.js 402 bytes {app} [built]
[./node_modules/webpack/hot sync ^./log$] (webpack)/hot sync nonrecursive ^./log$ 170 bytes {app} [built]
[./node_modules/webpack/hot/dev-server.js] (webpack)/hot/dev-server.js 1.59 KiB {app} [built]
[./node_modules/webpack/hot/emitter.js] (webpack)/hot/emitter.js 75 bytes {app} [built]
[./node_modules/webpack/hot/log-apply-result.js] (webpack)/hot/log-apply-result.js 1.27 KiB {app} [built]
[./node_modules/webpack/hot/log.js] (webpack)/hot/log.js 1.34 KiB {app} [built]
[./src/main.js] 1.85 KiB {app} [built] [failed] [1 error]
+ 20 hidden modules

WARNING in EnvironmentPlugin - MAPBOX_ACCESS_TOKEN environment variable is undefined.

You can pass an object with default values to suppress this warning.
See https://webpack.js.org/plugins/environment-plugin for example.

ERROR in ./src/main.js
Module build failed (from ./node_modules/babel-loader/lib/index.js):
ReferenceError: [BABEL] /Users/jh186076/Documents/03_Privat/manifold/examples/manifold/src/main.js: Unknown option: base.rootMode. Check out http://babeljs.io/docs/usage/options/ for more information about options.

A common cause of this error is the presence of a configuration options object without the corresponding preset name. Example:

Invalid:
{ presets: [{option: value}] }
Valid:
{ presets: [['presetName', {option: value}]] }

For more detailed information on preset configuration, please see https://babeljs.io/docs/en/plugins#pluginpresets-options.
at Logger.error (/Users/jh186076/Documents/03_Privat/manifold/examples/manifold/node_modules/babel-core/lib/transformation/file/logger.js:41:11)
at OptionManager.mergeOptions (/Users/jh186076/Documents/03_Privat/manifold/examples/manifold/node_modules/babel-core/lib/transformation/file/options/option-manager.js:226:20)
at OptionManager.init (/Users/jh186076/Documents/03_Privat/manifold/examples/manifold/node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)
at File.initOptions (/Users/jh186076/Documents/03_Privat/manifold/examples/manifold/node_modules/babel-core/lib/transformation/file/index.js:212:65)
at new File (/Users/jh186076/Documents/03_Privat/manifold/examples/manifold/node_modules/babel-core/lib/transformation/file/index.js:135:24)
at Pipeline.transform (/Users/jh186076/Documents/03_Privat/manifold/examples/manifold/node_modules/babel-core/lib/transformation/pipeline.js:46:16)
at transpile (/Users/jh186076/Documents/03_Privat/manifold/examples/manifold/node_modules/babel-loader/lib/index.js:50:20)
at Object.module.exports (/Users/jh186076/Documents/03_Privat/manifold/examples/manifold/node_modules/babel-loader/lib/index.js:173:20)
Child html-webpack-plugin for "index.html":
1 asset
Entrypoint undefined = index.html
[./node_modules/html-webpack-plugin/lib/loader.js!./node_modules/html-webpack-plugin/default_index.ejs] 376 bytes {0} [built]
[./node_modules/lodash/lodash.js] 528 KiB {0} [built]
[./node_modules/webpack/buildin/global.js] (webpack)/buildin/global.js 472 bytes {0} [built]
[./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 497 bytes {0} [built]
ℹ ｢wdm｣: Failed to compile.
`

Context and Environment

npm version 6.13.4
node v12.14.1

Steps to Reproduce

yarn
cd examples/manifold
yarn
npm run start

In Jupyter binding, when there are multiple Manifold instance running inside different cells, change of one Manifold instance in one cell will change the Manifold instances in the other cells.

Note: this behavior is not related to ipywidget as the bug only occurs to cells with Manifold. This is likely related to the reducer settings of the Manifold.

Expected Behavior:

Current Behavior when the first cell was run after the second:

Addressed by #45

Is it possible to get a docker image with manifold?

Summary

not very familiar with yarn and getting errors on build. Docker will help a lot!

Expected Behavior

Docker run uber/manifold and voila :-)

Possible Solution

alternatively, I can build and share one, just having temporary troubles with yarn install 😬

Taking inspiration from deck.gl, we need to have some major release branches, e.g.

This is necessary for us to maintain those releases. Because we also have a major Jupyter binding, we can have two lines releases for the major packages and the Python libraries.

release-1.0.0 
...
release-jupyter-0.1.0
...

Patches to these releases use the minor version number, e.g.

release-1.0.1
release-jupyter-0.1.1

Create templates for issues and PRs.

Type analyzer used in Kepler is creating performance bottleneck, especially for large amount of features:

I am running Manifold from Jupyter notebook on Chrome browser.

Clicking on "Export Segmentation" button in the output widget outputs a list with the indexes of the input data. Is is possible to obtain the segment number. For example, Segment_1 or Segment_2 along with the index.

The license compliance tool seems to be not working, which prevents successful builds. It seems to have something to do with the connection to the FOSSA service.

• add readme to npm homepage https://www.npmjs.com/package/@mlvis/manifold
• add publishing script to Travis CI
• auto bump version in the main package.json (currently ocular-publish only bumps versions in modules package.json
• install storybook for development purpose
• properly structure published module (below is the current structure, which causes import Manifold from '@mlvis/manifold' to not work #27
  
• add depcheck
• use gh-pages module to deploy website

Summary

base-ui requires a node engine version greater than v10. Thus the node engine requirement should be updated.

There should be a way to calculate the performance-based decision boundary that maximizes the cumulative distribution difference in features.

Hello everybody, i just tried to get the demo app running, but sadly can't get all the packages installed.

When running the yarn command in the examples/manifold folder, i receive the following error message:
An unexpected error occurred: "https://unpm.uberinternal.com/react-is/-/react-is-16.9.0.tgz: Request failed \"401 Unauthorized\"".

EDIT: When installing react-is externally, the next same error is for gud. It seems, like yarn tries to find the packages in uberinternal, no matter which.

the current website publishing script is hacky. Maybe we can leverage `gh-pages' to make it simpler.

ocular-build builds three bundles by default: es5/, es6/ and esm/. This structure disables importing subfolders of each module. Currently, the es5/ and es6/ bundles are not used at all, so it may be better to configure ocular to only build the esm/ bundle. This way, tree-shaking is enabled, so is the subfolder import, both of them have merits on reducing the bundle size, not mentioning the saved build time by only building one bundle. This can be done by either modifying ocular or by simply adding some build script in each package.json to overwrite the default ocular build behavior. @ibgreen may have an idea on how to do it.

pull request (uber-web/ocular#251)
pull request (#56)

It would be useful to export the feature attribution view data to record the feature differences discovered based on some performance slicing criteria, for future usage

Neither Manifold running locally nor http://manifold.mlvis.io/ returns any result after uploading .csv feature, prediction and ground truths data sets.

I have tried following the instructions on how to format .csv files, but it does not seem to work for me.

The csv files I have tried it with are attached (renamed extension to .txt, to be able to attach to issue).

What am I missing?

truth.txt
prediction.txt
features.txt

The doc website needs to be public now since our codebase has been migrated to the OSS repo. It is also necessary to move all the website code from the internal repo to the OSS one so it can be maintained here.

Clicking on the close button has no effect.

Summary

I have a hard time contributing to big projects I am a student who uses a chromebook all day which just doesn't have the power I need. With Gitpod people with lower end computers can contribute easily without waiting for builds or having to deal with setting up a Development Environment to contribute.

Expected Behavior

1. User clicks open in gitpod button from README
2. Start Coding

Current Behavior

1. Read build instructions
2. Install node and yarn
3. Debug any issues you find doing that
4. Clone Repo
5. Install dependencies and wait for that to finish
6. Code

Context and Environment

• Module & version: [e.g. @mlvis/stacked-calendar, v1.0.0]

Steps to Reproduce

Possible Solution

Gitpod
#100

Summary

In app where styletron is already used, the styletron engine within manifold component will conflict with the engine outside.

Current Behavior

E.g. if the app uses server side rendering, it will conflict with the engine within manifold which uses client side rendering

ReferenceError: document is not defined
    at new StyletronClient (/Users/agreco/dev/michelangelo-web-fresh/node_modules/styletron-engine-atomic/dist/index.js:660:7)
    at Object. (/Users/agreco/dev/michelangelo-web-fresh/node_modules/@mlvis/src/components/manifold.js:35:16)

Possible Solution

Remove styletron engine from within manifold; ask users to use styletron provider separately.

Summary

Several users have reported (#103, #108) that no results were shown after uploading CSV files, possibly due to format/column mismatch.

Expected Behavior

Users should be able to get detailed warning/error messages if data validation has failed.

Current Behavior

Nothing being rendered and no actionable message is surfaced for the cause of the format mismatch.

Possible Solution

1. As an MVP, output (e.g. console.log) if there are any expected columns missing.
2. A wizard to guide the data column mapping and validation.

if there are null or undefined values in input data, current behavior is to remove the feature from the list. Desired behavior is:

• create a NA category for categorical features
• fill values using median (or other desired values) for numerical features

Summary

Create a visualization prototype sketched in this doc

#88

Currently, within Manifold package we are using styled-components to do custom styling; it becomes complicated for advanced UI widgets (e.g. select with searching functionalities) and hinders development speed.

Pros:

• Using standard UI widget will shorten the development time of UI part and ensures we focus on visualization part
• Helps the project to scale in the long run -- developers who are familiar with Baseui can easily jump in
• Aligns with other Uber UI product

Cons:

• Need to refactor in the short term
• Not good for package size -- Baseui is using styletron as engine; Kepler.gl is using styled-components, and if we switch, Manifold will be depending on both (it seems not hard for Baseui to support styled-components, but it's depriotized https://github.com/uber-web/baseui/issues/114)

Middle-ground solution: we could keep styled-components for e.g. styling divs and only use Baseui for UI widgets for now, so that the refactoring work will not involve refactoring styled-component css-like strings to styletron objects:

// styled-components
const Centered = styled.div`
  display: flex;
  justify-content: center;
`;
// styletron
const Centered = styled('div', {
  display: 'flex',
  justifyContent: 'center',
});

Steps to Reproduce

• remove "@score:model_1" from "Cluster by" selection box
• reduce "Number of clusters" from 4 to 3
  

Expected Behavior

Expect to see the clear-cut separation of green curves wrt their positions along X axis in the "Performance Comparison" chart

Current Behavior

Green curves are overlapping with each other wrt their positions along X axis in the "Performance Comparison" chart

Context and Environment

• Module & version: [e.g. @mlvis/manifold, v1.1.1]

Possible Solution

Summary

I used Jupyter notebook and did the Manifold example without any problem. But when I try with the regression model, it doesn't show any result. I couldn't find any regression model example of this.

Expected Behavior

Works just like classification model

Current Behavior

Manifold doesn't return result for Regression model. even no error raised

Possible Solution

Any regression model example

Migrate the current Jupyter Binding from the internal repo to the public one. The bindings code should stay together with the modules to minimize maintenance overhead. This migration should also include stacked-calendar and graph-builder module because the current version of Jupyter binding contains these two modules.

Summary / Current Behavior

The publish script is throwing ERR instead of bypassing private modules as expected.

npm ERR! This package has been marked as private
npm ERR! Remove the 'private' field from the package.json to publish it.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/gnavvy/.npm/_logs/2020-01-10T19_31_04_771Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] lerna:publish: `lerna exec cp package.json dist && lerna exec npm publish dist`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] lerna:publish script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/gnavvy/.npm/_logs/2020-01-10T19_31_04_804Z-debug.log
error Command failed with exit code 1.

Expected Behavior

Modules marked as private should be skipped during publishing.

Context and Environment

We currently have a few and will keep adding experimental modules into this repo. It is desired to exclude them from being published to NPM in early development. One viable solution is to use the "private": true flag in the module's package.json (example) to bypass the publish step, but this breaks npm publish.

Steps to Reproduce

To reproduce, run the script in the following order

npm run build
npm run publish

Possible Solution

Explicitly adding the --no-private flag after lerna exec during publish bypasses npm publish inside the private modules.

problem

Currently the following logic is hardcoded in Manifold:

• when choosing auto segmentation, user can choose comparison metric, but when choosing manual segmentation, they can't. The value will be whatever is set when they're in auto mode

Because of this, the following use cases cannot be easily implemented:

Use cases

Real-world usage this improvement will support include (from customer interview):

1. Highlight direction of errors
2. Identify really badly-performing data points (outliers) for inspection

Solutions

Use case 1 can be implemented by setting performance metrics to indicate over/under prediction (instead of absolute prediction error), and allow users to manually segment data based on this metric column (i.e. set segmentation threshold to 0).

Use case 2 can be implemented by allowing users to manually segment data based on this metric column (i.e. set segmentation threshold to some really high value so that only a few datapoints are in group 0).

To enable these, we need to make the following fields in state independent knobs (instead of hard-coding the value of one field base on the value of another), and then hook each of them to UI controls:

• isManualSegmentation: whether to apply manual (filter-based) or automatic (kmeans) data slicing
• baseCols: use which columns to slice (either through creating filters for these columns, or through inputting them to kmeans clustering)
• nClusters: number of clusters to use in automatic slicing (only applicable to automatic slicing)
• segmentFilters: filter logic corresponding to data segment (only applicable to manual slicing)
• segmentGroups: which segments to group together for comparing against each other

Milestone

To validate the success of the change, we will evaluate how the 2 user tasks in the "Use cases" section can be achieved.

Appendix

Current implementation involves converting tensor data between array and tf.Tensor format through synchronized data reading, which blocks the UI thread until the values are ready, which can cause performance issues. Migrate to async instead.

Some DS use manifold mainly as a feature segmentation and comparison tool. In that case, they don't necessarily need a ground-truth data and they demand the ability to use Manifold without having to specify the yTrue variable due to the difficulty of obtaining these ground truth data. Considering this use case, it is useful to enable an option to use Manifold without having to specify the yTrue variable, even though this option will make some of the model evaluation functionalities unavailable.

Add Google analytics to the website to track our user activity.

Summary

To understand the user activity on our website, we will need to add GA to our website and observe the user usage and activities.

Expected Behavior

We should be able to get the google analytics result.

Current Behavior

No GA available.

Possible Solution

Inject the GA code in the header of the website.

Most of Manifold related tasks (besides those related to the internal platforms) should be migrated to this github repo. This will ease the tracking of those tasks.

Use tensorflow.js instead

Summary

Demo site http://manifold.mlvis.io/ was down after adding google analytics (#65)

Currently we have 2 ways to show numerical values over spatially distributed data:

1. Overlapping heatmaps, used for displaying two data slices on a map:

2. Hexagon binned heatmaps, used for displaying spatially aggregated 1-Dimensional data (e.g. values of one feature):

The drawbacks are:

1. Overlapping heapmaps might occlude information
2. It's hard to show multiple dimensions at the same time

We need to invent new visual encodings to fix it.

Right now only unit tests for each functions are in place. It would be ideal to add integration tests considering the signature of individual functions might change during the development process.

I've installed the mlvis extension and followed the steps installing the nbextension and then enabling it.

When following the manifold example notebook (https://github.com/uber/manifold/blob/master/bindings/jupyter/notebooks/manifold.ipynb) and running the first cell I get the output:

Manifold(props='{"data": {"x": [{"feature_0": 318, "feature_1": "A"}, {"feature_0": 213, "feature_1": "B"}, {"…

And no visual output. Is there a step that I need to change to get the notebook visual output working?

I've even tried disabling the other extensions to see whether it would work and no change.

I'm running jupyter version 1.0.

If input data yTrue is a column of integers indicating classes, there will be this error

is it possible to add other metrics (e.g. I need MAPE)?
I obviously can pre-generate those, but it would be nice to have those as part of the app

Manifold repo needs to be converted to monorepo structure and CI tools have to be updated to facilitate development and team collaborations. The goal is to migrate the codes from the internal repo to this github repo and slowly deprecate the internal repo.

The Jupyter binding should enable dynamic loading of React component. This will allow the users to install custom React components into the mlvis library. This will also help to reduce the amount of boilerplate in the code.

A planned implementation would leverage a config file called "jrequirements.json" which is shared by both the Python and JS code. This file will specify the Python wrapper name and the corresponding JS module and widget customizations, along with other functionalities. These specifications will be used to dynamically generate the corresponding Python and JS code.

Dynamically generate widget frontend (#47)
Mockup ext code for JS component widget extension (#48)
Correctly handle jrequirements.json in Python (#49)

When there is a value concentration in numerical feature values (due to filling null values etc), the distribution difference will be overshadowed by the concentration effect.

Possible ways to solve the issue:

• enable log scales for charts
• enable displaying the feature even if some feature values are null

Summary

When the app is using react-redux>=6, manifold has issue finding the store instance.

Expected Behavior

Manifold should work when the embedded app is using react-redux@6 or above

I am trying to use Manifold in jupyter notebook. My dataset consists of features, ground truth and predictions for 1 model. CSV files are attached. I get the following error -

yTrue must be a list/ndarray.

I am able to get the results with the demo app.
TestData_Results.zip

ocular-build seems to only work for modules in the immediate subfolder, e.g. modules/ in the current monorepo. However, it doesn't seem to build modules within deeper level subfolders, e.g. bindings/jupyter-modules, as shown in this PR (#25). Currently, modules within jupyter-modules seem to only use lerna run build instead. This behavior introduces some inconsistencies between modules inside the jupyter-modules/ folder and the modules inside the modules/ folder. Either ocular has to be changed to allow building modules within deeper level subfolders or jupyter-modules/ have to be moved to the root level to accommodate the current ocular-build behavior. @ibgreen may have an idea on which is a better way to do this.

Pull Request (uber-web/ocular#261)

This repo currently references some data files from S3 in some of its documentation/examples. Although they are harmless at the moment, It is best to move these data files to "https://github.com/uber-common/deck.gl-data" for security compliance.