• Highlight selected object
• Object popups
• Hover tooltip

• converter: generate velocity and acceleration when converting kitti dataset (using XvizBuilder)
• loader: load the time series metrics and format data for visualizing
• viewer: visualize metrics with component from monochrome-ui

Streetscape must support v2 camera metadata and XVIZ binary streams with images to render in our camera control.

The current image support in streetscape is based on data blob sent over the websocket that is not packed within XVIZ binary. We must support the images from XVIZ binary and ensure that images are handled appropriately for rendering in the camera control.

The camera's will be include by default, but they are too large. We need defaults that are sensible and do not require any special parameters to make the application usable.

I had to use the following to "see" the scene.
yarn start --image-max-width 400 --disable-stream image_00,image_01,image_02 -d 2011_09_26/2011_09_26_drive_0005_sync

Also, the images obscure the UI, so it's not a very nice experience yet.

There seems to be an error when running npm install streetscape.gl, in particular when dealing with loaders.gl/gltf dependency
`
npm ERR! path /home/user/app/node_modules/@loaders.gl/gltf/bin/glbdump.js
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '/home/user/app/node_modules/@loaders.gl/gltf/bin/glbdump.js'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR! /home/user/.npm/_logs/2019-03-11T08_54_26_434Z-debug.log
`

Enabling the Declarative UI in the kitti-converter.js file causes an error in the xviz-viewer.

1. You may be adding a ref to a functional component
2. You may be adding a ref to a component that was not created inside a component's render method
3. You have multiple copies of React loaded
See https://fb.me/react-refs-must-have-owner for more information.
    at invariant (invariant.js:42)
    at coerceRef (react-dom.development.js:11887)
    at createChild (react-dom.development.js:12092)
    at reconcileChildrenArray (react-dom.development.js:12342)
    at reconcileChildFibers (react-dom.development.js:12656)
    at reconcileChildrenAtExpirationTime (react-dom.development.js:13016)
    at reconcileChildren (react-dom.development.js:13007)
    at updateHostComponent (react-dom.development.js:13341)
    at beginWork (react-dom.development.js:13829)
    at performUnitOfWork (react-dom.development.js:15864)
invariant @ invariant.js:42
coerceRef @ react-dom.development.js:11887
createChild @ react-dom.development.js:12092
reconcileChildrenArray @ react-dom.development.js:12342
reconcileChildFibers @ react-dom.development.js:12656
reconcileChildrenAtExpirationTime @ react-dom.development.js:13016
reconcileChildren @ react-dom.development.js:13007
updateHostComponent @ react-dom.development.js:13341
beginWork @ react-dom.development.js:13829
performUnitOfWork @ react-dom.development.js:15864
workLoop @ react-dom.development.js:15903
callCallback @ react-dom.development.js:100
invokeGuardedCallbackDev @ react-dom.development.js:138
invokeGuardedCallback @ react-dom.development.js:187
replayUnitOfWork @ react-dom.development.js:15311
renderRoot @ react-dom.development.js:15963
performWorkOnRoot @ react-dom.development.js:16561
performWork @ react-dom.development.js:16483
performSyncWork @ react-dom.development.js:16455
requestWork @ react-dom.development.js:16355
scheduleWork$1 @ react-dom.development.js:16219
enqueueSetState @ react-dom.development.js:11300
Component.setState @ react.development.js:270
(anonymous) @ connect.js:32
(anonymous) @ xviz-loader-interface.js:71
set @ xviz-loader-interface.js:71
(anonymous) @ xviz-stream-loader.js:257
onMessage @ parse-stream-message.js:34
Promise.then (async)
next @ worker-utils.js:136
(anonymous) @ worker-utils.js:138
Promise.then (async)
next @ worker-utils.js:138
process @ worker-utils.js:144
parseStreamMessage @ parse-stream-message.js:35
ws.onmessage @ xviz-stream-loader.js:187
23:24:09.538 index.js:92 Uncaught TypeError: this._stopPlayback is not a function
    at PlaybackControl.componentWillUnmount (index.js:92)
    at callComponentWillUnmountWithTimer (react-dom.development.js:14281)
    at HTMLUnknownElement.callCallback (react-dom.development.js:100)
    at Object.invokeGuardedCallbackDev (react-dom.development.js:138)
    at invokeGuardedCallback (react-dom.development.js:187)
    at safelyCallComponentWillUnmount (react-dom.development.js:14288)
    at commitUnmount (react-dom.development.js:14495)
    at commitNestedUnmounts (react-dom.development.js:14526)
    at unmountHostComponents (react-dom.development.js:14777)
    at commitDeletion (react-dom.development.js:14828)
componentWillUnmount @ index.js:92
callComponentWillUnmountWithTimer @ react-dom.development.js:14281
callCallback @ react-dom.development.js:100
invokeGuardedCallbackDev @ react-dom.development.js:138
invokeGuardedCallback @ react-dom.development.js:187
safelyCallComponentWillUnmount @ react-dom.development.js:14288
commitUnmount @ react-dom.development.js:14495
commitNestedUnmounts @ react-dom.development.js:14526
unmountHostComponents @ react-dom.development.js:14777
commitDeletion @ react-dom.development.js:14828
commitAllHostEffects @ react-dom.development.js:15417
callCallback @ react-dom.development.js:100
invokeGuardedCallbackDev @ react-dom.development.js:138
invokeGuardedCallback @ react-dom.development.js:187
commitRoot @ react-dom.development.js:15569
completeRoot @ react-dom.development.js:16619
performWorkOnRoot @ react-dom.development.js:16564
performWork @ react-dom.development.js:16483
performSyncWork @ react-dom.development.js:16455
requestWork @ react-dom.development.js:16355
scheduleWork$1 @ react-dom.development.js:16219
enqueueSetState @ react-dom.development.js:11300
Component.setState @ react.development.js:270
(anonymous) @ connect.js:32
(anonymous) @ xviz-loader-interface.js:71
set @ xviz-loader-interface.js:71
(anonymous) @ xviz-stream-loader.js:257
onMessage @ parse-stream-message.js:34
Promise.then (async)
next @ worker-utils.js:136
(anonymous) @ worker-utils.js:138
Promise.then (async)
next @ worker-utils.js:138
process @ worker-utils.js:144
parseStreamMessage @ parse-stream-message.js:35
ws.onmessage @ xviz-stream-loader.js:187
23:24:09.540 react-dom.development.js:14227 The above error occurred in the <svg> component:
    in svg
    in div
    in XYPlot
    in div
    in AutoSizer
    in SizeSensor
    in div
    in Chart
    in MetricChart
    in div
    in MetricCard
    in XvizMetricComponent
    in WrappedComponent
    in div
    in div
    in XvizContainer
    in div
    in XvizPanel
    in WrappedComponent (created by Example)
    in div (created by Example)
    in div (created by Example)
    in Example

Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://fb.me/react-error-boundaries to learn more about error boundaries.
logCapturedError @ react-dom.development.js:14227
logError @ react-dom.development.js:14266
update.callback @ react-dom.development.js:14919
callCallback @ react-dom.development.js:10879
commitUpdateQueue @ react-dom.development.js:10923
commitLifeCycles @ react-dom.development.js:14397
commitAllLifeCycles @ react-dom.development.js:15463
callCallback @ react-dom.development.js:100
invokeGuardedCallbackDev @ react-dom.development.js:138
invokeGuardedCallback @ react-dom.development.js:187
commitRoot @ react-dom.development.js:15604
completeRoot @ react-dom.development.js:16619
performWorkOnRoot @ react-dom.development.js:16564
performWork @ react-dom.development.js:16483
performSyncWork @ react-dom.development.js:16455
requestWork @ react-dom.development.js:16355
scheduleWork$1 @ react-dom.development.js:16219
enqueueSetState @ react-dom.development.js:11300
Component.setState @ react.development.js:270
(anonymous) @ connect.js:32
(anonymous) @ xviz-loader-interface.js:71
set @ xviz-loader-interface.js:71
(anonymous) @ xviz-stream-loader.js:257
onMessage @ parse-stream-message.js:34
Promise.then (async)
next @ worker-utils.js:136
(anonymous) @ worker-utils.js:138
Promise.then (async)
next @ worker-utils.js:138
process @ worker-utils.js:144
parseStreamMessage @ parse-stream-message.js:35
ws.onmessage @ xviz-stream-loader.js:187
23:24:09.541 react-dom.development.js:14227 The above error occurred in the <PlaybackControl> component:
    in PlaybackControl
    in PlaybackControl
    in WrappedComponent (created by Example)
    in div (created by Example)
    in div (created by Example)
    in Example

Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://fb.me/react-error-boundaries to learn more about error boundaries.
logCapturedError @ react-dom.development.js:14227
logError @ react-dom.development.js:14266
update.callback @ react-dom.development.js:14919
callCallback @ react-dom.development.js:10879
commitUpdateQueue @ react-dom.development.js:10912
commitLifeCycles @ react-dom.development.js:14397
commitAllLifeCycles @ react-dom.development.js:15463
callCallback @ react-dom.development.js:100
invokeGuardedCallbackDev @ react-dom.development.js:138
invokeGuardedCallback @ react-dom.development.js:187
commitRoot @ react-dom.development.js:15604
completeRoot @ react-dom.development.js:16619
performWorkOnRoot @ react-dom.development.js:16564
performWork @ react-dom.development.js:16483
performSyncWork @ react-dom.development.js:16455
requestWork @ react-dom.development.js:16355
scheduleWork$1 @ react-dom.development.js:16219
enqueueSetState @ react-dom.development.js:11300
Component.setState @ react.development.js:270
(anonymous) @ connect.js:32
(anonymous) @ xviz-loader-interface.js:71
set @ xviz-loader-interface.js:71
(anonymous) @ xviz-stream-loader.js:257
onMessage @ parse-stream-message.js:34
Promise.then (async)
next @ worker-utils.js:136
(anonymous) @ worker-utils.js:138
Promise.then (async)
next @ worker-utils.js:138
process @ worker-utils.js:144
parseStreamMessage @ parse-stream-message.js:35
ws.onmessage @ xviz-stream-loader.js:187
23:24:09.542 xviz-loader-interface.js:47 error Error: Element ref was specified as a string (series0) but no owner was set. This could happen for one of the following reasons:
1. You may be adding a ref to a functional component
2. You may be adding a ref to a component that was not created inside a component's render method
3. You have multiple copies of React loaded
See https://fb.me/react-refs-must-have-owner for more information.
    at invariant (invariant.js:42)
    at coerceRef (react-dom.development.js:11887)
    at createChild (react-dom.development.js:12092)
    at reconcileChildrenArray (react-dom.development.js:12342)
    at reconcileChildFibers (react-dom.development.js:12656)
    at reconcileChildrenAtExpirationTime (react-dom.development.js:13016)
    at reconcileChildren (react-dom.development.js:13007)
    at updateHostComponent (react-dom.development.js:13341)
    at beginWork (react-dom.development.js:13829)
    at performUnitOfWork (react-dom.development.js:15864)
emit @ xviz-loader-interface.js:47
(anonymous) @ xviz-stream-loader.js:272
Promise.catch (async)
next @ worker-utils.js:134
(anonymous) @ worker-utils.js:138
Promise.then (async)
next @ worker-utils.js:138
process @ worker-utils.js:144
parseStreamMessage @ parse-stream-message.js:35
ws.onmessage @ xviz-stream-loader.js:187
23:24:09.558 warning.js:33 Warning: Can't call setState (or forceUpdate) on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in the componentWillUnmount method.
    in WrappedComponent (created by Example)
    in div (created by Example)
    in div (created by Example)
    in Example
printWarning @ warning.js:33
warning @ warning.js:57
warnAboutUpdateOnUnmounted @ react-dom.development.js:15206
scheduleWork$1 @ react-dom.development.js:16227
enqueueSetState @ react-dom.development.js:11300
Component.setState @ react.development.js:270
(anonymous) @ connect.js:32
(anonymous) @ xviz-loader-interface.js:71
set @ xviz-loader-interface.js:71
(anonymous) @ xviz-stream-loader.js:257
onMessage @ parse-stream-message.js:34
Promise.then (async)
next @ worker-utils.js:136
(anonymous) @ worker-utils.js:138
Promise.then (async)
next @ worker-utils.js:138
process @ worker-utils.js:144
parseStreamMessage @ parse-stream-message.js:35
ws.onmessage @ xviz-stream-loader.js:187```

• Remove .eslintrc overrides (mainly camelcase)
• fix linter setup
• update code to pass linter and prettier.

Also see aurora-opensource/xviz#54

Bug: The SignLayer is displaying black over the front of faces at some angles and not covering the back of the sign, as seen below:

Enhancement: It would also be nice to be able to toggle this black overlay in the layer if that's possible.

Hi,

What would be the easiest way of adding your own mesh? I have my own OBJ file, is it possible to load it directly or is there a utility script that converts it to the format found in sedan.js?

Right now if you do not supply a /vehicle_pose poses stream you don't get any data on screen and get no errors. What should happen is that if there is no vehicle pose:

• We assume the vehicle position is identity
• We disable the mapbox layer (since there is no lat/long informaton)
• No vehicle mesh is drawn on screen

Right now for a circle primitive we have:

• radius style
• radius_m property

The builder API creates objects with those fields streetscape.gl only pays attentions to the style. What we want is the following priority, from lowest to highest:

• primitive radius_m property
• metadata radius style property
• inline radius style property

// metadata
xvizMetadataBuilder
  .stream('/vehicle_pose')
  .category('pose')
  
  .stream('/object/tracking_point')
  .category('primitive')
  .type('circle')
 
  .startTime(1000)
  .endTime(1010);

// frame
const timestamp = 1001.0;

xvizBuilder.pose('/vehicle_pose')
  .timestamp(timestamp)
  .mapOrigin(-122.4, 37.8, 0)
  .orientation(0, 0, 0);
  
xvizBuilder.primitive('/object/tracking_point')
  .circle([10, 10, 0])
  .style({
    fill_color: '#fb0'
  })
  .id('object-1')

  .circle([-5, 20, 0], 5)
  .style({
    fill_color: '#08f'
  })
  .id('object-2');

Feedback from @globetro:

This is in retrospect what I would've found helpful:

• Why there are 2 packages (xviz and streetscape), and the purpose of each one. #67
• High level overview of the steps to get a fully functioning system: 1. Converters to create the frames (and how to use xvizbuilder to do that), 2. Server that serves the frames, 3. Streetscape components for consuming the frames #64

The following should probably be documented in XVIZ?

• Was confusing that there are certain "special" topics that need to be called out in XVIZ_CONFIG, such as PRIMARY_POSE_STREAM, while all other topics are just rendered based on their type, and that the naming of those topics have no inherent meaning. In fact, is this even needed anymore? Can't the PRIMARY_POSE_STREAM be determined by finding the topic with the "pose" type? That would make things more consistent, where rendering of any stream is based on topic type.
• Document the fields that are expected for the primary pose stream
  (I haven't kept up with the documentation recently, so maybe some/all of these have been addressed).

We need a CodePen like environment that accounts for all the data we need to demonstrate minimal XVIZ examples.

This includes

• XVIZBuilder
• XVIZMetadataBuilder
• Declarative UI
• Support for predefined examples

On this page, I see a link to
https://github.com/uber/streetscape.gl/tree/master/examples/get-started/log-from-file.js
However, this link is broken.

Thanks for fixing the other link!

Support direct link to

• datasets
• timestamp
• view mode
• panel

Support ROSBAG extraction

Questions remain as to what can be extracted by default w/o an configuration to make this turnkey.
Research must be done.

Hi,

I have my own robotics project going on which streams data over UDP. I wanted to implement some kind of middleware that converts these streams to XVIZ streams. I am having trouble figuring out the best approach for doing this.

On the one end, we have the examples in XVIZ, but they all seem to load files or generate frames (scenarios) beforehand and serving them in a rather complex way with xviz-serve-data.js. Not really what I am looking for.

Looking at streetscap.gl, I guess this starter-kit section mentions what I would like to do, but the documentation is a bit limited. The client-side would as indicated be as easy as yarn start-streaming. The instructions on how to start a local XVIZ stream server just link back to the get-started docs which is not very helpful.

I really love the play ground example and is kind of what I want to do, but then I would like to do the JSON part in a server that builds and sends some metadata and then builds new frames and sends them over to the streaming client based on some internal events. Am I missing something here and is it already there?

Thanks again, love the work you guys are doing.

• Review component and props naming convention
• Support advanced use cases with experimental export
• Export and documentation alignment

@KevinGrandon Any chance you could help setup buildkit for this repo?

I want to use IP address to access through the network, not using localhost, how to set up

Implement a server that include the dynamic protocol to support:

• random seek
• multiple data sets
• dynamic stream filtering

This can still be over pre-extracted data

On this page, a link is given to:

https://github.com/uber/streetscape.gl/tree/master/examples/get-started/log-from-stream.js

I'm getting an error when I try to access it

• convert dataset: Udacity dataset is in rosbag format
  • rosbag helper functions to load bag for selected topics
  • use XvizBuilder to generate frame from bag data
• visualize the in XvizViewer

XVIZ viewer needs to support buffering and dropping data sent from the server. The server also needs to support more intelligent sending of data and some basic protocol commands.

Currently we retain all data we are sent which will make the application unresponsive as data size approaches the browsers limit or various visuals have to process increasing amounts of data.

Requirements are:

1. Implement dropping data outside of some range. Could be time based or size based from current playhead
2. Application should request a new range if it doesn't have data for a time (random access seeking)
3. Server implementation should support requires for additional data

It is unclear if we should have the simple server send all data, assuming a continuous stream. Or if we should require some limit explicitly. And leave the processing to the client application to request more as it approaches it's buffering end.

I'm leaning toward that latter

XVIZ change aurora-opensource/xviz#315 is adding data to satisfy the UI and control in streetscape.gl.

It would be nice if we could simplify this as the log_info is not technically part of the spec.

The discussion is around various modes of operation

• log mode, where we have known start/end and we assume they fit within the internal buffer for XVIZ data.
• live mode, we do not have start/end and where data is streaming in. We have a few modes for handling the buffering of XVIZ, but today we have to flag it so the PlaybackController and XVIZStreamBuffer play nicely together
• random access, similar to log mode, but the idea is there is more data than we can fit in so we need to manage buffering and making requests to load sections of the log. This implies a more complex interaction.

There are two issues:

• The demo app needs to close the active log when switching to another
• XVIZFileLoader does not stop loading/parsing after close

• prop to control stream visibility
• prop to override XVIZ styles
• add user-defined deck.gl layers

Current xviz-viewer does not pause playback when the playhead exceeds the current loaded data.

We need to:

1. stop the playback from advancing if we do not have any data
2. resume playback when we have sufficient data

There is an open question to answer which is what criteria we use to answer #2

Reproduce

In the xviz repo run:

./scripts/run-kitti-example.sh

Then in streetscape.gl repo:

cd examples/get-started
yarn start-streaming-local

Result

Expected.

Context

• in my project, I have a point cloud data (collection of points with coordinates and colors) and a JSON file describing the trajectory of vehicles (coordinates, orientation associated with timestamp).
• All the vehicles are equally important, there is no primary one. The coordinates are in absolute values.

Requirement

• I want to be able to dynamically select a vehicle (by mouse) to switch the Camera Focus so that I can have 3 views: Top-view, perspective and driver view of that vehicle.

The current gap

• I learned that XVIZ data (at least in the KITTI sample data) are organized around a primary vehicle (in the stream /vehicle_pose) that stream data from sensors can have a pose offset relative to that vehicle pose.

Wishlist

• data level (XVIZ): allows to represent coordinates in the absolute value, not offset relative to any primary vehicle.
• code level (streetscape.gl)
  • provide options to dynamically select a vehicle to switch the Camera focus to it.
  • allow to use the absolute value coordinates

Checklist / roadmap for tasks necessary before public launch

Binary protocol support:

• Packed point cloud format
• PLY point cloud
• Images

Frontend components:

• Full declarative UI support (React components)
• Add metrics panel for acceleration / velocity to KITTI demo

Demo - end to end concise example:

• Data transformation
• Stream server
• Client application

XVIZ:

• Builder module
• Load from static files for public demo
• Server example handles seeking
• Support limited stream buffer
• Each stream has it's own coordinate system

Documentation:

• Docs and specs fully completed

Examples:

• Examples support multiple datasets, not just KITTI
• Example of how to use underlying deck.gl with Streetscape so users don't feel they have to choose between the two

Conversion utilities:

• ROSbag converter example

There are event handlers for when an object is selected but we also ones for just anywhere in the 3D world. This will allow users to compose arbitrarily complex applications with those events and a custom deck.gl layer.

Trying to play around xviz but could not find the client folder as indicated in the 'xviz/docs/getting-started/running-the-example.md (Demo Start Application)'?

It is desirable to be able to select a specific stream and control playback not based on time but on the specific datum directly, stepping over sample by sample.

XvizViewer Should be a generic app.

Need move kitti specific code (mainly the XvizConfig file) out of XvizViewer

These are screen shots from the kitti example. The yellow tracklet is obscures the lidar data. Once the object is no longer tracked we can see the underlying lidar data is there.

• Separate object streams to align with original source
• Support higher framerate