mozilla / webxr-polyfill Goto Github PK
View Code? Open in Web Editor NEWA polyfill and example code for building augmented reality (AR) and virtual reality (VR) applications using WebXR.
License: Mozilla Public License 2.0
A polyfill and example code for building augmented reality (AR) and virtual reality (VR) applications using WebXR.
License: Mozilla Public License 2.0
We're receiving incorrect transform information during the ARKit update callbacks, so for now we're not updating anchors after they have been created.
The last code submission done by the exalt team is in the exalt branch.
We need to integrate the code, and make sure the ios app is updated at the same time. See mozilla-mobile/webxr-ios#38
There's an undeclared variable which break apps from working here:
https://github.com/mozilla/webxr-polyfill/blob/master/polyfill/XRPolyfill.js#L92
If a map has a floor anchor, it won't get used properly. If the user reset tracking, or the floor anchor gets deleted, things break.
Now that the CG has a plan for WebXR (including using the XR namespace) we need to make our polyfill as close as possible to the first version train (basically VR support) and separate out the bits that are experimental and for the second version train (the AR bits).
For v1 branch:
For vNext branch:
The examples currently don't show any indication if features like tracking (ARKit, ARCore, orientation events, etc) aren't supported.
Show a message that the current browser isn't supported and information on how to get a supported browser.
XRExampleBase is good for the examples, but we need something for devs to use in production code.
The new class should contain the boilerplate like XRExampleBase, but also:
I posted this issue as a question on Stack Overflow as I was unsure of whether it was a bug in this polyfill, three.xr.js
or aframe-xr
:
However, the recent pointer with SITTING_EYE_HEIGHT
and the fact that changing this constant to 0 goes some way to fix the raycasting issue has led me to believe that this is an issue that should be fixed in this polyfill.
What is the rationale for having a fixed eye height, as was introduced (or refactored) in this commit: 2a6b4e0
Why would the raycast direction still be off on the x-axis when setting SITTING_EYE_HEIGHT
to 0?
In ARKit/ARCore, new anchors are created for planes. IF/when image/face tracking is added, new anchors are created when images/faces are detected. Need a way for programmers to be notified.
Probably some on the session.
Add support for WebAROnARKit: https://github.com/google-ar/WebARonARKit
The examples currently default to using the first XRDisplay that matches the session initialization parameters. This will often be the wrong choice when the device has more than one display, for example a flat display and a VR HMD display.
Implement an example UI widget that allows the apps to offer up the available displays and, during a live XRSession, switch to a new display.
Add -webkit-tap-highlight-color: rgba(0, 0, 0, 0) to the session layer's canvas to prevent the default webkit behavior of flashing grey on taps.
Currently we don't pass the right flags so that when the user touches the eye icon they get access to the photo and video recording UI.
Pass the correct initialization flags to ARKit to turn on the recording UI.
The iOS ARKit platform wrapper, in ARKitWrapper.js, has no tests that check the iOS <-> JS bridge for creating and tracking anchors and receiving pose information.
Create a test suite that automates testing of ARKitWrapper.
there should be pictures in the overview page
The polyfill XRAnchors are currently stubbed out and just keep track of XRAnchors without linking them to ARKit anchors or providing hit testing in any way.
Update XRPresentationFrame.findAnchor and .addAnchor to use ARKit if available.
The camera video and 3D scene are not properly aligned.
The camera video is not properly resized and positioned to fill a handset's screen.
The app needs to provide more info (mozilla-mobile/webxr-ios#47), and we need to show whatever we can (not just "Session Failed to Initialize")
Right now, none of the polyfill classes fire the events defined in the API draft document.
There seems to be a lot of complexity created by having the XRAnchor
creation/finding methods inside XRPresentationFrame
.
For example, in the example code, we have to keep track of whether or not we already requested the floor anchor, since we can only get the floor anchor via the frame instance, and thus inside the frame callback:
// If we haven't already, request the floor anchor offset
if(this.requestedFloor === false){
this.requestedFloor = true
frame.findFloorAnchor('first-floor-anchor').then(anchorOffset => {
if(anchorOffset === null){
console.error('could not find the floor anchor')
return
}
this.addAnchoredNode(anchorOffset, this.floorGroup)
}).catch(err => {
console.error('error finding the floor anchor', err)
})
}
More so, this logic is incorrect, since it does not handle the case when the reality is changed, which would cause all the XRAnchors to become invalid (right?).
Can we just expose the various Anchor-related properties and methods (anchors
, getAnchor
, findAnchor
, findFloorAnchor
, etc.) in the Reality
class so that these things don't have to be called within a frame callback that is executed repeatedly, thus eliminating the need for things like the requestedFloor
flag above? For example, the anchor can easily be requested every time the Reality changes, which is the proper way to do it (assuming all anchors are invalid after a reality changes):
session.addEventListener('realitychanged', () => {
session.reality.findFloorAnchor('first-floor-anchor').then(anchorOffset => {
if(anchorOffset === null){
console.error('could not find the floor anchor')
return
}
this.addAnchoredNode(anchorOffset, this.floorGroup)
}).catch(err => {
console.error('error finding the floor anchor', err)
})
})
More so, we see this pattern of delaying the creation of anchors in the anchor sample:
addAnchoredModel(sceneGraphNode, x, y, z){
// Save this info for use during the next render frame
this.anchorsToAdd.push({
node: sceneGraphNode,
x: x, y: y, z: z
})
}
// Called once per frame
updateScene(frame){
const headCoordinateSystem = frame.getCoordinateSystem(XRCoordinateSystem.HEAD_MODEL)
// Create anchors and start tracking them
for(let anchorToAdd of this.anchorsToAdd){
// Create the anchor and tell the base class to update the node with its position
const anchorUID = frame.addAnchor(headCoordinateSystem, [anchorToAdd.x, anchorToAdd.y, anchorToAdd.z])
this.addAnchoredNode(new XRAnchorOffset(anchorUID), anchorToAdd.node)
}
this.anchorsToAdd = []
}
This can be simplified to:
addAnchoredModel(sceneGraphNode, x, y, z){
const anchorUID = session.reality.addAnchor(headCoordinateSystem, [anchorToAdd.x, anchorToAdd.y, anchorToAdd.z])
this.addAnchoredNode(new XRAnchorOffset(anchorUID), anchorToAdd.node)
}
We need to support WebVR on any browser that has native WebVR (1.0, 1.1) support. Desktop and mobile.
Right now, XRAnchors are tracked by the polyfill but not linked to ARCore anchors in Google's WebAROnARCore (https://github.com/google-ar/WebARonARCore).
Change XRPresentationFrame.findAnchor and .addAnchor to use ARCore when possible.
if you try to add an anchor before the floor has been found you should get an error or other indication that you should wait.
The XRCoordinateSystem is implemented for stage
, eyeLevel
, and headModel
but type geospatial
is stubbed out.
Implement the WSG84 geodetic frame calculations to position a given latitude / longitude / altitude so that the origin and orientation are positioned on the plane tangent to the geodetic frame with X, Y Z corresponding to East, Up, South.
Implement the transformation calculation between geospatial coordinate systems and the other types.
from my point of sight, it would be interesting for you to:
furthermore:
Two things shall be considered in first place:
by trying to replicate wave field synthesis in stereo and implementing it throughout webcl, for instance:
from my standing point, an informed reading of what these environment really are would be pertinent, so, here's some propedeutic web bibliography:
SuperCollider is an environment and programming language originally released in 1996 by James McCartney for real-time audio synthesis and algorithmic composition.
Since then it has been evolving into a system used and further developed by both scientists and artists working with sound. It is an efficient and expressive dynamic programming language providing a framework for acoustic research, algorithmic music, interactive programming and live coding.
Released under the terms of the GPLv2 in 2002, SuperCollider is free and open-source software.
Csound is a computer programming language for sound, also known as a sound compiler or an audio programming language, or more precisely, an audio DSL. It is called Csound because it is written in C, as opposed to some of its predecessors.
It is free software, available under the LGPL.
Csound was originally written at MIT by Barry Vercoe in 1985, based on his earlier system called Music 11, which in its turn followed the MUSIC-N model initiated by Max Mathews at the Bell Labs. Its development continued throughout the 1990s and 2000s, led by John ffitch at the University of Bath. The first documented version 5 release is version 5.01 on March 18, 2006. Many developers have contributed to it, most notably Istvan Varga, Gabriel Maldonado, Robin Whittle, Richard Karpen, Michael Gogins, Matt Ingalls, Steven Yi, Richard Boulanger, and Victor Lazzarini.
Developed over many years, it currently has nearly 1700 unit generators. One of its greatest strengths is that it is completely modular and extensible by the user. Csound is closely related to the underlying language for the Structured Audio extensions to MPEG-4, SAOL.
Pure Data (Pd) is a visual programming language developed by Miller Puckette in the 1990s for creating interactive computer music and multimedia works. While Puckette is the main author of the program, Pd is an open source project with a large developer base working on new extensions. It is released under a license similar to the BSD license. It runs on GNU/Linux, Mac OS X, iOS, Android and Windows. Ports exist for FreeBSD and IRIX.
Pd is very similar in scope and design to Puckette's original Max program, developed while he was at IRCAM, and is to some degree interoperable with Max/MSP, the commercial successor to the Max language. They may be collectively discussed as members of the Patcher[2] family of languages.
With the addition of the Graphics Environment for Multimedia (GEM) external, and externals designed to work with it (like Pure Data Packet / PiDiP for Linux, Mac OS X), framestein for Windows, GridFlow (as n-dimensional matrix processing, for Linux, Mac OS X, Windows), it is possible to create and manipulate video, OpenGL graphics, images, etc., in realtime with extensive possibilities for interactivity with audio, external sensors, etc.
Pd is natively designed to enable live collaboration across networks or the Internet, allowing musicians connected via LAN or even in disparate parts of the globe to create music together in real time. Pd uses FUDI as a networking protocol.
PWGL is a program that gives the user a graphical interface to doing computer programming to create music. The interface has been designed for musicians, with many objects that allow one to see, hear, and manipulate musical materials. PWGL's interface is similar to other applications, including OpenMusic, Max/MSP, and Pd. It is most similar to OpenMusic, because both share lineage as successors to the 1980s-90s application Patchwork (the PW in PWGL refers to Patchwork.)
For those familiar with Max/MSP or Pd, the biggest difference to know about PWGL is that generally all user patches are organized in the form of a tree, with many computations that happen in the "leaves" and "branches" that feed into one another and end at the bottom of the patch with one object that is the "root." The user activates the patch by evaluating this root object, which then calls all the other objects successively up the tree to the leaves, in a recursive fashion. The outermost leaves then evaluate and feed their results back down. This happens through all levels of the patch back to the root object. When the root object evaluates, it sends the final answer to the user.
Users may evaluate the patch at locations other than the "root" object. The object called for evaluation will call up its own branches and leaves and output its result to the user. Other branches of the patch will not evaluate, nor will levels of the patch below this node. To evaluate an object, select it and hit 'v' (for "eValuate"!). Instructions for how to select objects are below.
from my perspective including any of these environments in a webvr framework would be highly benefitial because:
bearing this in mind:
so, my question is
WFS is based on the Huygens–Fresnel principle, which states that any wave front can be regarded as a superposition of elementary spherical waves. Therefore, any wave front can be synthesized from such elementary waves. In practice, a computer controls a large array of individual loudspeakers and actuates each one at exactly the time when the desired virtual wave front would pass through it.
The basic procedure was developed in 1988 by Professor A.J. Berkhout at the Delft University of Technology.[1] Its mathematical basis is the Kirchhoff-Helmholtz integral. It states that the sound pressure is completely determined within a volume free of sources, if sound pressure and velocity are determined in all points on its surface.
Therefore, any sound field can be reconstructed, if sound pressure and acoustic velocity are restored on all points of the surface of its volume. This approach is the underlying principle of holophony.
For reproduction, the entire surface of the volume would have to be covered with closely spaced monopole and dipole loudspeakers, each individually driven with its own signal. Moreover, the listening area would have to be anechoic, in order to comply with the source-free volume assumption. In practice, this is hardly feasible.
According to Rayleigh II the sound pressure is determined in each point of a half-space, if the sound pressure in each point of its dividing plane is known. Because our acoustic perception is most exact in the horizontal plane, practical approaches generally reduce the problem to a horizontal loudspeaker line, circle or rectangle around the listener.
well, I have nothing against google, they make really outstanding stuff:
from my perspective, working with pwgl, csound, libpd and supercollider, within the context of vr may, within thyself, may be benefitial, as:
and better:
in fact: supercollider:
i think it could, within thyself, be a good option to at least consider:
Right now, there is no way to request the Reality camera data in order to do computer vision tasks like marker detection.
Stub out an API on Realities to request access to the camera data.
Stub out an API on Realities so that CV libs that detect markers can integrate them as XRAnchors.
Eventually the UA should display the camera data without giving the JS app access and only when the app requests direct access to the camera data will it trigger the security prompt and check. For now, UAs that are giving the camera data via the WebRTC media streams use that security prompt and check.
And add an examples
directory on it. I think is more important to emphasize the polyfill than the examples
It could be interesting to pass the delta information when requestanimationframe is called:
webxr-polyfill/polyfill/XRSession.js
Line 51 in a7abb65
I tryed to import controls using :
var orbitControls = new THREE.OrbitControls( this.camera,this.renderer.domElement);
so that I can rotate,zoom or pan on the THREE models like the "Teaport", but it doesn't work.
Is there any solutions?
Right now, if the device changes orientation then the THREE.Renderer is not updated to handle the new aspect and canvas size.
Handle the orientation change events and up date the Renderer.
Want to get the cardboard-style VR supported on mobile for doing stereo VR.
At this point, it is unclear if we want to support stereo AR on mobile devices with only one camera: in Argon4, we opted for "stereo" where the same video image is shown on both eyes. That may be sufficient for simple demos, but has issues when an object is up close.
Currently, platform logic is dispersed throughout the polyfill code, in sections that are separated by if statements such as
if(this._vrDisplay !== null){ // Using ARCore
...
} else if(ARKitWrapper.HasARKit()){ // Using ARKit
...
} else {
...
}
I think a platform abstraction layer would simplify things. This platform abstraction can be associated with XRDisplays:
XRDisplay {
_platform: PlatformAbstraction;
}
The XRDisplay should be able to choose the platform, and Realities should use whichever platform is exposed by the XRDisplay (without having to know anything about the platform implementation). This will also make it much easier to extend the polyfill to work with new platforms (especially without having to modify Reality implementations).
I wanted to let you know that I am working on adding support for the Argon Browser to the polyfill, and plan to expose a way to do marker tracking with Vuforia. I'm not sure yet what that should look like, but please suggest anything if you have ideas.
My thoughts are that there can be an extension API similar to webgl-extensions (https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/Using_Extensions), perhaps by exposing a similar "getExtension" API in the Reality class:
const vuforia = session.reality.getExtension('ARGON_Vuforia');
vuforia.init({}); // license data
vuforia.objectTracker.createDataset("dataset.xml").then((dataSet)=>{
vuforia.objectTracker.activateDataset(dataSet);
// etc.
})
Anyway, after loading and activating a Vuforia dataset, the trackables contained in that dataset would have to be made available somehow. Trackables can have a known or unknown pose, however right now it doesn't seem to be the case that XRAnchors can have an "unknown" pose state, so I'm not sure what the best way to make them available as XRAnchors would be. Would it be okay to extend the XRAnchor to have various states, so that applications can hold to a single Anchor reference as it gains and loses tracking?
I found that the WebXR Device API draft uses requestDevice to get XRDevice:
https://immersive-web.github.io/webxr/spec/latest/#xrdevice-interface
but the webxr-polyfill hasn't declared XRDevice yet?
Perhaps I’m unclear on how to load these examples outside the native apps. Is it helpful to have https://mozilla.github.io/webxr-polyfill/examples/ar_simplest.html, for example, be accessed by URL?
My reason for asking: @blairmacintyre wants Google Analytics to be added to the Polyfill.
Let me know if I’m missing something. Thanks!
The FlatDisplay currently only supports augmentation sessions, and rejects sessions that request a virtual reality.
Enhance the FlatDisplay on supported browsers to accept VR sessions and show a virtual Reality instead of a camera view.
In AR, the idea that the local area consists of a flat floor (aka stage) will often not be true. In VR, the platforms may expose a stage center in the tracker coordinate system and may expose a 2D polygon that determines the clear space in which to move.
Expose a 'floor' XRAnchor that is always at floor level (as near as the platform can determine) underneath the current head pose.
Expose the VR style stage info (center point, polygon) through API on XRSession, and remove the 'stage' XRCoordinateSystem.
Update the examples to use the floor anchor.
Hi there! I am trying to see an animation that I uploaded to the AR hit test test. I uploaded the gltf with an animation attached, however it does not play when I open the scene. It is a static object. Do I have to add something or delete something to/from the code to get it working?
Each demo should have a div overlay with a description of what the demo does. When the user taps on the div it should disappear.
In world-scale apps, existing anchors will be become untrackable in some cases like when the user walks out of a room. Right now, the polyfill assumes that anchors are permanently trackable and has no way to indicate otherwise.
Add a flag to anchors aor fire an event when they become untrackable.
When running a virtual reality display, both magic-window and stereo ones, on a mobile device that supports ARKit/ARCore, we should use those trackers for doing 6DOF tracking.
For devs who don't want to use the XRExampleBase, but do use THREE, it takes a fair amount of work to get going with WebXR.
Ease the THREE.js devs' lives by creating an XR equivalent to VREffects and VRControls.
The Boombox demo needs a better description of what it demonstrates.
When viewing the examples on the WebARCore viewer, it looks as though the rendering is upsampled with no AA:
https://i.imgur.com/4bWInpK.jpg
https://i.imgur.com/NGch3cl.jpg
Taken on a Galaxy S8.
Javascript programmers should be able to ask to destroy any anchor. For anchors they created, this should work. For system-created anchors, this may or may not work (system dependent.)
Some apis that reference an "Anchor" are actually expecting or returning an XRAnchorOffset
or a string
, and other apis either expect or return an XRAnchor, which seems confusing.
For example, here are my immediate expectations of various apis just based on the names of the methods, in comparison to the actual implementation:
XRPresentationSession.anchors
Expectation: a sequence of XRAnchor instances
Implementation: Works as expected
XRPresentationSession.addAnchor
Parameter Expectation: Accepts an instance of XRAnchor.
Implementation: Accepts a coordinate system, position, and orientation, and returns a string (a uuid for an Anchor). (No XRAnchor instance is passed or returned at all!). Also, the name addAnchor
implies that we are adding an existing instance of an XRAnchor (rather than creating a new instance), so I think it would be better named createAnchor
.
XRPresentationSession.removeAnchor
Expectation: Accept an instance of XRAnchor.
Implementation: Accepts an anchor uid (string). Rename to removeAnchorByUid
?
XRPresentationSession.getAnchor
Expectation: Should return a an instance of XRAnchor
Implementation: Somewhat works as expected, but can be made clearer if the method was named getAnchorByUid
XRPresentationSession.findAnchor
Expectation: Should return a promise that resolves to an instance of XRAnchor
Implementation: Returns a promise that resolves to an instance of XRAnchorOffset, which is NOT an XRAnchor (does not inherit from XRAnchor). If this is going to return an XRAnchorOffset, the name should probably be changed to findAnchorOffset
or something similar.
XRPresentationSession.findFloorAnchor
Expectation: Should return a promise that resolves to an instance of XRAnchor
Implementation: Returns a promise that resolves to an instance of XRAnchorOffset, which is NOT an XRAnchor (does not inherit from XRAnchor). Like the previous one, should probably be changed to findFloorAnchorOffset
The session options have changed to include the target canvas for the base layer.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.