In case a user doesn't specify a material for the planes they will no longer be shown (they will be there though). Currently that's not possible (the app crashes).

When it's dark it's a bit hard to track surfaces. Let's add a flashlight toggle to fix that.

Have you (or anyone) successfully rendered a model on a surface plane in android (ie andy.obj)?
Just wondering because I know android is not as high of a priority as ios, but the method addModel is implemented...

Which platform(s) does your issue occur on?

Physical iPhone X running iOS 13.1

Please, provide the following version numbers that your issue occurs with:

CLI: 6.1.0
Other versions will be latest because this occurs in the demo app.

Please, tell us how to recreate the issue in as much detail as possible.

I run the Pokémon demo from a fresh clone of the repo using npm run demo.pokemon.ios in the src directory.

The “augment card” and “search by card” pages work great, but as soon as I tap on “Try before you buy” the app crashes with this error.

file:///node_modules/nativescript-ar/ar.js:205:0: JS ERROR Error: This set of frame semantics is not supported on this configuration
*** JavaScript call stack:
(
0   UIApplicationMain@[native code]
1   _start@file:///node_modules/tns-core-modules/application/application.js:295:0
2   run@file:///node_modules/tns-core-modules/application/application.js:323:0
3   bootstrapNativeScriptApp@file:///node_modules/nativescript-angular/platform-common.js:210:0
4   bootstrapApp@file:///node_modules/nativescript-angular/platform-common.js:111:0
5   bootstrapModule@file:///node_modules/nativescript-angular/platform-common.js:91:0
6   @file:///app/bundle.js:395:144
7   ./main.ts@file:///app/bundle.js:400:34
8   __webpack_require__@file:///src/webpack/bootstrap:750:0
9   checkDeferredModules@file:///src/webpack/bootstrap:43:0
10  webpackJsonpCallback@file:///src/webpack/bootstrap:30:0
11  anonymous@file:///app/bundle.js:2:61
12  evaluate@[native code]
13  moduleEvaluation@:1:11
14  promiseReactionJob@:1:11

If the demo apps cannot help and there is no issue for your problem, tell us about it

I have installed the npm version 0.7.1 and followed https://github.com/EddyVerbruggen/nativescript-ar/blob/master/docs/ar-angular.md

I can install and start and for some seconds all works fine till I get this error:

***** Fatal JavaScript exception - application has been terminated. *****
Native stack trace:
1 0x101397370 NativeScript::reportFatalErrorBeforeShutdown(JSC::ExecState*, JSC::Exception*, bool)
2 0x1013c7dec NativeScript::FFICallbackNativeScript::ObjCMethodCallback::ffiClosureCallback(ffi_cif*, void*, void**, void*)
3 0x101e2c494 ffi_closure_SYSV_inner
4 0x101e301b4 .Ldo_closure
5 0x22d94beac
6 0x22d9495c4
7 0x2256d6d9c
8 0x2256d98b8
9 0x2256d9f24
10 0x2256da368
11 0x22578afa0
12 0x225622ca0
13 0x22574cc58
14 0x210c56484
15 0x210c035c8
16 0x22574cbc8
17 0x2156ea49c
18 0x21147db24 IODispatchCalloutFromCFMessage
19 0x211186f0c
20 0x2111ae758
21 0x2111adea4
22 0x2111a8c5c
23 0x2111a80e0 CFRunLoopRunSpecific
24 0x211b9e494
25 0x225623044
26 0x225623178
27 0x210e3825c
28 0x210e381bc _pthread_start
29 0x210e3bcf4 thread_start
JavaScript stack trace:
1 rendererNodeForAnchor@file:///app/tns_modules/nativescript-ar/ar.js:635:25
2 UIApplicationMain@[native code]
3 start@file:///app/tns_modules/tns-core-modules/application/application.js:275:26
4 run@file:///app/tns_modules/tns-core-modules/application/application.js:303:10
5 bootstrapNativeScriptApp@file:///app/tns_modules/nativescript-angular/platform-common.js:188:26
6 bootstrapApp@file:///app/tns_modules/nativescript-angular/platform-common.js:106:38
7 bootstrapModule@file:///app/tns_modules/nativescript-angular/platform-common.js:90:26
8 anonymous@file:///app/main.js:6:57
9 evaluate@[native code]
10 moduleEvaluation@[native code]
11 promiseReactionJob@[native code]
JavaScript error:
file:///app/tns_modules/nativescript-ar/ar.js:635:25: JS ERROR TypeError: null is not an object (evaluating 'faceGeometry.firstMaterial')

If I set detectPlanes="false" the app is able to run a little bit longer but still gets this error. I suppose this have to do something with faceGeometry but wondering why. I'm not using it and also tried to set trackingMode="WORLD" but still getting this problem.

Which platform(s) does your issue occur on?

iOS (only trying on iOS)

Please, provide the following version numbers that your issue occurs with:

tns version: 5.2.4

Instead of picking a chair from a catalogue and placing it in your house, we could use the Pokémon models we already have.

Simple tap a plane, pick a model and place it on the plane. Then allow the user to relocate, rotate, and scale it.

It would be helpful to fi. programmatically know if an object is in a certain place (x/y) on the screen. So fi. when a sphere/model is in or near the center of the screen we can consider the user looking at it.

Sceneform is a relatively new abstraction layer on top of ARCore. It doesn't require (plugin) developers to directly interact with OpenGL, which I needed to do previously (hence the embedded custom native library I created).

This hampered progress enormously because of the long development-build-test cycle it caused.

With Sceneform the Android API's are a lot like the iOS ARKit API's so let's use Sceneform and then get the Android implementation on par with iOS.

Generic tasks:

• Add Sceneform
• Remove our custom lib
• Bump demo to NativeScript 6

Port iOS features to Android

• World tracking
• Scaffolding for placing objects in the 3D world
• Adding a box
• Adding a sphere (thanks, @nickolanack! #38)
• Adding a tube / cylinder (see #40)
• Face detection/tracking (partly done)
• Image detection/tracking
• Gesture support
• Material support (we only support colors at the moment)

More demos

• Try to replicate the Android Solar system demo for iOS and Android in NativeScript. (implemented in #41)
• Try to replicate the Android Augmented Image demo for iOS and Android in NativeScript. (implemented in #62)
• Add an IKEA-like demo, but for Pokémon (implemented in #70)
• Try to replicate the Android Augmented Faces demo, or something custom like trying different pairs of spectacles 👓🕶(implemented in #72).
• Possibly one or more of the other Android demos.

Which platform(s) does your issue occur on?

iOS. Tested on iPhone 6S and an iPad Pro.

Please, provide the following version numbers that your issue occurs with:

• CLI: 3.4.0
• Cross-platform modules: 3.4.0
• Runtime(s): 3.4.0
• Plugin(s): nativescript-ar version 0.3.2

Please, tell us how to recreate the issue in as much detail as possible.

This one is a bit complex to explain. It happens when using this plugin’s addModel() function and setting both the scale and position properties. If you set the scale, the position at which the resized model is placed at does not take into account the new size.

For testing I’ve been using the Car.dae model in this repo’s demo app. To see this behavior in action, change the planeTapped() function in the demo’s main-page.ts file to do the following.

const ar: AR = args.object;
ar.addModel({
  name: "Models.scnassets/Car.dae",
  position: {
    x: args.position.x,
    y: args.position.y,
    z: args.position.z
  },
  scale: 0.1,
  mass: 20
});

Expected behavior: The car model appears exactly where you tap, as you set the position to the exact x/y/z coordinates of the plane tap.

Actual behavior: The car model floats above the plane, presumably because positioning occurs relative to the non-resized version of the model.

I took a stab at getting the car on the plane without hardcoding random adjustments to the positioning coordinates... and I failed. The problem is I don’t know the dimensions of the model itself, so I don’t the numbers I need to do the math against.

Currently, we require IMAGE tracking for this to work, but ARKit actually supports it during WORLD tracking as well. I think @nickolanack hinted me in another issue this is the case.

Hey @EddyVerbruggen ,

Finally got some time to try this plugin in my kitchen sink app. At first it crashed a few times as I'm using JS, not TS (I know, I need to switch over at some point) but now I'm getting the page to display, the AR component looks like it's loading but it stays dark with a hand holding a phone (which moves around a bit) and a toast message saying "Failed to create AR session". Have you seen this anywhere? Console doesn't say anything weird or specific and I've used this on two Android devices.

Hey Eddy, i appreciate your work,

Currently working on an AR project for Android where i have to integrate another THREE.js project into a multiplatform app that preferbly can run as a web app (for my bachelor thesis). Fully web based is technically impossible as far as i know.

Are you planning on integrating the second preview of ARCore? Any progress on this current project?
Is it possible to develop on Android? And if so, which platform do you suggest for developing? In your documentation it only says xCode for apple.

As per ARCore, ARCore can be added as AR Required or AR Optional.

The problem is, if I add the nativescript-ar plugin, it fails to build it on Android where minSdk < 24 and resulting into below exception.

Execution failed for task ':app:processDebugManifest'.

Manifest merger failed : uses-sdk:minSdkVersion 17 cannot be smaller than version 24 declared in library [:ar-release:] /Users/someuser/.gradle/caches/transforms-1/files-1.1/ar-release.aar/95be4c84c81ad0893ce785e810ee3205/AndroidManifest.xml as the library might be using APIs not available in 17
Suggestion: use a compatible library with a minSdk of at most 17,
or increase this project's minSdk version to at least 24,
or use tools:overrideLibrary="org.nativescript.tns.arlib" to force usage (may lead to runtime failures)

Is there a way to optionally add ARCore. So that we can check whether the device is compatible or not at runtime.

Hi @EddyVerbruggen,

Thank you for this awesome plugin !

Is it already possible to display animated models with it?

Thanks

I've tried using the experimental ARModel from #7 and also to convert the .dae file to .scn but the code does not work.

let modelScene = SCNScene.sceneWithURLOptionsError(NSURL.URLWithString(file), null);

The above code always returns null for the modelScene and there is no error thrown.

Try to replicate the Android Augmented Image demo for iOS and Android in NativeScript.

And we need an Angular demo as well, so that seems like a good opportunity.

It would be cool to replicate something like this: https://www.youtube.com/watch?v=ZhHExAxyVow

Got most of the code for this by email from @Emanuele-Fortunati 💙

Any plan to create a demo in NativeScript-Vue version? If not, i can convert?

Seems like there are a few warnings that can be fixed, and those may fix a crash I've been seeing.

Per Google's guidelines

On iOS we used to set the rotation property of the node, but it seems like the proper way to do it is setting the eulerAngles property. It behaves a lot more predictable in my experience.

Is there any reason not to?

Please publish new version (0.5) with grabScreenshot function! last npm is 0.4

If the demo apps cannot help and there is no issue for your problem, tell us about it

grabScreenshot not working

Which platform(s) does your issue occur on?

• iOS

Hello. We build app like snapchart: we have to capture camera stream, recognize some objects and place virtual objects to particular screen coordinates. We already have code that recognizes objects. Can we capture camera stream using this package now or in the future? Thanks in advance.

Scenario: an app that allows users to try on glasses (3D models of glasses, that is).

I'm pretty sure that rotateBy method for android is not using quaternions correctly
should be:

rotateBy(by: ARRotation): void {
    const currentRotation = this.android.getLocalRotation().eulerAngles();
    this.android.setLocalRotation(
      new (<any>com.google.ar.sceneform).math.Quaternion(
        new (<any>com.google.ar.sceneform).math.Vector3(
          currentRotation.x + by.x,
          currentRotation.y + by.y,
          currentRotation.z + by.z
        )
      )
    );
  }

Currently we can animate this stuff by calling fi. the rotateBy function every frame, but it would be nice to tap into native AR animations provided by both platforms.

You might want to link this article to the docs, as it is great for beginners:
https://market.nativescript.org/plugins/nativescript-ar

We should support empty nodes! Empty nodes can be use to rotate/scale/move groups of renderables

ARCore 1.1 has been released. Let's upgrade this puppy 🐶

If you look at the rings of Saturn in the image below, you can see the transparent part of the image is actually a little opaque.

@nickolanack Do you perhaps have any idea why this happens?

hello, I wanted to ask if it possible to load models dynamically from an API then be able to use it in the plugin?. and thanks for the plugin, :)

• Package.json has a bogus email address for the author; this messes up author tracking on places like https://plugins.nativescript.rocks (& probably the new marketplace too)

@EddyVerbruggen
Good news...
converting a folder of images into an .imgdb is relatively easy. Android provides a cmd line tool https://developers.google.com/ar/develop/c/augmented-images/arcoreimg and its much faster to load. So i think this should be the preferred method

we can also provide the measured width of images (similar to ios) which greatly speeds up the image tracking time! (btw. we can also specify the width when adding images manually or from folder..) - this also seems to have the effect of scaling the local dimensions of the augmented image so that adding/positioning nodes can be done consistently (just like ios is working atm)

so the scaling factor that I introduced should be removed! yay!

This is a new feature in iOS 12: you can track 1 face at a time and get back a ton of properties.

I using this in android S8+ why i dont cant add any models.

extremely small values for scale parameter don't work in android

Its more a question than a issue. The ARCore 1.0 is out and they added support down to samsung s7( https://www.blog.google/products/google-vr/announcing-arcore-10-and-new-updates-google-lens/ ). Is there going to be a android support for this plugin soon? I really love AR. Thank you

Following this tutorial for Angular, i got this I'm using iOS 12.1.2.

Getting Started with Augmented Reality in NativeScript

file:///app/tns_modules/nativescript-ar/ar.js:164:24: JS ERROR TypeError: undefined is not an object (evaluating 'this.nativeView.addSubview')

*** Terminating app due to uncaught exception 'NativeScript encountered a fatal error: TypeError: undefined is not an object (evaluating 'this.nativeView.addSubview')

• THE LOG

I tried to run demo app but seems to me not compatible with the latest CLI and Webpack.
I tried migration first:
$ tns migrate and then $ tns run ios
but failed

Unknown argument: preserve-symlinks

Can you update your repo for VueJS? Ta

Again, following the tutorial, the app now opens but all I get is the first image rendered, it then doesn't refresh and the app almost feels stuck.

The logs seem ok though:

ARImageSensor (0x280196490): Vision data delivery enabled

==========================================================
Device 0:
Device Open succeeded
{"msg":"CLGyroBiasEstimatorClientRemote::onWatchdogTimerExpiry", "event":"activity", "client":"0x2833dd8a0"}
ARImageSensor (0x280180c60): capture state changed (<private>)
ARSession (0x105c392b0): begin interruption
CONSOLE LOG file:///app/vendor.js:80736:20: >>> sessionWasInterrupted: The tracking session has been interrupted. The session will be reset once the interruption has completed
ARSession (0x105b76fc0): running
CONSOLE LOG file:///app/vendor.js:80771:24: Tracking state changed to: Limited
CONSOLE LOG file:///app/vendor.js:80773:28: Limited tracking state reason: Initializing
ARSession (0x105c392b0): interrupted
CONSOLE LOG file:///app/vendor.js:80771:24: Tracking state changed to: Normal
HTHangEventCreate: HangTracing is disabled. Not creating a new event.
World tracking performance is being affected by resource constraints [1]

ARKit 2 / iOS 12 has very cool image tracking capabilities. You can detect a know image and stick AR objects to its surface when detected in the scene.

Also, when an object is detected or added, the callback that returns a the objects should have methods to rotate and move the object, to make it more interactive.

Hello i am trying get the current device position, but I can not find anything about it in the docs.
I want to calculate the distance between a node and my device.

Hi Eddy

Is it possible to get the coordinates of where an object was Tapped or longPressed ?

Thanks

In order to streamline implementations between iOS and Android we need a little demo that showcases this feature.

In the past, I've been playing with models of glasses/spectacles on iOS and that worked surprisingly well. I recorded 2 short videos and threw them on YT: vid 1 and vid 2.

The code for those demos is currently part of the /demo app but it may be more convenient to create a dedicated demo. Not sure in which stack yet, possibly plain TS.

{N} 6 uses AndroidX.

Try to replicate the Android Solar system demo for iOS and Android in NativeScript.

And we need a Vue demo as well, so that seems like a good opportunity.

@EddyVerbruggen
I would like gesture events to propagate up the scene hierarchy until they reach a node that handles the gesture. Let me know if you see any issues with that? And I'll implement it.

I think the solar system example will benefit from this since atmosphere objects (ie: earth clouds sphere) could potentially block tap events set for the root object (planet)

Following the tutorial on https://www.nativescript.org/blog/getting-started-with-augmented-reality-in-nativescript, I get stuck after running tns plugin add nativescript-ar and editing app/home/home.component.ts.

I am getting the following error:

24  anonymous@file:///app/starter.js:<\M-b\M^@\M-&>
JavaScript error:
file:///app/vendor.js:80409:38: JS ERROR ReferenceError: Can't find variable: RecordAR

• tns version: 5.4.2
• tns-core-modules@~5.4.0

Any help much appreciated!