For example, if a person and a car are detected, determine their distances in meters from the camera.

Create 2 Mono Camera nodes + Color camera node and use 3 instances of VidEnc to perform encoding

Code samples currently aren't consistent in their usage of the python and pip commands. This can be an issue if both Python 2 & 3 are installed as usually python3 and pip3 must be used for Python3. This is also the case on our RPi Compute Edition.

To make it very clear that we're using Python3, change all code samples to use python3 and pip3.

/cc @Luxonis-Brandon

Hi,

The install_dependencies.sh script is failing on Ubuntu 22 because of python-dev and libdc1394-22-dev

at https://docs.luxonis.com/projects/api/en/latest/components/nodes/stereo_depth/#min-stereo-depth-distance something doesn't align with my understanding.

the 4th paragraph "However this distance..." writes there is an alternative and that alternative needs two setting changes: 1) change resolution to 640x400, and 2) enable extended disparity.

Later it writes the formula with those setting changes as...

min_distance = focal_length_in_pixels * baseline / disparity_in_pixels = 882.5 * 7.5cm / 190 = 34.84cm

But that formula doesn't align with my understanding. Here and in the section above this issue's "Calculate depth using disparity map", the focal length at 800p = 882.5 px. And focal length at 400p = 441.25 px.

That formula should have 441.25 in it. Not 882.5.
--or--
the text for two setting changes isn't correct.
--or--
I dont have clear understanding 🤪

It's easy for users to miss the Movidius install step when setting up OpenVINO. They aren't using an NCS - they are using DepthAI. We need the same USB rules to function.

Clarify this in our install steps.

Via https://smallseotools.com/google-index-checker/, I don't believe https://docs.luxonis.com is currently indexed by Google:

This makes it hard to find everything on Google about our products.

Setup Google Search Console for our docs.

Running this example:
https://docs.luxonis.com/projects/api/en/latest/samples/26_1_spatial_mobilenet/

When downloading the blob, the file that is downloaded is mobilenet-ssd_openvino_2021.2_6shave.blob but the script/example on the docs page is expecting mobilenet.blob, resulting in this error after download:

Traceback (most recent call last):
  File "261.py", line 60, in <module>
    spatialDetectionNetwork.setBlobPath(nnBlobPath)
RuntimeError: NeuralNetwork node | Blob at path: /Users/leeroy/plgnd/models/mobilenet.blob doesn't exist

Right now, we just say "Ubuntu": https://docs.luxonis.com/api/.

Update w/specific Ubuntu versions (ex: mate currently isn't supported).

Under Manual DepthAI installation, the commands for the section (Optional) Create python virtualenv is not shown on the docs.luxonis page. Looking at the source .rst file the commands are present, but are just not being shown in the docs.luxonis website.

Visual proof:

Tested using the following browsers:

• https://www.whatsmybrowser.org/b/F30FI
• https://www.whatsmybrowser.org/b/5XISC

There are cases where a user may need to roll their own bootable uSD card on an RPi capable of running DepthAI:

• Their existing uSD becomes corrupted / damaged
• They wish to use a larger uSD card want to migrate to that

Provide instructions for doing this (and/or setting up DepthAI from a standard RPi image) in a guide.

luxonis/depthai#41 added a max_fps setting. Add information on this to https://docs.luxonis.com/api/#depthai_create_pipeline.

Color

The IMX378 color module has a diagonal field of view (DFoV) of 81.1 +/-2 with a resolution of 4056x3040.

Stereo Cameras

The OV9282 grayscale (2x used for depth) has a diagonal field of view (DFoV) of 83.76, horizontal (HFoV) of 71.86, and vertical (VFoV) of 56.74

/cc @Luxonis-Brandon

luxonis/depthai#41 added an fps_limit option.

Update this step: https://docs.luxonis.com/products/stereo_camera_pair/#test_depth

To use this command:

python3 test.py -co '{"streams": [{"name": "depth_sipp", "max_fps": 12.0}]}'

The USB3 install instructions are the most involved as we need to help the user install software on their host and setup the cameras.

I'd like to get a hand on a first pass for the instructions. I've put together an outline, but this could be wrong.

To get started:

• Fork this repo in the GitHub UI
• Modify setup.md with your take on the instructions. You can use markdown and/or HTML, but don't worry much about style, proofreading, etc. I'll help with that.
• You can edit the file via the GitHub UI (see the pencil icon) and preview as well (since it is markdown). Alternatively, you can download the source and edit there.
• Don't worry about running Jekyll locally - I'm fine w/a quick pass.

12MP (4056x3040) : 21.09fps (390MB/s) on Gen1, 41.2fps (762MB/s) on Gen2
4K (3840x2160) : 30.01fps (373MB/s) on Gen1, 60.0fps (746MB/s) on Gen2

Adding job to the Github Workflow as follows led to various issues with Homebrew:

  install-macos:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v2
      - run: source/_static/install_dependencies.sh

Attempting to install python3 and git using Homebrew caused the following errors:

Error: [email protected] 3.9.1 is already installed
...
Error: git 2.29.2 is already installed

Attempting to upgrade those packages caused the following errors:

==> Installing [email protected]
==> Pouring [email protected]_6.catalina.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3'

To force the link and overwrite all conflicting files:
  brew link --overwrite [email protected]

To list all files that would be deleted:
  brew link --overwrite --dry-run [email protected]

It seems like Python and git are already installed by some other means than homebrew, perhaps using Xcode?

The image here shows a 22-pin connector for the left and right cameras, whereas it is actually a 20-pin connector.

When I used anti-spoof-mn3 pretrained model, I was this error

The code for this is effectively done, here:
https://luxonis-community.slack.com/archives/CT79Z6GP5/p1607116306041700?thread_ts=1606945766.006400&cid=CT79Z6GP5

From this issue:

So the depth stream not working turned out to be a loose flex cable on the right camera @itsderek23. I have had it slightly pop up a few times now.

I've had a similar issue once with the RPi Compute.

The use of nodeMap[id] = node; at line 89 of Pipeline.hpp fails.
Is this is trying to insert an item in an unordered map?

I just got the OAK-D Lite, connected it, and went to https://docs.luxonis.com/en/latest/ and then to https://docs.luxonis.com/en/latest/pages/tutorials/first_steps/#first-steps-with-depthai as instructed.

It said for windows to just use the installer.

I run the installer, and then the program automatically runs. It then says "Version 3.0.10 is available. Current version is Unknown Update?"
if I hit "yes" it just brings up this prompt again. If it hit "no" it starts the program. I get a GUI with mostly blank screens

windows then makes the "device detected" and then "not detected" sounds a couple times before the program just crashes and quits.
here's the output from the program

If i physically detach the camera from the computer i get this error when opening the program

Start with the why:

The Gen2 Pipeline Builder (luxonis/depthai#136) allows theoretically infinite permutations of pipelines to be built. So there is no way to do what we did with depthai_demo.py in Gen1 where effectively all permutations of configs can be done via CLI options.

So for Gen2 it will be necessary to show how the fundamental nodes can be used in easy, and simple ways. This way, the pipeline consisting of multiple nodes can be easily built for a given application by piecing together these simple/clear examples.

Allowing the power of these permutations, but with the speed of copy/paste plugging together of examples.

Move to the how:

Make a bunch of simple and easy-to-read examples of its usage, so that users can understand the API easier.

These examples will then also be integrated into automated QA, so that they will be known to work for any new releases/improvements of the DepthAI library.

Move to the what:

TODO list of the examples to prepare:

• Color camera preview
• Mono camera preview
• Mono camera + Depth map
• Color Camera + Video encoder
• All streams encoding
• Color camera + MobilenetSSD
• Mono camera + MobilenetSSD
• Color camera + MobilenetSSD + Depth info
• Color camera + Crop node
• Mono camera + Crop node; PR here: luxonis/depthai-python#184
• Depth + Crop node

Initial draft:

Nothing is displayed when the RPi Compute is connected to a monitor.

The RPi may be unable to detect a connected monitor if it is in power-save mode. When this occurs, the RPi enters headless mode (no HDMI drive). Ensure the monitor is awake and active when the RPi is booting so it can detect the monitor.

/cc @Luxonis-Brandon

Background

See this Slack Community thread.

The JavaScript copy button just results in multiple [object Object] when pasted into PyCharm.

Firefox: 125.0.2
MacOS: 14.4.1
PyCharm: 2014.1.2

Short-term fix: after copying and pasting the code as normal, use regex search and replace ^\d+ to remove the line numbers.

Start with the why:

Discoverability is very important with new products. Since DepthAI is its own new category of product, discoverability is even more important than normal.

The reason discoverability is so important is that very few have the time to fully learn/understand something just to know if it can do what they want. They have to be able to spend as little time as possible to just see for themselves if this works for them. They can’t be required to learn APIs, piece code together, etc. This will artificially block many, especially busy engineers who have tons to do already, and bought this to try out in parallel to existing development architectures/efforts.

So we need to start with the ‘wow, that does effectively what I need’ moment, where the customer is running mostly what they need - prior to them having had to learn anything about how or why it works. We must do everything we can to prevent errors occuring in this experience, as each error means the person may give up and move on, which is especially unfortunate as they may then have missed an opportunity to discover that this platform is perfect for them.

Then, after this ‘wow this thing works for me’ moment (which hopefully takes effective no time), then the user can choose to learn more about what makes the system tick and get into the deep configurability of the system.

So the discoverability means that the user can discover layers of capabilities customizability after the thing is working. The configurability and flexibility of the device does not get in their way, it’s something they can discover after the system is already up and running - and they are confident it can help them.

So in short, we don’t waste their time forcing them to learn a new codebase or system of thought just to be able to try out the device.

Trying out the device should be as easy/fast as possible.

Move to the how:

Have the first thing the users experience be various use-case examples that they can just run, without having to go download a model, figure out where to place files, copy/paste code.

Ideally the user should be able to just run what they want by copying and pasting a couple lines of code, plug in the device, and presto, they have an example that is already doing close to what they want (e.g. locating an object in 3D space and tracking it, or doing gaze estimation in 3D coordinates, etc.).

Then after this, users can move onto tutorials, code samples, and the actual API reference. And advanced users can of course just skip to API - so this allows a good flow for brand-new uninitiated, but doesn’t force this flow, so that folks can skip to wherever they want, if they want.

Move to the what:

We have a flow from least difficulty, just works; requiring the least know-how/learning -> to the most difficult/requiring the most know-how/learning.

1. Examples. That just work. Allows the user to discover if this platform works for them. After which they can learn how the platform works, and the flexibility/customizability/etc. So these examples should have easy flexibility on options/control, say by command line arguments or interactive GUIs. So that the flexibility is discoverable but this flexibility does not get in their way before they discover it can work for them.
2. Tutorials. At tutorials and after, the user has already discovered that this platform is useful for their problem. So now the focus is showing as clear as possible how to do various, custom things with the platform.
3. Code Samples. These are clear, small snippets of code that work, which are building blocks that folks can use to make their own custom pipelines in Gen2 Pipeline Builder (luxonis/depthai#136).
4. API Reference. Gives the full details on what is possible with each function/etc. so that folks can take example code from above, and then tweak calls/settings/etc. to their exact needs as they get deeper into implementation.

1. Examples

This would be both the demo/s and examples. An additional menu option would be presented (Demos / examples), as well as a item in main page (as is right now with DepthAI API and DepthAI GUI). Following the page would present a subpage (still under depthai-docs-website) that would list various subprojects/repositores that contain demos / examples with short descriptions what users can find there. Eg.: DepthAI Demo ('A demo application showcasing various functionality of DepthAI') -> links to depthai repository documentation, application description. Eg.: DepthAI Examples ('Examples which can be used as a starting point for building new applications or just to check out') -> links to depthai/examples documentation where these are listed, etc...

Add Demos under SUBPROJECTS: (to be renamed to Menu or merged with Content) here:

This would be above DepthAI API (to be renamed to Library (API) Python/C++) there (which itself houses its own TUTORIALS and CODE SAMPLES).

To start with, we want to make clear what is possible without the user even having to run anything. To do this, we should have GIFS of each use-case clearly next to a clearly-titled title and succinct description of the example.

Example for https://github.com/luxonis/depthai-experiments/tree/master/social-distancing#social-distancing would be:

Title: Social Distancing
Description: Detects the 3D location of people and measures the distance between them, overlaying the distance on the previous, and producing alerts if they are too close. This code can readily be repurposed for any time of object.
GIF: Something like this:

And for each example should be "Understand it right away" sort GIFS of these pipelines running (e.g. gaze estimation, or vehicle detection -> type -> license plate OCR) that allows folks to just kind of browse and look to see what each use-case is, such that the title and description only has to be read after understanding what the thing does from the GIF.

So some of the depthai-experiments (or many/most of them) that are there now fall into the Example Use Cases. Like gaze estimation, vehicle-security-barrier, interactive face demo (from ArduCam, here), etc. But we should also add examples like using DepthAI with ROS1 and ROS2. And other use-cases, including using BW1092 to upload neural network metadata to graphs on AWS (say people counting - in which case we'd show a side-by side GIF of people going by and a graph from an Azure webpage).

And just like depthai-experiments and Gen1 depthai-demo.py, the models (and all other requisite binaries/etc.) should either be directly included or automatically downloaded, so that users don’t have to muck around with downloading blob files (or other resources), moving them into the right directories, renaming, etc. They shouldn’t have to do any of this. As each one of these steps is a give up point where someone may give up on the platform, and not be able to discover that it could be perfect for their use-case - purely because they're already busy and so don't have time to fight code/debugging to just discover if this thing is useful.

We need broad coverage with these just work Example Use Cases. So covering pose estimation, hand-tracking, etc. The broader the better, as this is what provides validation to someone who finds us on the internet and snags a unit, that this is for them. So for example we should have both of geax hand-tracker (here) and 3D body-pose (here) examples in this Example Use Cases section.

This way, folks across all sorts of applications can get to a basic working proof of concept with as little time investment as possible. The more points we allow for something to go wrong (like having an incorrectly labeled blob file, or missing file, etc.) the higher the probability that a user will get blocked and not be able to discover the power of this platform, purely because of some file-structure issue.

The key is to remember is that to these new users, we’re just some company off the internet. We have to make our stuff just work on the first impression, otherwise there’s a high probability of just giving up. Everyone is busy, and if we can’t get a good first impression, there’s a high chance that engineers will have to get back to their other pile of work and label us as ‘doesn’t work’ or too hard even if we would have been perfect for their use-case. We would end up blocking the discoverability of if our platform would be useful with complexity of getting set up and running. That's what we want to avoid.

Final structure of demos / examples from depthai-docs-website perspective would look like:

   docs.luxonis.com
          │
          ▼
   Demos / examples
         │
     ┌───┴────┐
     ▼        ▼
 depthai   depthai/examples

  ....        ....
    List of entries

Then after folks have tried these out, and said cool, this thing will work for my application, they can move onto the following:

2. Tutorials

Just like demos / examples, we should have an overview page that lists tutorial for respective repository / group and following that link should present a list of tutorials in that section/group with GIFS next to the title and description for each tutorial. That way the user can discover if this tutorial would be applicable for what they are wanting to accomplish before having to even click on it, just by browsing this page of GIFS/Titles/Descriptions.

Tutorials are much-more in-depth on everything they cover, example here, and are intended to really dig in. So after a user has gone through a tutorial, they should thoroughly understand every line of code and options thereof.

As such, there will be fewer tutorials to start (as they're so much more time-consuming), and the tutorial format will not be for everyone, as some programmers will not want this depth, and will actually find these annoying in comparison to having a Code Samples that just gives the code to pull something off.

So then after Tutorials will be Code Samples, but Tutorials come first as they do allow the person who is very interested in learning something deep to do so without confusion, and more advanced users can always skip to code samples or the API reference directly.

3. Code Samples:

Code Samples will live in Library (API) Python / C++ section as they directly relate to that repository.

For those who have made it past tutorials and understand the system deeply now, or otherwise for those for which this is all that they need.

Similar to Tutorials we should have GIFs showing what the code sample does on an overview page with Title and succinct description of each code sample.

4. API Reference

API Reference will live in Library (API) Python / C++ section as it directly relates to that repository.

For those who are fully bought in to making something. These are for folks who are now at the point where they are doing something that is beyond what we have even done with the platform. And either came in with substantial know-how, or have since learned it to be able deftly use the API. The API reference should be thorough and hold no detail back and can be incredibly long.

Code Locations:

So we are thinking that 1 (Demos / examples) will link to "subprojects": https://github.com/luxonis/depthai, which will be relabeled to depthai-demo, which is where these demos of example use-cases will live and be maintained (just like in Gen1, with depthai_demo.py), https://github.com/luxonis/depthai-experiments as well as some potential 3rdparty demos / examples that would like to be showcased here.

Second (Tutorials) will present groups of various tutorials, touching topics from OpenVINO, Model training, Library tutorials (links to depthai-python subprojects -> tutorials), ...

Code samples and reference will be specific to Library (API) Python / C++ and will be accessible by visiting Library link and then navigating to samples or reference.

I request the docs clarify and add usage information for category and instanceNum from ImgFrame. I see the fields in RawImgFrame and there are APIs to get/set them on the ImgFrame. And examples\host_side\queue_add_callback.cpp uses the instanceNum.

Here are the topic areas I request be clarified in the documentation

1. ImgFrame can be created on device by cameras, NN, scripts, etc. It is unclear what sets the instanceNum. Is it set by dai::CameraBoardSocket? But then what is instanceNum on a depthmap made by StereoDepth? Is it the order in which the config.create<> call is made? Or order of ->out.link() to a specific xout from which the ImgFrame was get()?
2. What is category and who sets it on the device?
3. Can I declare on the device what the instanceNum and category should be? For example, I want instanceNum for StereoDepth=0 color=1 and pose=2. And even when I don't enable stereodepth...the color will still =1 and pose=2. In this way I can write consistent host-side logic to always process a frame.instancenum=1 as a color frame.
   • Maybe can make a device-side script node and pipe all data into it and (re)set the instanceNum and category as I want and then xout everything from the script node?
   • Or do I give the desired instanceNum as a parameter to the node's create, node.properties, xout.properties, etc. so all the ImgFrame they process have that specifically declared instancenum?

https://shop.luxonis.com/products/bw10980bc

Not part of CrowdSupply campaign, so less important.

Hi. I request the docs increase details on the homography rectification/mesh feature area... to include at least the details surfaced in discord.

With the upcoming useHomographyRectification() API, when false ...
What is the mesh step?
Does it use all 14 of the distortion coeffs?
Is it functionally the same as cv::initUndistortRectifyMap()?

[7:37 PM] Luxonis-Szabi: What is the mesh step?
Mesh step is what it's set in config, default 16.
Does it use all 14 of the distortion coeffs?
we use only 8.
Is it functionally the same as cv::initUndistortRectifyMap()? 
yes
loadMeshData overrides useHomographyRectification

Could be more to add of which I'm not aware. The above are a good start. 👍

In https://github.com/luxonis/depthai-docs-website/blob/master/_tutorials/openvino_model_zoo_pretrained_model.md#run-myriad_compile, we suggest a number of argument values for myriad_compile, but we don't provide background on why.

Provide this information.

For example, if A is 1.5 meters from the camera, and B is 2 meters from the cameras, the depth separation is 0.5 meters.

I tried to convert the file through the online MyriadX Blob converter at http://luxonis.com:8080/ but the server taking too long to respond and the page did not load.
then I follow the blobconverter PyPI package https://pypi.org/project/blobconverter/
when I use the command line to do the Conversion, I got this message " ** raise FileNotFoundError(file_path)** "

when I use the Python script, I got the same message

Am I doing something wrong?

It seems likely that folks will search by our products via their model numbers, esc. if they are staring at a board and aren't familiar with it. We use these model numbers in the page urls but they aren't displayed on the page or in the HTML title or description.

Sprinkle in the model numbers.