leonardopohl / fesd Goto Github PK

Somehow the point cloud is not oriented correctly

For better orientation an x-z plane grid would be very useful

The name of the repository and main Project are not the greatest

We need to filter the circles for their correct size for structure detection. We cannot use the OpenCV radius since it is not the accurate radius but rather a projected radius.

Take control over different parametrs of the Orbbec camera

The point cloud as it currently is does not have a correct projection. Each camera has a different v- and h-fov which needs to be accounted for. Hopefully we can just multiply the position of the point with a projection matrix. Blocks #14

Camera initialisation takes ages it would be nice to have that happen asynchronously

This Paper describes how to detect Floors, Ceilings and Walls. I need to find a way to detect this using our current data stream model #6 Depends on it!

A global log would be nice to have for now you always have to watch the console.

Instead of blindly initialising all cameras just find them and set them on standby

Move points in random direction and label as invalid.

I need them

Align multiple Cameras based on the point-cloud. Once this is done the actual thesis work can begin.

Might improve readability and makes it possible to use linter

How would human pose estimation look like?

Create a class for color schemes and don't store it in the Point class.

Something like this:

static enum class CMAP
	{
		VIRIDIS,
		MAGMA,
		INFERNO,
		HSV,
		TERRAIN,
		GREY
	};

class ColorScheme{
public:
    static void onImGuiUpdate(CMAP &cmap);
    static std::array<float, 4> getColorFromDepth(float depth, float depth_scale, CMAP cmap);
}

Or maybe more abstracted:

 static std::array<float, 4> getColorFromCmap(CMAP cmap, float val, float min, float max);

Framebuffers seem to be nice

When opening a point cloud for 2 cameras only one is showing. The slowdown is still slowing down significantly and the memory usage rises but only one is showing. This blocks #14

A major feature is the recording of the depth stream for future playback.

To improve the distance calculation speed in point clouds

Balls are useless. Lets do something else. Lets just detect the floor for every camera. With the floors we can determine the height of the cameras. Using the point clouds use some fancy ransac or something to find salient points.

• Using 2 Point+Normal Sets for fast Registration of Point Clouds with Small Overlap apparently does it in linear time cuds to them.
• DPDist
• The MIR website might also have some neat information on the topic, maybe 2D Salient points are applicable

Take control over different parameters of the Realsense camera

Use OpenGL to visualise the point cloud and stuff.

• Color Mapping does not work correctly
• Point Cloud is inverted

Quite a low priority, but implementing ImGui might help tweak the parameters and will need to be done anyways.

A lot of the pointcloud code can be moved into the shader. This is not high priority, since the recordings can be created without using pointclouds and therefore good results can be achieved. However, when playing back the recordings, this leads to delays and synchronisation problems.

A point in a pointcloud is basically just a function for the distortion of the hfov and vfov which is applied to the depth. In the Center of the image wont be any distortion and therefore it will just reflect the depth.

The current vertex shader looks like this:

#version 330 core

// Positions/Coordinates
layout(location = 0) in vec3 aPos;
// Colors
layout(location = 1) in vec3 aColor;

// Outputs the color for the Fragment Shader
out vec3 v_Color;

// Controls the scale of the vertices
uniform float u_Scale;

// Inputs the matrices needed for 3D viewing with perspective
uniform mat4 u_MVP;

void main()
{
	// Outputs the positions/coordinates of all vertices
	gl_Position = u_MVP * vec4(u_Scale * aPos, 1.0);
	// Assigns the colors from the Vertex Data to "color"
	v_Color = aColor;
}

For each vertex I receive the position and the color a new. but in theory I only need to receive the vertex offset and the depth. The color is depth dependend and can be hardcoded within the shader. So the new shader would look something like this:

#version 330 core

// Positions/Coordinates
layout(location = 0) in vec2 aPosFun;
// Depth
layout(location = 1) in float aDepth;
// X sign
layout(location = 2) in bool xSign;
// Y sign
layout(location = 3) in bool ySign;
// Z sign
layout(location = 4) in bool zSign;

// Outputs the color for the Fragment Shader
out vec3 v_Color;

// Controls the scale of the vertices
uniform float u_Scale;

// Edge Length
uniform float u_EdgeLength;

// Inputs the matrices needed for 3D viewing with perspective
uniform mat4 u_MVP;

vec3 getColorFromDepth(float depth);

void main()
{
	// Outputs the positions/coordinates of all vertices
       vec4 pos = vec4(aPosFun.x * aDepth + u_EdgeLength * (xSign ? 1 : -1), 
                       aPosFun.y * aDepth + u_EdgeLength * (ySign ? 1 : -1), 
                                   aDepth + u_EdgeLength * (zSign ? 1 : -1), 
                       1.0f);

	gl_Position = u_MVP * u_Scale * pos;
	// Assigns the colors from the Vertex Data to "color"
	v_Color = getColorFromDepth(depth);
}

vec3 getColorFromDepth(float depth)
{
   // A very long and ugly method
}

To detect faults in the skeleton, train a neural network on the faulty and correct data.

To maximise the frame rate separate recording and skeleton calculation.

Record different actions, with different camera setups. For now manually align the cameras and store the alignment, so it can be used to complete possible missing, or wrong skeletons.

Drawing a virtual physical depth camera at 0, 0, 0 of the point cloud would be nice for orientaion

Sometimes openni doesn't connect to the camera and the program needs to be restarted