ambianic / fall-detection Goto Github PK

Python ML library for people fall detection

License: Apache License 2.0

Jupyter Notebook 99.61% Python 0.38% Shell 0.01%

people-fall-detection fall-detection ai ml posenet python movenet

fall-detection's Introduction

Person Fall Detection

This repo is the new home of the fall detection model used in Ambianic Edge. We moved the code over from the ambianic-edge repo and made it available as a standalone library in order to foster further research and use outside the Ambianic.ai project.

What's included in this repo

Standalone Python ML library for people fall detection based on Tensorflow and PoseNet 2.0.
Python source code
Jupyter Notebook to interactively test and experiement with the model
CI & test suite
Training and testing data sets
Third party ML models used as building blocks
Wheel package published on PyPi

Project motivation

For many adults, one of the most difficult decisions to make is how to care for an elderly parent or relative that needs assistance. The AARP has found that almost 90% of folks over the age of 65 prefer to remain independent by living out their golden years at home.

Whether living alone or with family members, elderly parents need constant monitoring. Why? This is because as they age, their risk to potentially life-threatening accidents increases.

In fact, a slew of researches reveal that seniors are more prone to fall than other age classes. Falls are the leading cause of fatal injury and the most common cause of nonfatal trauma-related hospital admissions among older adults.

In a recent guest blog post for Linux Foundation AI & Data we shared the background of the problem and current market solutions.

How it works

The Fall Detection algorithm fits well with the Ambianic framework of privacy preserving AI for home monitoring and automation. The following diagram illustrates the overall system architecture. End users install an Ambianic Box to constantly monitor a fall risk area of the house. If a fall is detected, family and caregivers are instantly notified that their loved one has fallen down.

In the current design we use a combination of the PoseNet 2.0 Deep Neural Network model and domain specific heuristics to estimate a fall occurance. The following diagram illustates the main steps.

Experiment

Experiment with the fall-detection module using simple script, jupyter-notebook or command line input(CLI) by feeding 2 or 3 images. The input images should be spaced about 1 second apart.

Run a Python Script

python3 demo-fall-detection.py

Jupyter Notebook

Source code of the Demo.ipynb notebook is available in this repo.

To run an interactive version on binder, click here.

For more information how to work with Jupyter Notbooks on Github, see this document.

Execute the below command for CLI usage

To test fall-detection using the CLI for 2 images:

python3 demo-fall-detection-cmd.py --image_1 Images/fall_img_1.png --image_2 Images/fall_img_2.png

To test fall-detection using the CLI for 3 images:

python3 demo-fall-detection-cmd.py --image_1 Images/fall_img_1.png --image_2 Images/fall_img_2.png --image_3 Images/fall_img_3.png

Limitations

Based on testing and user feedback we are aware of the following limitations for the Fall Detection algorithm:

Distance from monitored area: Optimal about 15-25 feet (5-8 meters). The camera has to be able to see the whole person in standing position before the fall and fallen position after the fall. If the camera is not able to see a substantial part of the body, the Fall Detection model may not have sufficient confidence in its estimation of the situation.
Camera hight: Optimal at about human eye level: about 4-7ft (1-2 meters). If the camera is angled too low or too high overhead , the PoseNet model is not able to estimate body keypoints with confidence.
Lighting condition: Good lighting is required for optimal performance. Dim light reduces PoseNet's confidence in keypoint estimates.
Single person: The model is optimized for situation when a person is home alone. If there are multiple people in the camera view, that may confuse the model and lead to false fall detections.
No clutter: The model performs best when the area being monitored is relatively clear of various small objects. Areas cluttered with objects may confuse the model that some of these objects look like people.
Occlusions: The model suffers when the person falls behind furniture (e.g. a table or a chair) that blocks the camera from seeing a substantial part of the body.

Future work

Updating pose detection to MoveNet

The current version of the Fall Detector uses PoseNet Mobilnetv1. We have done research with a more recent MoveNet model from Google AI and published a comparison between PoseNet and MoveNet on fall data.

There are other recent models such as BlazePose 3D that have yet to be benchmarked on fall data. We welcome contributions from other researchers.

As we work with families and caregivers to test the system in real world scenarious, we expect to develop better intuition for the key factors that determine a fall in a sequence of video frames.

Moving towards on-device transfer learning for fall classification

Eventually we expect to replace some of the current heuristics with learned models that are able to more precisely distinguish between true falls and non-falls (e.g. bending over or squating to tie shoes).

We began testing an on-device transfer learning approach using TFLite Model Maker. Here is our working notebook with early results.

Your Contributions

Ideas and constructive criticism are always welcome. Feel free to join the discussion on Slack, open a github issue or a PR draft.

fall-detection's People

Contributors

Stargazers

Watchers

Forkers

datajoker1 piramid frikhaeya jainalphin bhavikapanara bhavikamkanani ivelin fernandotenorio qintxwd bernardngangbam thejasonfisher fishzhang1999 siddharthaarrolla zstreeter vamshi-amararapu

fall-detection's Issues

Optimize CPU usage - rotate image only if there is a standing up pose in the frame buffer

Looking at the ambianic edge logs in a real world usage, there is a constant stream of attempts to detect a pose in the original image and +/-90' rotations. This happens because most of the time there is no person in the camera view at all.

This is a suboptimal 3x use of CPU. Normally a single posenet pass on Raspberry Pi takes about 300ms (3fps). However after 2 rotations the total inference time goes up to 1-1.2sec (0.8-1fps). See log excerpt below:

ambianic-edge    | 2021-02-04 01:17:45 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1015.96 ms, 0.98 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:46 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1094.21 ms, 0.91 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:47 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1069.70 ms, 0.93 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:48 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1163.64 ms, 0.86 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:49 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1024.56 ms, 0.97 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:50 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1090.12 ms, 0.92 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:51 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1089.76 ms, 0.92 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:52 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1223.01 ms, 0.82 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:54 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1116.52 ms, 0.89 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:55 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1173.54 ms, 0.85 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:56 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1366.06 ms, 0.73 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:57 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1145.92 ms, 0.87 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:58 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1139.79 ms, 0.87 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:17:59 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1051.62 ms, 0.95 fps in pipeline area_watch

ambianic-edge    | 2021-02-04 01:18:01 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1107.74 ms, 0.90 fps in pipeline area_watch

move fall detection library out of ambianic-edge into independent pypi package

Pull fall detection code out of ambianic-edge into this standalone repo as an independent python library.
The goal is to allow data scientists to use, collaborate and improve people the fall detection ML model independently of ambianic deployments.

Publish pose-detection as a standalone repo and pypi package

Is your feature request related to a problem? Please describe.
Looking at Pypi and searching on google, I could not find a python version of the TFJS pose-detection API.

There are several pose estimation models (tf-pose) on github and pypi that are not optimized for running tflite on edge devices (such as RPI) with CPU only (no GPU) and do not seem to be actively maintained.

Describe the solution you'd like
Let's extract the pose detection part into its own repo and pypi package and offer it to the broader community for collaboration.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG] Fall detection test fails intermittently in PR #282

Describe the bug
It appears that the newly merged PR ambianic/ambianic-edge#282 has tests that fail intermittently in CI on ARM CPU.

___________________ test_fall_detection_2_frame_back_case_2 ____________________

    def test_fall_detection_2_frame_back_case_2():
        """
            Expected to detect a fall using frame[t] and frame[t-2].
            frame[t-2] : A person is in standing position.
            frame[t-1] : A person is mid-way of fall.
            frame[t]   : A person is fall down.
        """
        config = _fall_detect_config()
        result = None
    
        def sample_callback(image=None, inference_result=None, **kwargs):
            nonlocal result
            result = inference_result
    
        fall_detector = FallDetector(**config)
    
        output = _OutPipeElement(sample_callback=sample_callback)
        fall_detector.connect_to_next_element(output)
    
        # A frame at t-2 timestamp when person is in standing position.
        img_1 = _get_image(file_name='fall_img_1.png')
    
        # A frame at t-1 timestamp when person is mid-way of fall.
        img_2 = _get_image(file_name='fall_img_2_2.png')
    
        # A frame at t timestamp when person falls down.
        img_3 = _get_image(file_name='fall_img_2.png')
    
        fall_detector.receive_next_sample(image=img_1)
        fall_detector.min_time_between_frames = 0.01
        time.sleep(fall_detector.min_time_between_frames)
    
        fall_detector.receive_next_sample(image=img_2)
        fall_detector.min_time_between_frames = 0.01
        time.sleep(fall_detector.min_time_between_frames)
    
        assert not result
    
        fall_detector.receive_next_sample(image=img_3)
    
>       assert result
E       assert []

To Reproduce
https://github.com/ambianic/ambianic-edge/runs/1810906695?check_suite_focus=true#step:3:3921

Expected behavior
Tests should pass reliably in CI.

Host environment (please complete the following information):

Hardware: github actions CI, ARM virtual environment
OS: Ubuntu
Container: docker

Additional context
In the past this has happened a few times usually because the ARM version of TFLite outputs lower confidence scores for inferences than the x86 version.

Align our PoseDetector API with the TFJS version from Google

Is your feature request related to a problem? Please describe.
Currently our pose detection API supports multiple models, which is great.

Describe the solution you'd like
It would be nice to align with the TFJS version to make it easier for a broader community of ML researchers to participate in the fall detection project.

Additional context

Usage Docs
CreateDetector is the main class where the underlying ML model is selected.
PoseDetector is the main interface that provides pose estimation given an image input.

Enhance store.py (save_detections) for fall detection

Is your feature request related to a problem? Please describe.
Currently save_detections expects a bounding box for detections. This is the case for object detection and face detection, but not for pose detection and fall detection.

Describe the solution you'd like
Enhance detection event schema for storage, notifications and UI display.

Describe alternatives you've considered
Currently fall_detection is forced to output a fake bounding box to fit the expected format, but that's not optimal.

False positive - people bending over intentionally, not falling

Is your feature request related to a problem? Please describe.
After examining my timeline I noticed that my Ambianic camera reported a "Fall Detection" when I bent over to tie my running shoes.

[BUG] Fall detection saves debugging images at INFO level

Describe the bug
Currently the fall detection code saves debug images even if the log level is not DEBUG.

According to the python log level table, if the log level is equal or lower to DEBUG, the following if statement shall be true.

However it appears that the log level specified in the ambianic config.yaml file does not trickle down to the package level logger.

To Reproduce
Run a fall detection test case that initializes the config at INFO. You will see that temporary images are written down to the file system.

Expected behavior
Debug images should be only written to the file system if the log level is DEBUG or finer grained. Not when the log level is INFO, WARNING or above.

This bug needs to be fixed and covered with test cases for the CI.

[BUG] fall detection: spinal vector unused for leaning angle calculations

Describe the bug
An LGTM bot warning revealed a legitimate issue in the code.

Minor issue is unused import:

Import of 'stacktrace' is not used.

A more potentially impactful issue is not using the spinal vector to calculate leaning angles relative to vertical axis and previous body positions.


spinalVectorEstimate = find_spinalLine()
--
  | The value assigned to local variable 'spinalVectorEstimate' is never used.

This may not have a significant impact on the heuristic algorithm results, but it does make the code confusing since we are introducing the concept of spinal vector and not making use of it as intended.

Expected behavior
Let's simplify the code and replace redundant angle calculations for left and right side vectors with the single spinal vector estimate.

User labeling and model training of fall detection model

Is your feature request related to a problem? Please describe.
Currently we use a simple heuristic that takes a before and after images from a fall sequence and compares the angle between the spinal vectors of the poses of the person in each image.

Sometimes this results in false positives , for example when someone is leaning to tie shoes or squatting.

Describe the solution you'd like
We can use the simple heuristic model as a baseline and begin training a pose classification model to gradually reduce false positives. When a fall is detected with the current model, the user can be given an option to agree or disagree based on their own assessment of the before and after images. See attached diagram. The user can vote for each before and after image whether they agree or disagree with the estimation about the pose type (non-fallen person vs fallen person). In case they disagree, the use can choose one of two options, either pick the correct kind of pose (fallen vs non-fallen) or select Neither (not a person in the image or its hard to tell what the pose is - may only the head and shoulders are visible).

Once the end user provides feedback via PWA UI, the labeling data can be fed back to the Ambianic Edge for local training (potentially via TFLite Model Maker for image classification).

The custom pose classification model can be a type of image classification which is applied on black and white images that represent a person pose stick figure in white lines on a solid black background.

A further improvement on local model training, could be federated learning (with user opt-in) applied across all ambianic edge devices.

UPDATE, Aug 15, 2021: There is a good new tutorial on implementing a simple pose classification DNN that takes MoveNet key points as input. The example is for yoga poses. We can instead classify two labels: 'fallen person'/ 'not a fallen person'. We also need it done on TFLite Model Maker instead of TF/Keras.
Currently TFLite Model Maker does not support classification based on sparse inputs such as pose keypoints, but we can try using the image classification API as described above. Draw pose keypoints as simple black and white stickfigure images and classify those with fall/no-fall labels.

Describe alternatives you've considered
A more sophisticated approach will also improve missed fall detections. That requires further research since we want to keep user label effort to a minimum. Labeling missed detections means gathering user feedback on data samples which are not automatically detected. One potential approach is to occasionally ask user for feedback on fall detections with low confidence score.

Additional context

Flow diagram for labeling UX attached below.

Edge not detecting people walking in and out and not detecting falls, false positives for objects and falls

All,

Testing the Edge/UI for the last 24 hours and I have notice that the Edge does not capture a great deal of activity ( me walking in and out of my office ) and I have "fallen" over 30 times, with the Edge capturing only 2 of them. Once on the IF camera at night, ( I spent from 11:00 PM to 2:00 AM staging "falls" in the darkness to test the IF camera. Edge only captured one of the falls. I spent 8 hours today during daylight staging falls, the Edge only capture one falls.

I have walked in and out of the room all day (daylight hours ) and the Edge does not record 50% of them.
There are a lot of captures that are "false positive" nobody in the room.
I have attached a file of the detections. There are 5 different directors. If you have any questions please ping me on slack.

Thanks
Archive.zip

[BUG] pose_engine.py runtime exception

Describe the bug
Pose engine throws intermittent runtime exceptions.

To Reproduce
Run pose engine through a variety of fall images.

Expected behavior
There should be no unexpected runtime exceptions caused by image processing during fall detections.

Host environment (please complete the following information):

Hardware: Raspberry Pi
OS: Raspberry OS
Container: docker
Version: ambianic edge 1.13.2

Additional context

2021-02-07 13:09:02 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1077.19 ms, 0.93 fps in pipeline area_watch
2021-02-07 13:09:02 ERROR /opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py.process_sample(101): Error "only size-1 arrays can be converted to Python scalars" while processing sample. Dropping sample: {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1280x720 at 0xADB66570>}
Traceback (most recent call last):
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py", line 85, in process_sample
    inference_result, thumbnail = self.fall_detect(image=image)
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py", line 344, in fall_detect
    pose, thumbnail, pose_score, pose_dix = self.find_keypoints(image)
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py", line 146, in find_keypoints
    poses, thumbnail = self._pose_engine.DetectPosesInImage(image)
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/pose_engine.py", line 146, in DetectPosesInImage
    kps = self.parse_output(template_heatmaps, template_offsets, 0.3)
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/pose_engine.py", line 89, in parse_output
    pose_kps[i, 0] = int(remap_pos[0] + offset_data[max_val_pos[0], max_val_pos[1], i])
TypeError: only size-1 arrays can be converted to Python scalars
2021-02-07 13:09:03 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1070.10 ms, 0.70 fps in pipeline area_watch
2021-02-07 13:09:04 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1003.87 ms, 0.99 fps in pipeline area_watch
2021-02-07 13:09:05 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1042.34 ms, 0.96 fps in pipeline area_watch

Fall detection does not report all the falls

Running latest release on Edge and UI.PWA

I do multiple "test" falls per day, sometimes per hour. The Edge does not capture the falls 50% of the time. No Frame is recorded in the time line.

Here is the Ambianic Log. It captures me getting up from the fall, but not the actually fall itself.

ambianic-edge | 2021-10-11 15:01:40.094 | INFO | main:join_peer_room:80 - myRoom members {'clientsIds': ['b4166b0d-d8c5-481e-a7db-e3a3590fde11']}
ambianic-edge | 2021-10-11 15:01:40 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 833.72 ms,

0.87 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): ObjectDetector inference time 313.15 ms, 0.87 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/object_detect.py.convert_inference_result(53): label: person , confidence: 1, box: (0.33572521805763245, 0.5026041756014852, 0.631360650062561, 0.9994138274672468)
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py.find_keypoints(224): A pose detected with
ambianic-edge | spinal_vector_score=0.29140514398636597 >= 0.25
ambianic-edge | confidence threshold.
ambianic-edge | Pose keypoints: {'right shoulder': [141.0, 103.0], 'right hip': [136.0, 145.0]}"
ambianic-edge |
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 279.00 ms, 1.68 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): ObjectDetector inference time 310.77 ms, 1.68 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/object_detect.py.convert_inference_result(53): label: person , confidence: 1, box: (0.4862857460975647, 0.5147159099578857, 0.6158604025840759, 0.9832253470223331)
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 0.02 ms, 3.17 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): ObjectDetector inference time 310.47 ms, 3.18 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 0.02 ms, 3.19 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:42 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): ObjectDetector inference time 310.79 ms, 3.18 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:43 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 815.36 ms, 0.88 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:43.200 | INFO | __
20211011-154321.251638-inference.json.txt
m

feat: reduce false positives when there are multiple persons in view

Is your feature request related to a problem? Please describe.
Users report that fall detection triggers false positives when there are multiple persons in view. See attached screenshots.
This is a known issue for single person PoseNet models.

Describe the solution you'd like
A sophisticated solutions would be to implement multiple person tracking in a sequence of video frames.

An interim practical improvement would be to discard frames with multiple persons. This is a reasonable approach because:

Fall detector is critically important for situations when elderly people fall while they are alone and unattended by caregivers.
If there are multiple people in a room, then triggering a fall alert is less helpful, because presumably these people can help each other and make emergency calls as needed.

For the interim solution we can use a multi person posenet model and simply discard frames when the model detects multiple persons in a frame. That will avoid potential false fall detections. Testing shows that multi-person posenet models take slightly more CPU time for inference than single person models (2-3 fps vs 3-4 fps on rpi4), which is a reasonable tradeoff since we only need 1fps for fall detection.

Describe alternatives you've considered
A sophisticated solutions would be to implement multiple person tracking in a sequence of video frames. Respectively track and detect falls for each individual person as they move between frames.

Additional context
Screenshots from users with false positives below.

Jupyter Notebook on Fall Detection data

Is your feature request related to a problem? Please describe.
We currently have a notebook that shows how PoseNet and MoveNet perform on various fall data.
https://github.com/ambianic/fall-detection/blob/main/MoveNet_Vs_PoseNet.ipynb

We also have a testsuite that establishes a baseline for fall detection performance.
https://github.com/ambianic/fall-detection/tree/main/tests

However we don't have an interactive representation of data samples which demonstrates the cases where fall detection is accurate and where it fails.

Describe the solution you'd like
It would be helpful to have an interactive notebook that provides examples of fall detection performance in the various cases of challenges that we've discovered and documented in this blog post:
https://blog.ambianic.ai/2021/09/02/movenet-vs-posenet-person-fall-detection.html

Respectively we should label each sample in the interactive data set as True/False in order to begin establishing best practices for fall detection labeling that other researches can use to contribute additional data training and test data. Ultimately the goal is to use the accumulated community contributed data for a new fall detection DNN classification model.

For each known category with challenges we should have around 10 examples with positive and negative labeled detections.

Upgrade pose detection from PoseNet MobileNetV1 to MoveNet, PoseNet 2.0 ResNet50, or BlazePose

UPDATE: June 11, 2021

Is your feature request related to a problem? Please describe.
Currently we use mobilnetv2 with a 300x300 input tensor (image) by default for object detection.

It struggles with poses of people laying down on the floor. We experimented with rotating images +/-90', which improves overall detection rates, but it still misses poses of fallen people, even when the full body is clearly visible by a human eye.

Clearly the model has not been trained on fallen people poses.

Describe the solution you'd like

Google AI introduced MoveNet on May 17, 2021:
30fps on a mobile phone. Initially for TensorflowJS with a follow up model release coming to TFLite.
- TFLite version of the model
- Interactive colab.
Google AI released PoseNet 2.0 with a Resnet50 in 2020 base model which has 5-6fps performance on desktop CPU and noticeably better detection rates. Interactive web demo here. However testing shows that even with these improvements, it still misses some poses of people laying down (fallen poses) that are otherwise easy for a human eye to recognize. See an example recorded video below that provides a reference for situations when resnet misses poses.
Google AI MediaPipe released a new iteration of BlazePose, which detects 33 (vs 15) keypoints at 25-55fps on desktop CPU (5-10 times faster than PoseNet 2 ResNet50). Testing shows that blazepose does a better job with horizontal people poses, although it still misses some laying positions. See attached video for reference. BlazePose interactive web demo here. Pose detection TFLite model here.

Additional context

Other 2D pose detection models

See TensorFlow 2 Detection Model Zoo
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md

Notice the high performance and dual purpose (object + keypoints) for CenterNet Resnet50 V1 FPN Keypoints 512x512 and CenterNet Resnet50 V2 Keypoints 512x512.

More on CenterNet and its various applications for object detection, posedetection and object motion tracking.
https://github.com/xingyizhou/CenterNet

3D pose detection
There are new models being developed for 3D pose estimation, which could further increase fall detection performance in the future.

Clean up python package naming

Describe the bug
Currently all fall detection code is under a pipeline package which is a legacy leftover from the ambianic-edge code. This is noisy and not helping with code readability.

Expected behavior
Clean up code and tests naming so it makes sense to a developer using it as an independent library outside of ambianic edge.

Screenshots

Image pre-processing for improved PoseNet inference results

Is your feature request related to a problem? Please describe.
Users report that our model detects falls better in well lighted areas than it does in areas with poor lighting conditions. Since falls can occur any time of the day, including night time, it is important to extend the range of lighting conditions we can handle. Night time is arguably more important because it is less likely for a fallen person to be supervised by a family caregiver.

Describe the solution you'd like
It appears that the current PoseNet models have been trained on images with good lighting.

Maybe we can pre-process images so their colors shift closer to daytime lighting range that the PoseNet model can pick up.

One relatively quick and simple approach to try is conventional image filter libraries such as ImageMagick. For example the -auto-level filter appears to have visible effect on dark image illumination while it leaves in place images with good lighting. See example attached of Mac Preview with Auto Levels.

The enhancement should be a particularly fast operation if we apply it only to the 20K (~300x300) thubmnail fed to the DNN model instead of the original HD image.

Here is a discussion on the comparable -auto-level ImageMagick operator.

Wand: Python bindings for ImageMagick.

Describe alternatives you've considered
New Deep neural models are also available for this task. See resources in a related issue: ambianic/ambianic-edge#233

Switch to MoveNet as a default ML model

Is your feature request related to a problem? Please describe.
We currently use PoseNet 2.0 as our default pose detection model.
However benchmarking has shown that MoveNet performs better:
https://blog.ambianic.ai/2021/09/02/movenet-vs-posenet-person-fall-detection.html

Describe the solution you'd like
Keep PoseNet as an alternative option that can be configured for use, but switch default model to MoveNet.

Implement more test cases based on public and user contributed video and image data of elderly falls

Is your feature request related to a problem? Please describe.
We continuously receive images from users with examples of false positives or missed positives. This is expected iterative process of improvement.

Describe the solution you'd like
We should add to the testsuite examples from publicly available elderly fall videos in order to improve the performance of our fall detection algorithm.

Here is one such dataset.

Additional context
Attached are additional images reported from ambianic users with false positives.

ambianic / fall-detection Goto Github PK

fall-detection's Introduction

Person Fall Detection

What's included in this repo

Project motivation

How it works

Experiment

Run a Python Script

Jupyter Notebook

Execute the below command for CLI usage

Limitations

Future work

Updating pose detection to MoveNet

Moving towards on-device transfer learning for fall classification

Your Contributions

fall-detection's People

Contributors

Stargazers

Watchers

Forkers

fall-detection's Issues

Recommend Projects

Recommend Topics

Recommend Org