dctian / deeppicar Goto Github PK

Deep Learning Autonomous Car based on Raspberry Pi, SunFounder PiCar-V Kit, TensorFlow, and Google's EdgeTPU Co-Processor

License: GNU General Public License v3.0

Python 0.53% Jupyter Notebook 99.47%

raspberry-pi tensorflow tensorflow-tutorials picar edgetpu autonomous-vehicles nvidia colab-notebook computer-vision python

deeppicar's Introduction

DeepPiCar — Part 1: How to Build a Deep Learning, Self Driving Robotic Car on a Shoestring Budget

An overview of how to build a Raspberry Pi and TensorFlow powered, self-driving robotic car

By David Tian

Hacker, tinkerer, and engineer. I am passionate about machine learning, AI, and anything technology related.

Introduction

Today, Tesla, Google, Uber, and GM are all trying to create their own self-driving cars that can run on real-world roads. Many analysts predict that within the next 5 years, we will start to have fully autonomous cars running in our cities, and within 30 years, nearly ALL cars will be fully autonomous. Wouldn’t it be cool to build your very own self-driving car using some of the same techniques the big guys use? In this and next few articles, I will guide you through how to build your own physical, deep-learning, self-driving robotic car from scratch. You will be able to make your car detect and follow lanes, recognize and respond to traffic signs and people on the road in under a week. Here is a sneak peek at your final product.

Lane Following

Traffic Sign and People Detection (right) from DeepPiCar’s DashCam

Our Road Map

Part 2: I will list what hardware to buy and how to set them up. In short, you will need a Raspberry Pi board($50), SunFounder PiCar kit ($115), Google’s Edge TPU ($75) plus a few accessories, and how each part is important in later articles. The total cost of the materials is around $250–300. We will also install all the software drivers needed by Raspberry Pi and PiCar.

Raspberry Pi 3 B+

SunFounder PiCar-V Robotic Car Kit

Google Edge TPU Accelerator

Part 3: We will set up all the Computer Vision and Deep Learning software needed. The main software tools we use are Python (the de-facto programming language for Machine Learning/AI tasks), OpenCV (a powerful computer vision package) and Tensorflow (Google’s popular deep learning framework). Note all the software we use here are FREE and open source!

Part 4: With the (tedious) hardware and software setup out of the way, we will dive right into the FUN parts! Our first project is to use python and OpenCV to teach DeepPiCar to navigate autonomously on a winding single lane road by detecting lane lines and steer accordingly.

Step-by-Step Lane Detection

Part 5: we will train DeepPiCar to navigate the lane autonomously without having to explicitly write logic to control it, as was done in our first project. This is achieved by using “behavior cloning”, where we use just the videos of the road and the correct steering angles for each video frame to train DeepPiCar to drive itself. The implementation is inspired by NVIDIA’s DAVE-2 full-sized autonomous car, which uses a deep Convolutional Neural Network to detect road features and make the correct steering decisions.

Lane Following in Action

Lastly, in Part 6: We will use deep learning techniques such as single shot multi-box object detection and transfer learning to teach DeepPiCar to detect various (miniature) traffic signs and pedestrians on the road. And then we will teach it to stop at red lights and stop signs, go on green lights, stop to wait for a pedestrian to cross, and change its speed limit according to the posted speed signs, etc.

Traffic Signs and People Detection Model Training in TensorFlow

Prerequisite

Here are the prerequisites of these articles:

First and foremost is the willingness to tinker and break things. Unlike in a car simulator, where everything is deterministic and perfectly repeatable, real-world model cars can be unpredictable and you must be willing to get your hands dirty and start to tinker with both the hardware and software.
Basic Python programming skills. I will assume you know how to read python code and write functions, if statements and loops in python. Most of my code is well documented, specifically the harder to understand parts.
Basic Linux operating system knowledge. I will assume you know how to run commands in Bash shell in Linux, which is Raspberry Pi’s operating system. My articles will tell you exactly which commands to run, why we run them, and what to expect as output.
Lastly, you will need about $250-$300 to buy all the hardware and working PC (Windows/Mac or Linux). Again, all the software used will be free.

What’s Next

That’s all for the first article. I will see you in Part 2 where we will get our hands dirty and build a robotic car together!

Here are the links to the whole guide:

Part 1: Overview (This article)

Part 2: Raspberry Pi Setup and PiCar Assembly

Part 3: Make PiCar See and Think

Part 4: Autonomous Lane Navigation via OpenCV

Part 5: Autonomous Lane Navigation via Deep Learning

Part 6: Traffic Sign and Pedestrian Detection and Handling

deeppicar's People

Contributors

Stargazers

Watchers

Forkers

tkc285 hezhongkai zyoo-info nevellson aleonnet joel-belton flavio58it rashidbakirov tangowhisky37 ucolchen goosemeister0 trunksio jdrew1303 gbrt46 vedantamvamsi ochi96 w17731138318 faizmisman yoganandhan01 sidra3 collector-m 408881465 ashish-tupate ratal henrywoo feethink moisesrc13 nestyme jcstarry jianxu305 chayanforyou coreyvessar khu834 amirunpri2018 michdo93 lizhaodong ykevin xjohnxjohn dart-bird fatmakaram97 sabuj073 jpfogato ossdc applesun8799 pddang recorsa leemgs stevepc0404 annak1n matrixue anishanne ando11235 xiangzhaoxiangzhao renping0535 amrmahmoud2 richasdy asarolia abdelpaset juppi1986 zbot473 eichizen17 shini-gami pizzamargherita21 arigadget joserfjuniorllms kappernie nithin866 techgabor emmytheo suhailpatel94 kanishkaisreal avernethy niveshkrishna alejandrocordon itsmonterey sattar-mazouzi amoidimi sangnguyen7 mpofukelvintafadzwa kaiyuryozin lucasalbini chaobo521 lealvona khaledjallouli lungaropietro tomoyosiki esylltmedi seiya303 drozdda whztt07 dimatokyo2020 bs3537 chinkevin vidi-vi bwhizzy morningcloud geee001 xzhen1175 geee1 denidampf

deeppicar's Issues

objects_on_road_processor.py

Since (https://coral.ai/docs/edgetpu/multiple-edgetpu/# ) EdgeTPU Python API has been deprecated, is there update code on objecs_on_road_processor.py? end_to_end.py ?..

_Originally posted by @dctian in #6

Error in calculating angle

Hello brother,
Calculation of steering angle is confusing me. We need to take the distance of in between the left and right detected lines. So, that where car will detect lines it will automatically calculate the angle through that lines. But you are calculating the angle on total width and height of image.
Please help me in solving this problem.
Thank you!

Create traffic sign detector with tensorflow and Edge TPU

Use transfer learning to train my road sign images with mobilenet SSD v2 on coco data set.

PiCar doesn't follow the lines and seems like it can't process the data fast enough

Hello,
I'm working as a student and my task is to build the DeepPiCar.
I followed the steps in documentaion and now the DeepPiCar should drive on its own and stay in the lanes but it is not working.
The Car is driving but it looks like it dosn't recognize the lanes. I observed that the video i see has a delay of +2sec. I think the data can't be processed.
Is the an updated version of the code or any other known issue that fixes this problem?

Thanks a lot

issue on deep_pi_car code

INFO :2020-01-08 14:56:22,072: Creating a DeepPiCar...
DEBUG:2020-01-08 14:56:22,095: Set up camera
Traceback (most recent call last):
File "deep_pi_car.py", line 136, in
main()
File "deep_pi_car.py", line 129, in main
with DeepPiCar() as car:
File "deep_pi_car.py", line 28, in init
self.pan_servo = picar.Servo.Servo(1)
File "/usr/local/lib/python3.7/dist-packages/SunFounder_PiCar-1.0.1-py3.7.egg/picar/SunFounder_PCA9685/Servo.py", line 36, in init
self.pwm = PCA9685.PWM(bus_number=bus_number, address=address)
File "/usr/local/lib/python3.7/dist-packages/SunFounder_PiCar-1.0.1-py3.7.egg/picar/SunFounder_PCA9685/PCA9685.py", line 47, in init
self.bus = smbus.SMBus(self.bus_number)
TypeError: an integer is required (got type NoneType)

Help with path planning, obstacle bypass and voice command (Jarvis) integration

Hello David,
I'm Emmanuel by name, an undergraduate of Electronics and Computer Engineering, and I must say, your work on this project has been my biggest inspiration this year. I've seen other low budget self driving car projects but honestly none shows much promise as yours and the fact that it involves stuff like computer vision, deep learning, object detection, regression and some dedicated intuition makes it more than suitable for an amazing paper and project. Thank you for opening my eyes to this level of AI Sophistication.

Now I've been following your work in detail and I'm kinda reproducing this project except for the pi car(cos it's really pricey for me right now) cos it's my absolute favorite when it comes to robotic self driving cars.

I'm using a different car chassis and a motor hat.
So my issue is,
I want to implement some cool bypassing of obstacles and some path planning (with the current sensors ie ultrasonic and front facing camera) and also some Jarvis integration.

You mentioned that the bypassing could be achieved by either deep learning or hand coded rules.
Can you walk me through how I can integrate the latter(hand coded rules) into the existing model and driving script.
And also some help with the Jarvis integration will be very much appreciated.

I look forward to hearing from you really soon cos I've got a deadline of 1month.
Thanks for your time.

end_to_end_lane_follower.follow_lane() execution is slow.

I ran end_to_end_lane_follower.py on Raspberry pi 4. It took about 200 ms to execute end_to_end_lane_follower.follow_lane().
Is it normal?

Overflow error: cannot convert float infinity to integer

Hello,

I'm currently following all steps which are awesome and well explained.
It works great, however sometimes (usually when turning), the car stops and shows an overflow error specifying : cannot convert float infinity to integer (line 294 in hand_coded_lane_follower) :
x1 = max(-width, min(2 * width, int((y1 - intercept) / slope)))

Has anyone had this issue and has been able to fix it ?

model.predict to slow (Low FPS) while running deep_pi_car.py

Hello,
First of all thanks for the great work and tutorial.
I managed to train my own model following your instruction.
When I run the coco_object_detection.py the FPS are turning around 16 FPS (which seem normale according to your explanation)
but when I execute deep_pi_car.py, the FPS of the model.predict are so low that the pi-car front wheels turn way to late and always get out of the track. (same track working fine in opencv mode).

When I execute python3 end_to_end_lane_follower.py, the diff between the desired and the model never excide 6, so I guess the model is ok, i seem that only its execution is slow.

I am running on an Pi4 with:
Python 3.7.3
tf version 2.3.0
numpy version 1.20.3
keras version 2.4.3

Do you have any idea how to debug/fix this low FPS ?
Thanks in advance for your help

Research Paper

I'm sorry i had to comment here. But could you tell me about research papers which you referred or are related to this project.

issues with deep_pi_car.py

I followed this great tutorial step-by-step, and I really appreciate the author for making this project available to public.
I run the command python3 deep_pi_car.py and got this error:

Traceback (most recent call last):
  File "deep_pi_car.py", line 2, in <module>
    import picar
ModuleNotFoundError: No module named 'picar'

does anyone know which commands I need to install picar module?

Thanks in advance

Car running in a circle

I have followed the steps and also passed all the front wheel and back wheel tests, but I when run python3 deep_pi_car.py the car is driving in circles instead of following the lane. Does anyone run into the same issue?

When retraining your code to detect traffic signs and people, both value of AP & AR are 0

I retraining this code https://github.com/dctian/DeepPiCar/blob/master/models/object_detection/code/tensorflow_traffic_sign_detection.ipynb using TensorFlow 1.15.2, and result of AP & AR are 0.

but Loss decreased during iteration like below:

So i can not detect object & its bounding box on test images.

Output shape is always [1, 10]

closed

Issue with end_to_end_lane_navigation.ipynb

Hello David,
I have this problem when running your codes on google colab
Could you tell me why I can't mount the pw.txt file?

Running end-to-end-lane follower on TPU

Hello,
I am interesting to build and experiment, but I am missing a step....

You are training with Nvidia Model and generate a h5 file.
Then you run the follower script using this file : lane_navigation.h5

Question: I there a supplemental step I am missing to get this KERAS file running with TPU ?

Segmentation Fault

I have downloaded the edge-tpu device using the coral edge tpu website and when I run:
python3 code/coco_object_detection.py
I receive a segmentation fault

Trained network not following lane

I have got the OpenCV lane tracking to work, collected the data and trained the network but when I run the network on the car it doesn't seem to be able to follow the lane very well and keeps turning to the left out of the lane.
Everything else works fine so I'm not sure what the issue could be.
Has anyone else encountered a similar issue or have any idea what the problem could be?

Thanks

Issue with save_training_data.py

Hi, I first just want to say that your guide for the DeepPiCar has been very insightful and interesting, especially for someone who is relatively new to deep learning! I managed to get through part 4 with no hiccups, and I was ready to use my newly recorded videos to train the deep learning model. However, when I tried to run save_training_data.py ~/DeepPiCar/models/lane_navigation/images/video01, I got the error VIDIOC_REQBUFS: Inappropriate ioctl for device. I made sure that I had a .avi file in the correct directory, but it's not letting me get the still images from the video. Any help on this issue would be appreciated! Thanks!

ModuleNotFoundError: No module name 'edgetpu'

Hello,
I am trying to implement an autonomous self driving car y use of CV. When I run python3 coco_object_detection.py just to test things, I receive the following Module Error: ModuleNotFoundError: No module named 'edgetpu'. Is this because I need to have the colab accelerator? I also tried sudo apt-get install Edgetpu. How can I run this code with out EdgeTPU?

Compatibility problem for the model training

Hi everyone,

We are having a big issue with the model training Jupyter Notebook, there is a huge compatibilty problem betwin tensorflow and nuppy and all theirs depedencies, and none of their previus versions are compatibles

Does anyone has a solution for that problem ?

Thanks

help... about drive direction change.

First, I would like to thank you for sharing the materials and methods so that I can learn machine learning while studying DeepPiCar. I'm a beginner who just started studying machine learning. I'm following your guide, Part 4: Autonomy Lane Navigation via OpenCV, and running on the backward of the sunfounder's picar. To drive by forward, I searched for dip_pi_car.py and back_whels.py, and I modified it, but it didn't go well. Can you give me a hint about what needs to be revised so that Picar can proceed?

Segmentation fault

Hi all, so I tried to run python3 code/coco_object_detection.py but I got a Segmentation fault error. Here's my trace output.

 --- modulename: coco_object_detection, funcname: <genexpr>
coco_object_detection.py(28):         labels = dict((int(k), v) for k, v in pairs)
 --- modulename: coco_object_detection, funcname: <genexpr>
coco_object_detection.py(27):         pairs = (l.strip().split(maxsplit=1) for l in f.readlines())
coco_object_detection.py(31):     IM_WIDTH = 640
coco_object_detection.py(32):     IM_HEIGHT = 480
coco_object_detection.py(33):     camera = cv2.VideoCapture(0)
coco_object_detection.py(34):     ret = camera.set(3,IM_WIDTH)
coco_object_detection.py(35):     ret = camera.set(4,IM_HEIGHT)
coco_object_detection.py(37):     font = cv2.FONT_HERSHEY_SIMPLEX
coco_object_detection.py(38):     bottomLeftCornerOfText = (10,IM_HEIGHT-10)
coco_object_detection.py(39):     fontScale = 1
coco_object_detection.py(40):     fontColor = (255,255,255)  # white
coco_object_detection.py(41):     boxColor = (0,0,255)   # RED?
coco_object_detection.py(42):     boxLineWidth = 1
coco_object_detection.py(43):     lineType = 2
coco_object_detection.py(45):     annotate_text = ""
coco_object_detection.py(46):     annotate_text_time = time.time()
coco_object_detection.py(47):     time_to_show_prediction = 1.0 # ms
coco_object_detection.py(48):     min_confidence = 0.20
coco_object_detection.py(51):     engine = edgetpu.detection.engine.DetectionEngine(args.model)
 --- modulename: engine, funcname: __init__
engine.py(69):     if device_path:
engine.py(72):       super().__init__(model_path)
 --- modulename: basic_engine, funcname: __init__
basic_engine.py(36):     if device_path:
basic_engine.py(40):       self._engine = BasicEnginePythonWrapper.CreateFromFile(model_path)
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
Aborted

Any help is highly appreciated. Thanks!