Hello,
I'm not understanding how you are creating labels and what's inside in labels.p file as you are dividing labels by 255.Why?

Normalize labels - training images get normalized to start in the network

labels = labels / 255
Please guide me in this.
Thank You!

Hi,

Is there any related paper or documentation for the neural network architecture you build? If yes, please share the file or the link with me.

Thanks

where do I get scaler.p from to reverse label normalization?

The current way the dataset is loaded for training is super inefficient and loads the whole dataset all at once. As such, I should consider changing the dataset from being stored as a pickle file, as well as whether to use flow_from_directory or similar techniques.

1# I got an error of different input types on add weighted method so I've to specify an output type

before change:
result = cv2.addWeighted(image, 1, lane_image, 1, 0)
after change:
result = cv2.addWeighted(image, 1, lane_image, 1, 0, dtype= cv2.CV_32F).astype(np.uint8)

now there are no lanes on mn output video
2# and also I've used "from skimage.transform import resize as imresize" for image resize
but I didn't change any other lines related to imresize is it ok?
i.e lane_image = imresize(lane_drawn, (720, 1280, 3))
its same before and after

How can I draw lines from "Images with coefficient labels" dataset on image?

Do u have any evaluation code to evaluate the accuracy of the model？Can u share the code?

How can I get the annotation tools that you used?
And why annotation G value in RGB is not 255?

I drew coefficient labels as curves on the downsized pictures and I found that curves can not match lanes. What are these labels related to?

how to make labels without that redrawn lanes ? or how to get those redrawn lanes
can we change the final output to be a convolution and not a deconv and what should the output be

When I run conda env create -f lane_environment.yml I get the following error: Collecting actionlib==1.11.7 Could not find a version that satisfies the requirement actionlib==1.11.7

If actionlib is required for this, can you maybe update the environment file with another compatible version?

Thanks

I got an error after i had run python3 MLND-Capstone/draw_detected_lanes.py:

Traceback (most recent call last):
File "MLND-Capstone/draw_detected_lanes.py", line 9, in
model = load_model('full_CNN_model.h5')
File "/usr/lib/python3.7/site-packages/keras/engine/saving.py", line 458, in load_wrapper
return load_function(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/keras/engine/saving.py", line 549, in load_model
with H5Dict(filepath, mode='r') as h5dict:
File "/usr/lib/python3.7/site-packages/keras/utils/io_utils.py", line 191, in init
self.data = h5py.File(path, mode=mode)
File "/usr/lib/python3.7/site-packages/h5py/_hl/files.py", line 394, in init
swmr=swmr)
File "/usr/lib/python3.7/site-packages/h5py/_hl/files.py", line 170, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 85, in h5py.h5f.open
OSError: Unable to open file (unable to open file: name = 'full_CNN_model.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

Hi Michael. Thank you so much for this intensive work! I wonder in your final method, is there any way that could calculate the offset of the car from lanes, comparing with the method that uses perspective transformation?

when i run the check_labels.py in line 36 the issue is ValueError: operands could not be broadcast together with shapes and one more question what is list index out of range. Can you help me pls?

Hii
I tried implementating the code with another video and got this error.

I tried to run this lane_lines.py and got the following error:

Traceback (most recent call last):
File "lane_lines.py", line 26, in
label_scaler = pickle.load(open( "scaler.p", "rb" ))
FileNotFoundError: [Errno 2] No such file or directory: 'scaler.p'

Hi,
I use python3.5 to run fully_conv_NN.py, but got errors as follows, could you please give me some idea? I do not change any source code and use the original datasets download from dropbox.

1. python version: 3.5
   Python 3.5.2 (default, Nov 23 2017, 16:37:01)
   [GCC 5.4.0 20160609] on linux
   Type "help", "copyright", "credits" or "license" for more information.

2. git branch -a
   early_steps

• master
  remotes/origin/HEAD -> origin/master
  remotes/origin/early_steps
  remotes/origin/master

#git log .
commit 22b81a9
Author: mvirgo [email protected]
Date: Sat Dec 9 17:43:27 2017 -0800

feat: modified architecture, condense model save/load

#python fully_conv_NN.py
/usr/local/lib/python3.5/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
2018-02-12 04:51:01.135837: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
Epoch 1/10
90/89 [==============================] - 465s 5s/step - loss: 0.0514 - val_loss: 0.0216
Epoch 2/10
90/89 [==============================] - 447s 5s/step - loss: 0.0162 - val_loss: 0.0132
Epoch 3/10
90/89 [==============================] - 471s 5s/step - loss: 0.0122 - val_loss: 0.0103
Epoch 4/10
90/89 [==============================] - 667s 7s/step - loss: 0.0104 - val_loss: 0.0092
Epoch 5/10
90/89 [==============================] - 531s 6s/step - loss: 0.0092 - val_loss: 0.0082
Epoch 6/10
90/89 [==============================] - 651s 7s/step - loss: 0.0087 - val_loss: 0.0080
Epoch 7/10
90/89 [==============================] - 682s 8s/step - loss: 0.0083 - val_loss: 0.0078
Epoch 8/10
90/89 [==============================] - 451s 5s/step - loss: 0.0080 - val_loss: 0.0074
Epoch 9/10
90/89 [==============================] - 655s 7s/step - loss: 0.0078 - val_loss: 0.0073
Epoch 10/10
90/89 [==============================] - 545s 6s/step - loss: 0.0076 - val_loss: 0.0077

Layer (type) Output Shape Param #

batch_normalization_1 (Batch (None, 80, 160, 3) 12

Conv1 (Conv2D) (None, 78, 158, 8) 224

Conv2 (Conv2D) (None, 76, 156, 16) 1168

max_pooling2d_1 (MaxPooling2 (None, 38, 78, 16) 0

Conv3 (Conv2D) (None, 36, 76, 16) 2320

dropout_1 (Dropout) (None, 36, 76, 16) 0

Conv4 (Conv2D) (None, 34, 74, 32) 4640

dropout_2 (Dropout) (None, 34, 74, 32) 0

Conv5 (Conv2D) (None, 32, 72, 32) 9248

dropout_3 (Dropout) (None, 32, 72, 32) 0

max_pooling2d_2 (MaxPooling2 (None, 16, 36, 32) 0

Conv6 (Conv2D) (None, 14, 34, 64) 18496

dropout_4 (Dropout) (None, 14, 34, 64) 0

Conv7 (Conv2D) (None, 12, 32, 64) 36928

dropout_5 (Dropout) (None, 12, 32, 64) 0

max_pooling2d_3 (MaxPooling2 (None, 6, 16, 64) 0

up_sampling2d_1 (UpSampling2 (None, 12, 32, 64) 0

Deconv1 (Conv2DTranspose) (None, 14, 34, 64) 36928

dropout_6 (Dropout) (None, 14, 34, 64) 0

Deconv2 (Conv2DTranspose) (None, 16, 36, 64) 36928

dropout_7 (Dropout) (None, 16, 36, 64) 0

up_sampling2d_2 (UpSampling2 (None, 32, 72, 64) 0

Deconv3 (Conv2DTranspose) (None, 34, 74, 32) 18464

dropout_8 (Dropout) (None, 34, 74, 32) 0

Deconv4 (Conv2DTranspose) (None, 36, 76, 32) 9248

dropout_9 (Dropout) (None, 36, 76, 32) 0

Deconv5 (Conv2DTranspose) (None, 38, 78, 16) 4624

dropout_10 (Dropout) (None, 38, 78, 16) 0

up_sampling2d_3 (UpSampling2 (None, 76, 156, 16) 0

Deconv6 (Conv2DTranspose) (None, 78, 158, 16) 2320

Final (Conv2DTranspose) (None, 80, 160, 1) 145

Total params: 181,693
Trainable params: 0
Non-trainable params: 181,693

Exception ignored in: <bound method BaseSession.del of <tensorflow.python.client.session.Session object at 0x7f488a682e48>>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 702, in del
TypeError: 'NoneType' object is not callable

When I run draw_detected_lanes.py, I always get an error, the error is as follows, how can I solve it? thank you very much!

can someone plz explain the outputs of the cnn what is that array

Can u share the orignal training images and lables instead of in format of .p file?

Hi,
Iam facing following issues:

1.I am not able to open the full_CNN_train.p and CNN_label.p in Matlab.
2.How can I train and label my own dataset using this code?

Please can you guide where I am doing wrong

I use this code to get output file which is a road with detected path according to detected lines:

from os.path import splitext, basename

import numpy as np
import cv2

from moviepy.editor import VideoFileClip
from keras.models import load_model

model = load_model('full_CNN_model.h5')

class Lanes():
def init(self):
self.recent_fit = []
self.avg_fit = []

lanes = Lanes()
vid_in = "line_1.mp4"
vid_output = 'line_1_out.mp4'

video_capture = cv2.VideoCapture()
fourcc = cv2.VideoWriter_fourcc(*'XVID')
name, extension = splitext(basename(vid_in))
output = name + '.avi'

if video_capture.open(vid_in):
property_id = int(cv2.CAP_PROP_FRAME_COUNT)
total_frames = int(cv2.VideoCapture.get(video_capture, property_id))
frame_no = 1
width, height = int(video_capture.get(cv2.CAP_PROP_FRAME_WIDTH)),
int(video_capture.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = video_capture.get(cv2.CAP_PROP_FPS)

video_writer = cv2.VideoWriter(output, fourcc, fps, (width, height))
while video_capture.isOpened():
    ret, frame = video_capture.read()
    if not ret:
        break
    
    small_img = np.array(np.resize(frame, (80, 160, 3)))
    small_img = small_img[None,:,:,:]
    
    prediction = model.predict(small_img)[0] * 255

    lanes.recent_fit.append(prediction)
    if len(lanes.recent_fit) > 5:
        lanes.recent_fit = lanes.recent_fit[1:]

    lanes.avg_fit = np.mean(np.array([i for i in lanes.recent_fit]), axis = 0)

    blanks = np.zeros_like(lanes.avg_fit).astype(np.uint8)
    lane_drawn = np.dstack((blanks, lanes.avg_fit, blanks))
    lane_image = np.resize(lane_drawn, (720, 1280, 3))

    result = cv2.addWeighted(lane_image, 1, lane_image, 1, 0)


    if frame_no % 20 == 0:
        print('# {}/{} frames processed!'.format(frame_no, total_frames))

    frame_no += 1
    video_writer.write(frame)
video_capture.release()
video_writer.release()

I also used the code in the original repository, but I did not get the same output in repository, Although I use the same steps.

Do I have to train another model now?

1. How did you create the labels file? I have created a program to output the coefficients of the lane fit, but i am not sure how to append them to a labels file. Are they just simply added to a labels.p file or do we have to append the image name and then the coefficients.
2. Also were the coefficients of the lane calculated from the resized 80*160 image or on the normal image
   size.

Please post the training loss graph and validation loss graph for your trained model. I am using as a reference for my dissertation. It will be very helpful.

Well-done job. I'm longing to know something about samples preparation.
Could you show some original images and labels which were not converted to *.p format?
Furthermore, How to convert my data to *.p format?
THX.

Will edit the code to add the capability for it to use CUDA GPU (As I am trying to use this as a benchmark)

how i can modify .p file acording to my need.

Hi @mvirgo ,

I am curios how you make the label like below,

I have read the report you post here.

in the Algorithms and Techniques section, you said

Next, I must process the images for use in making labels, for which I will use the same techniques I previously used in my computer vision-based model

But I can't find the previous model you used.

In the reflection part, you said

I then created six coefficient labels, three each for both lane lines. Next, I created a program to
make those labels into re-drawn lanes

then I look into make_labels.py, you define load_drawn_images,

def load_drawn_images():
    """Load re-drawn lane image locations"""
    drawn_image_locs = glob.glob('draw/*.jpg')
    sort_drawn_image_locs = sorted(drawn_image_locs, key=natural_key)
    return sort_drawn_image_locs    

I don't know how to make re-drawn images.

Can you show an example about labeling the image step by step and which python files you used?

thanks

Very well documented project. Thanks a lot for this detail. You have mentioned all about software. I have few doubts regarding hardware requirement for this project?

1. Which GPU/Cloud platform you have used for this project?
2. Is it possible to run this neural network on Laptop with 4GB RAM and 240GB SSD (intel i3)?
3. How much time did you get to train all images?

Dear @mvirgo ,
Thanks for creating such an amazing project! I am sure that I will learn from it.
For now, I am using this as a benchmark for my new PC and I was wandering where the file 'Full_CNN_train.p' is as it is giving me a File Not Found error.

I simply cloned the github repo. by .p, do you mean .py? Many thanks 👍

Clement
Secondary School Student :)

Hello. First, cong. Awesome project, I was wanting to learn deep learning with making lane detection. I can learn now but how can I control the motors sir? How we can calculate the lanes? I have idea with using Tan90=Sin90/Cos90 like that.

Have a nice day!

hello!
In the projuct, you said that the neural network was used to directly output the road line.However the output was the line area. I am confused.
Have a nice day!

I am getting this following error when I run the fully_conv_NN.py file.

Traceback (most recent call last):
File "fully_conv_NN.py", line 27, in
train_images = pickle.load(train)
File "C:\Users\User\Anaconda3\envs\py3\lib\pickle.py", line 1384, in load
return Unpickler(file).load()
File "C:\Users\User\Anaconda3\envs\py3\lib\pickle.py", line 864, in load
dispatchkey
File "C:\Users\User\Anaconda3\envs\py3\lib\pickle.py", line 892, in load_proto

raise ValueError, "unsupported pickle protocol: %d" % proto

ValueError: unsupported pickle protocol: 3

Could you help me with this issue?
Thanks

Scipy deprecated imresize as of 1.3.0 and now suggests using PIL to resize images. This will require both updating code as well as the environment file to eventually remove scipy and replace it with pillow.

Can also consider using alternative image re-sizing functions from other libraries as well.

the .p dataset is just like a .jpg?how can i see this picture?thanks !

Can you provide images of dataset?

I Open the training dataset for fully convolutional networks and the lables on dropbox. But the downloading failed everytime.