vinhngx / carnd-advanced-lane-lines Goto Github PK

The goals / steps of this project are the following:

• Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
• Apply a distortion correction to raw images.
• Use color transforms, gradients, etc., to create a thresholded binary image.
• Apply a perspective transform to rectify binary image ("birds-eye view").
• Detect lane pixels and fit to find the lane boundary.
• Determine the curvature of the lane and vehicle position with respect to center.
• Warp the detected lane boundaries back onto the original image.
• Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

You're reading it!

The code for this step is contained in the IPython notebook camera_calibration.ipynb.

I start by preparing "object points", which will be the (x, y, z) coordinates of the chessboard corners in the world. Here I am assuming the chessboard is fixed on the (x, y) plane at z=0, such that the object points are the same for each calibration image. Thus, objp is just a replicated array of coordinates, and obj_points will be appended with a copy of it every time I successfully detect all chessboard corners in a test image. img_points will be appended with the (x, y) pixel position of each of the corners in the image plane with each successful chessboard detection.

I then used the output obj_points and img_points to compute the camera calibration and distortion coefficients using the cv2.calibrateCamera() function. I applied this distortion correction to the test image using the cv2.undistort() function and obtained this result:

The pipeline for single images is detailed in the Jupyter notebook pipeline-single-image.ipynb.

To demonstrate this step, I will describe how I apply the distortion correction to one of the test images like this one:

I used a combination of color transform and gradient thresholds to generate a binary image. This step is detailed in Step 2 of the Jupyter notebook pipeline-single-image.ipynb. Here's an example of my output for this step.

The code for my perspective transform is detailed in Step 3 of the Jupyter notebook pipeline-single-image.ipynb.

The source and destination points are defined as below:

#Four source cordinates
src = np.float32(
    [[254, 700],
     [595, 449],
     [688, 448],
     [1062, 693]
    ])
#Four desired cordinates
dst = np.float32(
    [[254, 700],
     [254, 0],
     [1062, 0],
     [1061, 693]
    ])

This resulted in the following source and destination points:

I verified that my perspective transform was working as expected by drawing the src and dst points onto a test image and its warped counterpart to verify that the lines appear parallel in the warped image.

The code for finding lane lines is detailed in Step 4 of the Jupyter notebook pipeline-single-image.ipynb. We employ a sliding window and histogram based method. This step is applied on a binary warped image. The vertical dimension of the image is divided into 9 equi-height sections. The image is devided into two halves: the left lane and right lane are detected seperately. The base points for start searching the lane is the peak location of the histogram.

Starting from the bottom section of the image, for each window centred at the base point, if the number of pixel is larger than the threshold, they are determined to form part of the respective lane lines. Finally, second order polinomials are fitted to the left and right lanes seperately. The outcome of this step for the above test sample is demonstrated in the below image:

The code for calculating road curvature and position of the vehicle is detailed in Step 5 of the Jupyter notebook pipeline-single-image.ipynb.

Step 6 of the Jupyter notebook pipeline-single-image.ipynb demonstrates the final lane area detection result super-imposed on the original image.

The code for processing videos is contained within the Jupyter notebook pipeline-video.ipynb.

Here's a link to my video result

The approach carried out in this project performs quite well on the project video. However, it did not perform that well on the more chalenging videos (see, for example, challenge_video_proccessed.mp4).

In order to improve the lane detection results, I have tried the following alternatives (detailed in pipeline-video-alternative.ipynb):

• Define a region of interest using the approach in the first lane detection project:

#Define a region of interest
vertices = np.array([[(0,Y),(0.45*X, 0.55*Y), (0.55*X, 0.55*Y), (X,Y)]], dtype=np.int32)
selected_img = region_of_interest(result, vertices)

• Combine hough transform for lane line detection with a window based approach

• Memorize the base location of the lanes in the current frame and use that as a starting search point for the next frame. This alternative approach is implemented in the notebook "pipeline-video-alternative.ipynb".

• Combining Hough transform with histogram based approach (pipeline-single-image-Hough.ipynb)

Despite these modification, it is still challenging to detect lanes in the challenging videos.

The approach presented in this project relies heavily on a hand-designed pipeline, with expert understanding of the color space, and some heuristical rules (such as binary thresholding, histogram threshold to detect lane pixels). Learning-based approaches using human-annotated lane lines might improve the existing approach, as well as resolving corner cases (such as left lane appearing in the right-half of the image (due to vehicle veering off road).

The approach carried out in the first submission, although doing well on most of the frames, seriously mis-identified the lanes in several frames which could potentially lead to dangerous driving actions. In the 2nd attempt, we trialed several approaches, detailed in the Jupyter notebook pipeline-video-alternative.ipynb.

(1) - Plotting the intermediate lane detection step side-by-side with the final detection result to help debugging. This visualization reveals that the lane detection pipeline failed where there are trees casting heavy shades onto the road, making a large patch of pixels in the binary image.

(2) - Take derivative in the S channel instead:

sobelx = cv2.Sobel(s_channel, cv2.CV_64F, 1, 0) # Take the derivative in x

This make the left lane (white color) a bit more unstable. Thus we did not adopt this approach.

(3) - Sanity check to see if the lanes detected in the current frame deviates significantly from the previously detected lanes (by at most 0.5m).

def sane_lane(left_fitx, right_fitx, left_fit, right_fit, THRESHOLD=700/3.7/2):
    global sane_left
    global sane_right
    global PREV_left_fit
    global PREV_right_fit
    
    violation = False
    if np.average(np.abs(left_fitx - sane_left)) < THRESHOLD:
        sane_left = left_fitx
        PREV_left_fit = left_fit
    else: 
        violation = True
        #lane deviates too much from the previous frame, keep previous detection   
        left_fit = PREV_left_fit
    if np.average(np.abs(right_fitx - sane_right)) < THRESHOLD:
        sane_right = right_fitx
        PREV_right_fit = right_fit
    else: 
        violation = True
        #lane deviates too much from the previous frame, keep previous detection   
        right_fit = PREV_right_fit
    return violation, left_fit, right_fit

(4) - Smoothing out curve fitting, by merging the lane pixels detected in the previous frame with the current frame.

In the end, we found that combining (1) and (4) works well in improving the lane detection results, presented in project_video_proccessed_2nd.mp4)