Python | Smile detection using OpenCV

Smile detection is used in many fields including media where it helps companies understand public reactions to their content. Here we see how to build a simple smile detection system using OpenCV. By using pre-trained Haar cascade classifiers, we will create a real-time application that detects smiles from a live webcam feed. This approach gives us a practical introduction to facial expression recognition and it can be expanded for more complex tasks or used in different projects.

Implementation of Smile detection using OpenCV

Here we will build a simple system that detects smiles in real-time using OpenCV and Haar Cascade classifiers. The process involves capturing video from a webcam, detecting faces and checking for smiles within those faces. Let's see various steps involved in this process.

Step 1: Importing OpenCV library

We import the OpenCV library which allows us to process images, work with video streams and apply machine learning-based algorithms like Haar Cascade classifiers for face and smile detection.

import cv2

Step 2: Loading Pre-trained Classifiers

We load the pre-trained Haar Cascade classifiers for detecting faces and smiles. These classifiers are already trained to recognize these specific features in images using machine learning techniques.

face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades +'haarcascade_frontalface_default.xml')
eye_cascade = cv2.CascadeClassifier(cv2.data.haarcascades +'haarcascade_eye.xml')
smile_cascade = cv2.CascadeClassifier(cv2.data.haarcascades +'haarcascade_smile.xml')

Step 3: Starting Webcam Capture

Here we open the webcam using cv2.VideoCapture(0). This initializes the webcam and starts capturing video frames in real-time.

cap = cv2.VideoCapture(0)

Step 4: Processing Each Frame

Now we continuously capture video frames and convert them to grayscale. We then detect faces using the face_cascade and for each detected face, we check for smiles in the region around the face using smile_cascade. Rectangles are drawn around detected faces and smiles and the processed frame is displayed in real-time.

• gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY): We convert the color frame to grayscale which is needed for Haar cascade detection.
• faces = face_cascade.detectMultiScale(gray, scaleFactor=1.3, minNeighbors=5): We detect faces using the face Haar cascade classifier.
• cv2.rectangle(frame, (x, y), (x + w, y + h), (255, 0, 0), 2): We draw a blue rectangle around each detected face on the original frame.
• roi_gray = gray[y:y + h, x:x + w]: We define the region of interest (ROI) for the detected face to focus on it for smile detection.
• smiles = smile_cascade.detectMultiScale(roi_gray, scaleFactor=1.8, minNeighbors=20, minSize=(25, 25)): We detect smiles within the face ROI using the smile Haar cascade classifier.

while True:
    ret, frame = cap.read()
    
    if not ret:
        print("Failed to grab frame.")
        break

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    faces = face_cascade.detectMultiScale(gray, scaleFactor=1.3, minNeighbors=5)

    for (x, y, w, h) in faces:
        cv2.rectangle(frame, (x, y), (x + w, y + h), (255, 0, 0), 2)
        roi_gray = gray[y:y + h, x:x + w]

        smiles = smile_cascade.detectMultiScale(roi_gray, scaleFactor=1.8, minNeighbors=20, minSize=(25, 25))

        for (sx, sy, sw, sh) in smiles:
            cv2.rectangle(frame, (x + sx, y + sy), (x + sx + sw, y + sy + sh), (0, 255, 0), 2)

    cv2.imshow('Smile Detection', frame)

Step 5: Exit the Loop and Release the Webcam

We define main function in this step. After execution, the function can be terminated by pressing the "q" key. 

if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cap.release()
cv2.destroyAllWindows()

Output: 

You can download the full notebook from here.

Advantages of Smile Detection using OpenCV

1. Real-time Processing: The system allows real-time detection of faces and smiles using a webcam, making it ideal for interactive applications such as virtual meetings or social media platforms.
2. Easy Implementation with Pre-trained Classifiers: Using Haar Cascades for face and smile detection simplifies the implementation as the classifiers are pre-trained and easy to integrate, reducing the need for complex model training.
3. Low Computational Overhead: Haar Cascades are computationally efficient which makes the system lightweight and suitable for devices with limited processing power such as webcams or entry-level PCs.

Limitations of Smile Detection using OpenCV

1. Limited Accuracy in Complex Environments: The accuracy of detection might decrease in real-world conditions with poor lighting, different face angles or occlusions (like glasses or facial hair).
2. Non-Robust to Variations in Face Features: Haar cascade classifiers may struggle to detect faces and smiles accurately for different age groups, ethnicities or face shapes, as they are trained on limited datasets.
3. Performance Decline with Multiple Faces: The system can become less efficient when multiple faces are present in the frame, as the Haar cascades may not be able to accurately detect or track all faces and smiles simultaneously in crowded scenarios.