Face Recognition

Detect and search for similar faces in images and videos

IPED includes advanced face recognition capabilities powered by the Face Recognition library, enabling detection of faces in images and videos, and similarity-based face search across the entire case.

Overview

The face recognition feature provides:

• Face Detection - Automatically locate faces in images and video frames
• Face Encoding - Generate 128-dimensional feature vectors for each face
• Similar Face Search - Find faces similar to a reference face across the case
• Video Support - Process video thumbnails for face detection
• GPU Optimization - Optimized to run efficiently without GPU requirements

Technology

IPED uses the Face Recognition library built on dlib’s state-of-the-art face recognition:

# Python face recognition based on Face Recognition Project
# https://pypi.org/project/face-recognition/

Key components:

• dlib - C++ machine learning library
• face_recognition - Python wrapper with simple API
• OpenCV - Image processing operations
• NumPy - Numerical computing for feature vectors

Configuration

Face recognition is configured in FaceRecognitionConfig.txt:

# Enable face recognition processing
enableFaceRecognition = true

# Number of parallel face recognition processes
numFaceRecognitionProcesses = 2

# Maximum image dimension (larger images downscaled)
maxResolution = 1024

# Face detection model: 'hog' (faster, CPU) or 'cnn' (accurate, GPU)
faceDetectionModel = hog

# Upsampling factor (higher = detect smaller faces, slower)
upSampling = 1

# Minimum face size in pixels (smaller faces ignored)
minSize = 48

Face Detection Models

HOG (Histogram of Oriented Gradients)

Recommended for CPU processing

detection_model = 'hog'

Characteristics:

• Fast execution on CPU
• Good accuracy for frontal faces
• Lower memory usage
• Handles standard lighting conditions

Performance: ~200-300ms per image on modern CPU

CNN (Convolutional Neural Network)

Recommended for GPU processing

detection_model = 'cnn'

Characteristics:

• Higher accuracy
• Better with various angles and lighting
• Detects partially occluded faces
• Requires GPU for acceptable speed

Performance: ~50-100ms per image on GPU, ~2-3s on CPU

Implementation

The face recognition task is implemented as a Python script integrated into IPED’s processing pipeline:

Initialization

class FaceRecognitionTask:
    def init(self, configuration):
        # Check required modules
        import face_recognition
        import cv2
        import numpy as np
        
        # Verify numpy version (<2.x)
        if np.__version__ >= '2':
            logger.error('numpy version must be <2.x')
            return
        
        # Create external process pool
        createProcessQueue()

Processing Flow

def process(self, item):
    # 1. Filter items
    if not item.getExtraAttribute('hasThumb'):
        return  # Skip non-image items
    
    # 2. Check minimum size
    width = item.getMetadata().get('Width')
    height = item.getMetadata().get('Height')
    if width < min_size or height < min_size:
        return  # Skip small images
    
    # 3. Get or create external process
    proc = processQueue.get(block=True)
    
    # 4. Send image path to process
    print(img_path, file=proc.stdin, flush=True)
    
    # 5. Read face detection results
    num_faces = int(proc.stdout.readline())
    
    # 6. Read face locations
    for i in range(num_faces):
        location = eval(proc.stdout.readline())
        face_locations.append(location)
    
    # 7. Read face encodings (128 values per face)
    for i in range(num_faces):
        encoding = []
        for j in range(128):
            encoding.append(float(proc.stdout.readline()))
        face_encodings.append(encoding)
    
    # 8. Store results
    item.setExtraAttribute('face_count', num_faces)
    item.setExtraAttribute('face_locations', face_locations)
    item.setExtraAttribute('face_encodings', face_encodings)

Face Attributes

Detected faces stored as item extra attributes:

face_count

Number of faces detected in the image.

Integer count = item.getExtraAttribute(ExtraProperties.FACE_COUNT);

face_locations

Bounding box coordinates for each face (top, right, bottom, left).

List<List<Integer>> locations = 
    item.getExtraAttribute(ExtraProperties.FACE_LOCATIONS);

face_encodings

128-dimensional feature vectors for similarity comparison.

List<float[]> encodings = 
    item.getExtraAttribute(ExtraProperties.FACE_ENCODINGS);

External Process Architecture

IPED uses external Python processes for face recognition:

Benefits

1. Stability - Crashes don’t affect main IPED process
2. Parallelism - Multiple processes for concurrent processing
3. Resource isolation - Memory leaks contained
4. Process reuse - Avoid model reload overhead

Process Communication

Communication via stdin/stdout:

# Send image path
print(image_path, file=process.stdin)

# Receive results
num_faces = int(process.stdout.readline())

Health monitoring:

• Ping/pong mechanism verifies process alive
• Automatic restart if process crashes
• Error logging for debugging

Similar Face Search

The analysis interface provides face search functionality:

From Image Viewer

1. Right-click on detected face
2. Select “Search Similar Faces”
3. Adjust similarity threshold slider
4. View results ranked by similarity

From External Image

1. Tools menu → Face Search
2. Select reference image containing face
3. Set similarity threshold
4. Search entire case for matches

Similarity Calculation

Based on Euclidean distance between 128D encodings:

import numpy as np

def face_distance(face_encoding1, face_encoding2):
    return np.linalg.norm(face_encoding1 - face_encoding2)

# Typical threshold: 0.6
# Lower distance = more similar

Threshold guidelines:

• 0.3 - Very strict, same person in similar conditions
• 0.4 - Strict, likely same person
• 0.6 - Default, good balance (recommended)
• 0.8 - Relaxed, may include similar-looking people

Performance Optimization

Image Downscaling

Large images automatically downscaled:

max_size = 1024  # Configurable
if width > max_size or height > max_size:
    # Downscale maintaining aspect ratio
    scale = max_size / max(width, height)
    new_width = int(width * scale)
    new_height = int(height * scale)

Benefits:

• Faster processing
• Reduced memory usage
• Minimal accuracy impact

Process Pooling

maxProcesses = int(max(1, numThreads / 2))
processQueue = queue.Queue(maxProcesses)

• Reuses Python processes
• Avoids model reload overhead
• Balances parallelism vs. memory

Result Caching

cache[hash + '_locations'] = locations
cache[hash + '_encodings'] = encodings
cache[hash + '_count'] = count

• Duplicate files processed once
• In-memory cache for fast access
• Significant speedup for duplicates

Video Support

Face recognition works on video files via thumbnails:

if mediaType.startswith('video') and videoSubitems:
    # Process video thumbnail
    img_path = previewManager.readPreview(item)

Configuration:

• Requires video thumbnail generation enabled
• Processes representative frame
• Detects faces in thumbnail image

Installation Requirements

Required Python packages:

pip install face-recognition
pip install opencv-python  
pip install numpy<2.0

System dependencies:

• CMake - For compiling dlib
• C++ compiler - GCC/Clang or Visual Studio
• CUDA (optional) - For GPU acceleration

Detailed setup: IPED User Manual - FaceRecognition

Use Cases

Person of Interest Search

Find all images containing a specific individual:

1. Identify reference image with target person
2. Run similarity search
3. Review ranked results
4. Bookmark matches for export

Victim Identification

Locate victims in CSAM investigations:

• Compare against known victim database
• Identify co-occurrences
• Build victimology timeline

Suspect Association

Map relationships:

• Find images showing multiple suspects together
• Identify unknown associates
• Build network diagrams

Social Media Analysis

Analyze social media content:

• Identify persons in photos
• Track presence across platforms
• Timeline of appearances

Limitations

Detection Challenges

• Extreme angles - Profile views harder than frontal
• Poor lighting - Underexposed or overexposed faces
• Occlusion - Masks, sunglasses reduce accuracy
• Low resolution - Faces smaller than minSize ignored
• Motion blur - Blurry faces may not detect

Search Limitations

• Age progression - Significant aging affects similarity
• Facial hair - Beards/mustaches impact matching
• Cosmetics - Heavy makeup changes features
• Image quality - Low quality reduces accuracy

Troubleshooting

No Faces Detected

• Check minimum image size (width/height > minSize)
• Verify faces are clearly visible
• Try increasing upSampling parameter
• Switch to ‘cnn’ model for better detection

Slow Processing

• Reduce maxResolution for faster processing
• Use ‘hog’ instead of ‘cnn’ model
• Decrease upSampling value
• Increase numFaceRecognitionProcesses

High Memory Usage

• Reduce numFaceRecognitionProcesses
• Lower maxResolution
• Process videos as subitems to reduce memory

Process Crashes

• Check Python environment compatibility
• Verify numpy version less than 2.0
• Review error logs for stack traces
• Restart with fewer parallel processes