人脸检测（二）

古城微笑少年丶 2022-06-16 05:36 338阅读 0赞

**2.级联CNN人脸检测方法**  
采用级联网络来进行人脸检测，参考2015年CVPR上的一篇论文A Convolution Neural Network Cascade for Face Detection，它采用了12-net，24-net，48-net级联网络用于人脸检测，12-calibration-net，24-calibration，48-calibration边界校订网络用于更好的定位人脸框。它最小能够检测12x12大小的人脸，相比于单个CNN的人脸检测方法，大大加快了人脸检测的速度，并提高了人脸框的准确度，人脸检测的准确率和召回率也很高，在FDDB数据集上达到了当时最高的分数。

![这里写图片描述][SouthEast]

[论文下载地址][Link 1]  
[github开源代码][github]  
[github开源模型][github 1]

作者的代码写的非常优美易懂，注释简洁明了，其级联CNN人脸检测的基本思路如下：  
![这里写图片描述][SouthEast 1]  
（1）12-net：首先用一个输入为12 x 12图像的小网络来训练人脸非人脸二分类器，将最后的全连接层修改成卷积层，这样的全卷积网络即12-full-conv-net就可以接受任意大小的输入图像，输出的特征图表示对应感受野区域属于人脸的概率。在检测时，假设要检测的最小人连尺寸为K x K，例如40 x 40，将待检测的图像缩放到原来的12/K，然后将整幅图像输入到训练好的12 x 12的全卷积网络，得到特征图，设定阈值过滤掉得分低的，这样就可以去除绝大部分不感兴趣区域，保留一定数量的候选框。  
（2）12-calibration-net：训练一个输入为12 x 12图像的校订网络，来矫正上一步12-net得到的人脸框边界，它实质上是一个45类分类器，判断当前图像中包含的人脸是左右偏了、上下偏了还是大了小了，即包括：  
x方向上-0.17、0、0.17共3种平移  
y方向上-0.17、0、0.17共3三种平移  
以及0.83、0.91、1.0、1.1、1.21共5种缩放尺度  
检测时，将上一步12-net得到的所有人脸候选框作为12-calibration-net的输入，根据其分类结果对候选框位置进行校订。  
（3）localNMS：对上述12-calibration-net矫正后的人脸候选框做局部非极大值抑制，过滤掉其中重叠的得分较低的候选框，保留得分更高的人脸候选框。  
![这里写图片描述][SouthEast 2]  
（4）24-net：训练输入为24 x 24图像的人脸分类器网络。测试时，以上一步localNMS得到的人脸候选框缩放到24 x 24大小，作为24-net网络输入，判定是否属于人脸，设置阈值保留得分较高的候选框。  
（5）24-calibration-net：同样训练一个输入图像为24 x 24大小的边界校订分类网络，来矫正上一步24-net保留的人脸候选框的位置，候选框区域图像缩放到24 x 24大小，其它与12-calibration-net一致。  
（6）localNMS：将24-calibration-net矫正后的人脸候选框进行局部非极大值抑制，过滤掉重叠的得分较低的候选框，保留得分更高的。  
![这里写图片描述][SouthEast 3]  
（7）48-net：训练一个更加准确的输入为48 x 48的人脸非人脸分类器。测试时，将上一步localNMS得到的人脸候选框缩放到48 x 48大小，作为48-net输入，保留得分高的人脸候选框。  
（8）globalNMS：将48-net得到的所有人脸候选框进行全局非极大值抑制，保留所有的最佳人脸框  
（9）48-calibration-net：训练一个输入为48 x 48的边界校订分类网络。测试时，将globalNMS得到的最佳人脸框缩放到48 x 48作为输入进行人脸框边界校订。

因此，我们需要先单独训练人脸非人脸二分类12-net，24-net，48-net网络，以及人脸框边界校订12-calibration-net，24-calibration，48-calibration网络。测试时，用图像金字塔来做多尺度人脸检测，对于任意的输入图像，依次经过12-net(12-full-conv-net) -> 12-calibration-net -> localNMS -> 24-net -> 24-calibration-net -> localNMS -> 48-net -> globalNMS -> 48-calibration-net，得到最终的人脸框作为检测结果。  
其中localNMS和globalNMS(Non-maximum Suppression,NMS,非极大值抑制)的区别主要在于前者仅使用了IoU(Intersection over Union)，即交集与并集的比值，而后者还用到了IoM(Intersection over Min-area)，即交集与两者中最小面积的比值，来过滤重叠的候选框。  
博主没有自己训练，而是直接用了作者训练好的模型。使用CNN\_face\_detection-master\\face\_detection文件夹下的face\_cascade\_fullconv\_single\_crop\_single\_image.py脚本可以对但张图像进行测试，代码如下，其中需要注意的是，我们可以根据需求来设置检测最小的人脸尺寸min\_face\_size，它与检测速度直接相关，如果我们需要检测的人脸比较大，例如在128x128以上时，检测可以达到实时水平。

import numpy as np
    import cv2
    import time
    import os
    #from operator import itemgetter
    from load_model_functions import *
    from face_detection_functions import *
    
    # ==================  caffe  ======================================
    #caffe_root = '/home/anson/caffe-master/'  # this file is expected to be in {caffe_root}/examples
    import sys
    #sys.path.insert(0, caffe_root + 'python')
    import caffe
    # ==================  load models  ======================================
    net_12c_full_conv, net_12_cal, net_24c, net_24_cal, net_48c, net_48_cal = \
        load_face_models(loadNet=True)
    
    nets = (net_12c_full_conv, net_12_cal, net_24c, net_24_cal, net_48c, net_48_cal)
    
    start_time = time.time()
    
    read_img_name = 'C:/Users/Administrator/Desktop/caffe/matlab/demo/1.jpg'
    img = cv2.imread(read_img_name)     # BGR
    
    print img.shape
    
    min_face_size = 48 #最小的人脸检测尺寸，设置的越小能检测到更小人脸的同时，速度下降的很快
    stride = 5 #步长，实际并未使用
    
    # caffe_image = np.true_divide(img, 255)      # convert to caffe style (0~1 BGR)
    # caffe_image = caffe_image[:, :, (2, 1, 0)]
    img_forward = np.array(img, dtype=np.float32)
    img_forward -= np.array((104, 117, 123))
    
    rectangles = detect_faces_net(nets, img_forward, min_face_size, stride, True, 2, 0.05)
    for rectangle in rectangles:    # draw rectangles
            cv2.rectangle(img, (rectangle[0], rectangle[1]), (rectangle[2], rectangle[3]), (255, 0, 0), 2)
    
    end_time = time.time()
    print 'aver_time = ',(end_time-start_time)*1000,'ms'
    
    cv2.imshow('test img', img)
    cv2.waitKey(0)
    cv2.destroyAllWindows()

上述脚本文件中的detect\_faces\_net()函数在CNN\_face\_detection-master\\face\_detection文件夹下的face\_detection\_functions.py脚本中实现，face\_detection\_functions.py是整个级联网络的检测过程脚本，写的非常优雅，思路和注释清晰，本人针对自己需求进行了修改，代码如下：

import numpy as np
    import cv2
    import time
    from operator import itemgetter
    # ==================  caffe  ======================================
    #caffe_root = '/home/anson/caffe-master/'  # this file is expected to be in {caffe_root}/examples
    import sys
    #sys.path.insert(0, caffe_root + 'python')
    import caffe
    
    def find_initial_scale(net_kind, min_face_size):
        '''
        :param net_kind: what kind of net (12, 24, or 48)
        :param min_face_size: minimum face size
        :return:    returns scale factor
        '''
        return float(min_face_size) / net_kind
    def resize_image(img, scale):
        '''
        :param img: original img
        :param scale: scale factor
        :return:    resized image
        '''
        height, width, channels = img.shape
        new_height = int(height / scale)     # resized new height
        new_width = int(width / scale)       # resized new width
        new_dim = (new_width, new_height)
        img_resized = cv2.resize(img, new_dim)      # resized image
        return img_resized
    def draw_rectangle(net_kind, img, face):
        '''
        :param net_kind: what kind of net (12, 24, or 48)
        :param img: image to draw on
        :param face: # list of info. in format [x, y, scale]
        :return:    nothing
        '''
        x = face[0]
        y = face[1]
        scale = face[2]
        original_x = int(x * scale)      # corresponding x and y at original image
        original_y = int(y * scale)
        original_x_br = int(x * scale + net_kind * scale)    # bottom right x and y
        original_y_br = int(y * scale + net_kind * scale)
        cv2.rectangle(img, (original_x, original_y), (original_x_br, original_y_br), (255,0,0), 2)
    def IoU(rect_1, rect_2):
        '''
        :param rect_1: list in format [x11, y11, x12, y12, confidence, current_scale]
        :param rect_2:  list in format [x21, y21, x22, y22, confidence, current_scale]
        :return:    returns IoU ratio (intersection over union) of two rectangles
        '''
        x11 = rect_1[0]    # first rectangle top left x
        y11 = rect_1[1]    # first rectangle top left y
        x12 = rect_1[2]    # first rectangle bottom right x
        y12 = rect_1[3]    # first rectangle bottom right y
        x21 = rect_2[0]    # second rectangle top left x
        y21 = rect_2[1]    # second rectangle top left y
        x22 = rect_2[2]    # second rectangle bottom right x
        y22 = rect_2[3]    # second rectangle bottom right y
        x_overlap = max(0, min(x12,x22) -max(x11,x21))
        y_overlap = max(0, min(y12,y22) -max(y11,y21))
        intersection = x_overlap * y_overlap
        union = (x12-x11) * (y12-y11) + (x22-x21) * (y22-y21) - intersection
        return float(intersection) / union
    def IoM(rect_1, rect_2):
        '''
        :param rect_1: list in format [x11, y11, x12, y12, confidence, current_scale]
        :param rect_2:  list in format [x21, y21, x22, y22, confidence, current_scale]
        :return:    returns IoM ratio (intersection over min-area) of two rectangles
        '''
        x11 = rect_1[0]    # first rectangle top left x
        y11 = rect_1[1]    # first rectangle top left y
        x12 = rect_1[2]    # first rectangle bottom right x
        y12 = rect_1[3]    # first rectangle bottom right y
        x21 = rect_2[0]    # second rectangle top left x
        y21 = rect_2[1]    # second rectangle top left y
        x22 = rect_2[2]    # second rectangle bottom right x
        y22 = rect_2[3]    # second rectangle bottom right y
        x_overlap = max(0, min(x12,x22) -max(x11,x21))
        y_overlap = max(0, min(y12,y22) -max(y11,y21))
        intersection = x_overlap * y_overlap
        rect1_area = (y12 - y11) * (x12 - x11)
        rect2_area = (y22 - y21) * (x22 - x21)
        min_area = min(rect1_area, rect2_area)
        return float(intersection) / min_area
    def localNMS(rectangles):
        '''
        :param rectangles:  list of rectangles, which are lists in format [x11, y11, x12, y12, confidence, current_scale],
                            sorted from highest confidence to smallest
        :return:    list of rectangles after local NMS
        '''
        result_rectangles = rectangles[:]  # list to return
        number_of_rects = len(result_rectangles)
        threshold = 0.3     # threshold of IoU of two rectangles
        cur_rect = 0
        while cur_rect < number_of_rects - 1:     # start from first element to second last element
            rects_to_compare = number_of_rects - cur_rect - 1      # elements after current element to compare
            cur_rect_to_compare = cur_rect + 1    # start comparing with element after current
            while rects_to_compare > 0:      # while there is at least one element after current to compare
                if (IoU(result_rectangles[cur_rect], result_rectangles[cur_rect_to_compare]) >= threshold) \
                        and (result_rectangles[cur_rect][5] == result_rectangles[cur_rect_to_compare][5]):  # scale is same
    
                    del result_rectangles[cur_rect_to_compare]      # delete the rectangle
                    number_of_rects -= 1
                else:
                    cur_rect_to_compare += 1    # skip to next rectangle
                rects_to_compare -= 1
            cur_rect += 1   # finished comparing for current rectangle
    
        return result_rectangles
    def globalNMS(rectangles):
        '''
        :param rectangles:  list of rectangles, which are lists in format [x11, y11, x12, y12, confidence, current_scale],
                            sorted from highest confidence to smallest
        :return:    list of rectangles after global NMS
        '''
        result_rectangles = rectangles[:]  # list to return
        number_of_rects = len(result_rectangles)
        threshold = 0.3     # threshold of IoU of two rectangles
        cur_rect = 0
        while cur_rect < number_of_rects - 1:     # start from first element to second last element
            rects_to_compare = number_of_rects - cur_rect - 1      # elements after current element to compare
            cur_rect_to_compare = cur_rect + 1    # start comparing with element ater current
            while rects_to_compare > 0:      # while there is at least one element after current to compare
                if IoU(result_rectangles[cur_rect], result_rectangles[cur_rect_to_compare]) >= 0.2  \
                        or ((IoM(result_rectangles[cur_rect], result_rectangles[cur_rect_to_compare]) >= threshold)):
                            #and (result_rectangles[cur_rect_to_compare][5] < 0.85)):  # if IoU ratio is higher than threshold #10/12=0.8333?
                    del result_rectangles[cur_rect_to_compare]      # delete the rectangle
                    number_of_rects -= 1
                else:
                    cur_rect_to_compare += 1    # skip to next rectangle
                rects_to_compare -= 1
            cur_rect += 1   # finished comparing for current rectangle
    
        return result_rectangles
    
    # ====== Below functions (12cal ~ 48cal) take images in style of caffe (0~1 BGR)===
    def detect_face_12c(net_12c_full_conv, img, min_face_size, stride,
                        multiScale=False, scale_factor=1.414, threshold=0.05):
        '''
        :param img: image to detect faces
        :param min_face_size: minimum face size to detect (in pixels)
        :param stride: stride (in pixels)
        :param multiScale: whether to find faces under multiple scales or not
        :param scale_factor: scale to apply for pyramid
        :param threshold: score of patch must be above this value to pass to next net
        :return:    list of rectangles after global NMS
        '''
        net_kind = 12
        rectangles = []   # list of rectangles [x11, y11, x12, y12, confidence, current_scale] (corresponding to original image)
    
        current_scale = find_initial_scale(net_kind, min_face_size)     # find initial scale
        caffe_img_resized = resize_image(img, current_scale)      # resized initial caffe image
        current_height, current_width, channels = caffe_img_resized.shape
    
        while current_height > net_kind and current_width > net_kind:
            caffe_img_resized_CHW = caffe_img_resized.transpose((2, 0, 1))  # switch from H x W x C to C x H x W
            # shape for input (data blob is N x C x H x W), set data
            net_12c_full_conv.blobs['data'].reshape(1, *caffe_img_resized_CHW.shape)
            net_12c_full_conv.blobs['data'].data[...] = caffe_img_resized_CHW
            # run net and take argmax for prediction
            net_12c_full_conv.forward()
            out = net_12c_full_conv.blobs['prob'].data[0][1, :, :]
            # print out.shape
            out_height, out_width = out.shape
    
            for current_y in range(0, out_height):
                for current_x in range(0, out_width):
                    # total_windows += 1
                    confidence = out[current_y, current_x]  # left index is y, right index is x (starting from 0)
                    if confidence >= threshold:
                        current_rectangle = [int(2*current_x*current_scale), int(2*current_y*current_scale),
                                                 int(2*current_x*current_scale + net_kind*current_scale),
                                                 int(2*current_y*current_scale + net_kind*current_scale),
                                                 confidence, current_scale]     # find corresponding patch on image
                        rectangles.append(current_rectangle)
            if multiScale is False:
                break
            else:
                caffe_img_resized = resize_image(caffe_img_resized, scale_factor)
                current_scale *= scale_factor
                current_height, current_width, channels = caffe_img_resized.shape
    
        return rectangles
    def cal_face_12c(net_12_cal, caffe_img, rectangles):
        '''
        :param caffe_image: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
        height, width, channels = caffe_img.shape
        result = []
        all_cropped_caffe_img = []
    
        for cur_rectangle in rectangles:
    
            original_x1 = cur_rectangle[0]
            original_y1 = cur_rectangle[1]
            original_x2 = cur_rectangle[2]
            original_y2 = cur_rectangle[3]
    
            cropped_caffe_img = caffe_img[original_y1:original_y2, original_x1:original_x2] # crop image
            all_cropped_caffe_img.append(cropped_caffe_img)
    
        if len(all_cropped_caffe_img) == 0:
            return []
    
        output_all = net_12_cal.predict(all_cropped_caffe_img)   # predict through caffe
    
        for cur_rect in range(len(rectangles)):
            cur_rectangle = rectangles[cur_rect]
            output = output_all[cur_rect]
            prediction = output[0]      # (44, 1) ndarray
    
            threshold = 0.1
            indices = np.nonzero(prediction > threshold)[0]   # ndarray of indices where prediction is larger than threshold
    
            number_of_cals = len(indices)   # number of calibrations larger than threshold
    
            if number_of_cals == 0:     # if no calibration is needed, check next rectangle
                result.append(cur_rectangle)
                continue
    
            original_x1 = cur_rectangle[0]
            original_y1 = cur_rectangle[1]
            original_x2 = cur_rectangle[2]
            original_y2 = cur_rectangle[3]
            original_w = original_x2 - original_x1
            original_h = original_y2 - original_y1
    
            total_s_change = 0
            total_x_change = 0
            total_y_change = 0
    
            for current_cal in range(number_of_cals):       # accumulate changes, and calculate average
                cal_label = int(indices[current_cal])   # should be number in 0~44
                if (cal_label >= 0) and (cal_label <= 8):       # decide s change
                    total_s_change += 0.83
                elif (cal_label >= 9) and (cal_label <= 17):
                    total_s_change += 0.91
                elif (cal_label >= 18) and (cal_label <= 26):
                    total_s_change += 1.0
                elif (cal_label >= 27) and (cal_label <= 35):
                    total_s_change += 1.10
                else:
                    total_s_change += 1.21
    
                if cal_label % 9 <= 2:       # decide x change
                    total_x_change += -0.17
                elif (cal_label % 9 >= 6) and (cal_label % 9 <= 8):     # ignore case when 3<=x<=5, since adding 0 doesn't change
                    total_x_change += 0.17
    
                if cal_label % 3 == 0:       # decide y change
                    total_y_change += -0.17
                elif cal_label % 3 == 2:     # ignore case when 1, since adding 0 doesn't change
                    total_y_change += 0.17
    
            s_change = total_s_change / number_of_cals      # calculate average
            x_change = total_x_change / number_of_cals
            y_change = total_y_change / number_of_cals
    
            cur_result = cur_rectangle      # inherit format and last two attributes from original rectangle
            cur_result[0] = int(max(0, original_x1 - original_w * x_change / s_change))
            cur_result[1] = int(max(0, original_y1 - original_h * y_change / s_change))
            cur_result[2] = int(min(width, cur_result[0] + original_w / s_change))
            cur_result[3] = int(min(height, cur_result[1] + original_h / s_change))
    
            result.append(cur_result)
    
        result = sorted(result, key=itemgetter(4), reverse=True)    # sort rectangles according to confidence
                                                                            # reverse, so that it ranks from large to small
        return result
    def detect_face_24c(net_24c, caffe_img, rectangles):
        '''
        :param caffe_img: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
        result = []
        all_cropped_caffe_img = []
    
        for cur_rectangle in rectangles:
    
            x1 = cur_rectangle[0]
            y1 = cur_rectangle[1]
            x2 = cur_rectangle[2]
            y2 = cur_rectangle[3]
    
            cropped_caffe_img = caffe_img[y1:y2, x1:x2]     # crop image
            all_cropped_caffe_img.append(cropped_caffe_img)
    
        if len(all_cropped_caffe_img) == 0:
            return []
    
        prediction_all = net_24c.predict(all_cropped_caffe_img)   # predict through caffe
    
        for cur_rect in range(len(rectangles)):
            confidence = prediction_all[cur_rect][1]
            if confidence > 0.05:
                cur_rectangle = rectangles[cur_rect]
                cur_rectangle[4] = confidence
                result.append(cur_rectangle)
    
        return result
    def cal_face_24c(net_24_cal, caffe_img, rectangles):
        '''
        :param caffe_image: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
        height, width, channels = caffe_img.shape
        result = []
    
        for cur_rectangle in rectangles:
    
            original_x1 = cur_rectangle[0]
            original_y1 = cur_rectangle[1]
            original_x2 = cur_rectangle[2]
            original_y2 = cur_rectangle[3]
            original_w = original_x2 - original_x1
            original_h = original_y2 - original_y1
    
            cropped_caffe_img = caffe_img[original_y1:original_y2, original_x1:original_x2] # crop image
            output = net_24_cal.predict([cropped_caffe_img])   # predict through caffe
            prediction = output[0]      # (44, 1) ndarray
    
            threshold = 0.1
            indices = np.nonzero(prediction > threshold)[0]   # ndarray of indices where prediction is larger than threshold
    
            number_of_cals = len(indices)   # number of calibrations larger than threshold
    
            if number_of_cals == 0:     # if no calibration is needed, check next rectangle
                result.append(cur_rectangle)
                continue
    
            total_s_change = 0
            total_x_change = 0
            total_y_change = 0
    
            for current_cal in range(number_of_cals):       # accumulate changes, and calculate average
                cal_label = int(indices[current_cal])   # should be number in 0~44
                if (cal_label >= 0) and (cal_label <= 8):       # decide s change
                    total_s_change += 0.83
                elif (cal_label >= 9) and (cal_label <= 17):
                    total_s_change += 0.91
                elif (cal_label >= 18) and (cal_label <= 26):
                    total_s_change += 1.0
                elif (cal_label >= 27) and (cal_label <= 35):
                    total_s_change += 1.10
                else:
                    total_s_change += 1.21
    
                if cal_label % 9 <= 2:       # decide x change
                    total_x_change += -0.17
                elif (cal_label % 9 >= 6) and (cal_label % 9 <= 8):     # ignore case when 3<=x<=5, since adding 0 doesn't change
                    total_x_change += 0.17
    
                if cal_label % 3 == 0:       # decide y change
                    total_y_change += -0.17
                elif cal_label % 3 == 2:     # ignore case when 1, since adding 0 doesn't change
                    total_y_change += 0.17
    
            s_change = total_s_change / number_of_cals      # calculate average
            x_change = total_x_change / number_of_cals
            y_change = total_y_change / number_of_cals
    
            cur_result = cur_rectangle      # inherit format and last two attributes from original rectangle
            cur_result[0] = int(max(0, original_x1 - original_w * x_change / s_change))
            cur_result[1] = int(max(0, original_y1 - original_h * y_change / s_change))
            cur_result[2] = int(min(width, cur_result[0] + original_w / s_change))
            cur_result[3] = int(min(height, cur_result[1] + original_h / s_change))
    
            result.append(cur_result)
    
        return result
    def detect_face_48c(net_48c, caffe_img, rectangles):
        '''
        :param caffe_img: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
        result = []
        all_cropped_caffe_img = []
    
        for cur_rectangle in rectangles:
    
            x1 = cur_rectangle[0]
            y1 = cur_rectangle[1]
            x2 = cur_rectangle[2]
            y2 = cur_rectangle[3]
    
            cropped_caffe_img = caffe_img[y1:y2, x1:x2]     # crop image
            all_cropped_caffe_img.append(cropped_caffe_img)
    
            prediction = net_48c.predict([cropped_caffe_img])   # predict through caffe
            confidence = prediction[0][1]
    
            if confidence > 0.3:
                cur_rectangle[4] = confidence
                result.append(cur_rectangle)
    
        result = sorted(result, key=itemgetter(4), reverse=True)    # sort rectangles according to confidence
                                                                            # reverse, so that it ranks from large to small
        return result
    def cal_face_48c(net_48_cal, caffe_img, rectangles):
        '''
        :param caffe_image: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
        height, width, channels = caffe_img.shape
        result = []
        for cur_rectangle in rectangles:
    
            original_x1 = cur_rectangle[0]
            original_y1 = cur_rectangle[1]
            original_x2 = cur_rectangle[2]
            original_y2 = cur_rectangle[3]
            original_w = original_x2 - original_x1
            original_h = original_y2 - original_y1
    
            cropped_caffe_img = caffe_img[original_y1:original_y2, original_x1:original_x2] # crop image
            output = net_48_cal.predict([cropped_caffe_img])   # predict through caffe
    
            prediction = output[0]      # (44, 1) ndarray
    
            threshold = 0.1
            indices = np.nonzero(prediction > threshold)[0]   # ndarray of indices where prediction is larger than threshold
    
            number_of_cals = len(indices)   # number of calibrations larger than threshold
    
            if number_of_cals == 0:     # if no calibration is needed, check next rectangle
                result.append(cur_rectangle)
                continue
    
            total_s_change = 0
            total_x_change = 0
            total_y_change = 0
    
            for current_cal in range(number_of_cals):       # accumulate changes, and calculate average
                cal_label = int(indices[current_cal])   # should be number in 0~44
                if (cal_label >= 0) and (cal_label <= 8):       # decide s change
                    total_s_change += 0.83
                elif (cal_label >= 9) and (cal_label <= 17):
                    total_s_change += 0.91
                elif (cal_label >= 18) and (cal_label <= 26):
                    total_s_change += 1.0
                elif (cal_label >= 27) and (cal_label <= 35):
                    total_s_change += 1.10
                else:
                    total_s_change += 1.21
    
                if cal_label % 9 <= 2:       # decide x change
                    total_x_change += -0.17
                elif (cal_label % 9 >= 6) and (cal_label % 9 <= 8):     # ignore case when 3<=x<=5, since adding 0 doesn't change
                    total_x_change += 0.17
    
                if cal_label % 3 == 0:       # decide y change
                    total_y_change += -0.17
                elif cal_label % 3 == 2:     # ignore case when 1, since adding 0 doesn't change
                    total_y_change += 0.17
    
            s_change = total_s_change / number_of_cals      # calculate average
            x_change = total_x_change / number_of_cals
            y_change = total_y_change / number_of_cals
    
            cur_result = cur_rectangle      # inherit format and last two attributes from original rectangle
            cur_result[0] = int(max(0, original_x1 - original_w * x_change / s_change))
            cur_result[1] = int(max(0, original_y1 - 1.1 * original_h * y_change / s_change))
            cur_result[2] = int(min(width, cur_result[0] + original_w / s_change))
            cur_result[3] = int(min(height, cur_result[1] + 1.1 * original_h / s_change))
    
            result.append(cur_result)
    
        return result
    
    def detect_faces(nets, img_forward, caffe_image, min_face_size, stride,
                     multiScale=False, scale_factor=1.414, threshold=0.05):
        '''
        Complete flow of face cascade detection
        :param nets: 6 nets as a tuple
        :param img_forward: image in normal style after subtracting mean pixel value
        :param caffe_image: image in style of caffe (0~1 BGR)
        :param min_face_size:
        :param stride:
        :param multiScale:
        :param scale_factor:
        :param threshold:
        :return: list of rectangles
        '''
        net_12c_full_conv = nets[0]
        net_12_cal = nets[1]
        net_24c = nets[2]
        net_24_cal = nets[3]
        net_48c = nets[4]
        net_48_cal = nets[5]
    
        rectangles = detect_face_12c(net_12c_full_conv, img_forward, min_face_size,
                                     stride, multiScale, scale_factor, threshold)     # detect faces
        rectangles = cal_face_12c(net_12_cal, caffe_image, rectangles)      # calibration
        rectangles = localNMS(rectangles)      # apply local NMS
        rectangles = detect_face_24c(net_24c, caffe_image, rectangles)
        rectangles = cal_face_24c(net_24_cal, caffe_image, rectangles)      # calibration
        rectangles = localNMS(rectangles)      # apply local NMS
        rectangles = detect_face_48c(net_48c, caffe_image, rectangles)
        rectangles = globalNMS(rectangles)      # apply global NMS
        rectangles = cal_face_48c(net_48_cal, caffe_image, rectangles)      # calibration
    
        return rectangles
    
    # ========== Adjusts net to take one crop of image only during test time ==========
    # ====== Below functions take images in normal style after subtracting mean pixel value===
    def detect_face_12c_net(net_12c_full_conv, img_forward, min_face_size, stride,
                        multiScale=False, scale_factor=1.414, threshold=0.05):
        '''
        Adjusts net to take one crop of image only during test time
        :param img: image in caffe style to detect faces
        :param min_face_size: minimum face size to detect (in pixels)
        :param stride: stride (in pixels)
        :param multiScale: whether to find faces under multiple scales or not
        :param scale_factor: scale to apply for pyramid
        :param threshold: score of patch must be above this value to pass to next net
        :return:    list of rectangles after global NMS
        '''
        net_kind = 12
        rectangles = []   # list of rectangles [x11, y11, x12, y12, confidence, current_scale] (corresponding to original image)
    
        current_scale = find_initial_scale(net_kind, min_face_size)     # find initial scale
        caffe_img_resized = resize_image(img_forward, current_scale)      # resized initial caffe image
        current_height, current_width, channels = caffe_img_resized.shape
    
        # print "Shape after resizing : " + str(caffe_img_resized.shape)
    
        while current_height > net_kind and current_width > net_kind:
            caffe_img_resized_CHW = caffe_img_resized.transpose((2, 0, 1))  # switch from H x W x C to C x H x W
            # shape for input (data blob is N x C x H x W), set data
            net_12c_full_conv.blobs['data'].reshape(1, *caffe_img_resized_CHW.shape)
            net_12c_full_conv.blobs['data'].data[...] = caffe_img_resized_CHW
            # run net and take argmax for prediction
            net_12c_full_conv.forward()
    
            out = net_12c_full_conv.blobs['prob'].data[0][1, :, :]
            # print out.shape
            out_height, out_width = out.shape
    
            # threshold = 0.02
            # idx = out[:, :] >= threshold
            # out[idx] = 1
            # idx = out[:, :] < threshold
            # out[idx] = 0
            # cv2.imshow('img', out*255)
            # cv2.waitKey(0)
    
            # print "Shape of output after resizing " + str(caffe_img_resized.shape) + " : " + str(out.shape)
    
            for current_y in range(0, out_height):
                for current_x in range(0, out_width):
                    # total_windows += 1
                    confidence = out[current_y, current_x]  # left index is y, right index is x (starting from 0)
                    if confidence >= threshold:
                        current_rectangle = [int(2*current_x*current_scale), int(2*current_y*current_scale),
                                                 int(2*current_x*current_scale + net_kind*current_scale),
                                                 int(2*current_y*current_scale + net_kind*current_scale),
                                                 confidence, current_scale]     # find corresponding patch on image
                        rectangles.append(current_rectangle)
            if multiScale is False:
                break
            else:
                caffe_img_resized = resize_image(caffe_img_resized, scale_factor)
                current_scale *= scale_factor
                current_height, current_width, channels = caffe_img_resized.shape
    
        return rectangles
    def cal_face_12c_net(net_12_cal, img_forward, rectangles):
        '''
        Adjusts net to take one crop of image only during test time
        :param caffe_image: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
    
        height, width, channels = img_forward.shape
        result = []
        for cur_rectangle in rectangles:
    
            original_x1 = cur_rectangle[0]
            original_y1 = cur_rectangle[1]
            original_x2 = cur_rectangle[2]
            original_y2 = cur_rectangle[3]
            original_w = original_x2 - original_x1
            original_h = original_y2 - original_y1
    
            cropped_caffe_img = img_forward[original_y1:original_y2, original_x1:original_x2] # crop image
    
            caffe_img_resized = cv2.resize(cropped_caffe_img, (12, 12))
            caffe_img_resized_CHW = caffe_img_resized.transpose((2, 0, 1))
            net_12_cal.blobs['data'].reshape(1, *caffe_img_resized_CHW.shape)
            net_12_cal.blobs['data'].data[...] = caffe_img_resized_CHW
            net_12_cal.forward()
    
            output = net_12_cal.blobs['prob'].data
    
            # output = net_12_cal.predict([cropped_caffe_img])   # predict through caffe
    
            prediction = output[0]      # (44, 1) ndarray
    
            threshold = 0.1
            indices = np.nonzero(prediction > threshold)[0]   # ndarray of indices where prediction is larger than threshold
    
            number_of_cals = len(indices)   # number of calibrations larger than threshold
    
            if number_of_cals == 0:     # if no calibration is needed, check next rectangle
                result.append(cur_rectangle)
                continue
    
            total_s_change = 0
            total_x_change = 0
            total_y_change = 0
    
            for current_cal in range(number_of_cals):       # accumulate changes, and calculate average
                cal_label = int(indices[current_cal])   # should be number in 0~44
                if (cal_label >= 0) and (cal_label <= 8):       # decide s change
                    total_s_change += 0.83
                elif (cal_label >= 9) and (cal_label <= 17):
                    total_s_change += 0.91
                elif (cal_label >= 18) and (cal_label <= 26):
                    total_s_change += 1.0
                elif (cal_label >= 27) and (cal_label <= 35):
                    total_s_change += 1.10
                else:
                    total_s_change += 1.21
    
                if cal_label % 9 <= 2:       # decide x change
                    total_x_change += -0.17
                elif (cal_label % 9 >= 6) and (cal_label % 9 <= 8):     # ignore case when 3<=x<=5, since adding 0 doesn't change
                    total_x_change += 0.17
    
                if cal_label % 3 == 0:       # decide y change
                    total_y_change += -0.17
                elif cal_label % 3 == 2:     # ignore case when 1, since adding 0 doesn't change
                    total_y_change += 0.17
    
            s_change = total_s_change / number_of_cals      # calculate average
            x_change = total_x_change / number_of_cals
            y_change = total_y_change / number_of_cals
    
            cur_result = cur_rectangle      # inherit format and last two attributes from original rectangle
            cur_result[0] = int(max(0, original_x1 - original_w * x_change / s_change))
            cur_result[1] = int(max(0, original_y1 - original_h * y_change / s_change))
            cur_result[2] = int(min(width, cur_result[0] + original_w / s_change))
            cur_result[3] = int(min(height, cur_result[1] + original_h / s_change))
    
            result.append(cur_result)
    
        result = sorted(result, key=itemgetter(4), reverse=True)    # sort rectangles according to confidence
                                                                            # reverse, so that it ranks from large to small
        return result
    def detect_face_24c_net(net_24c, img_forward, rectangles):
        '''
        Adjusts net to take one crop of image only during test time
        :param caffe_img: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
        result = []
        for cur_rectangle in rectangles:
    
            x1 = cur_rectangle[0]
            y1 = cur_rectangle[1]
            x2 = cur_rectangle[2]
            y2 = cur_rectangle[3]
    
            cropped_caffe_img = img_forward[y1:y2, x1:x2]     # crop image
    
            caffe_img_resized = cv2.resize(cropped_caffe_img, (24, 24))
            caffe_img_resized_CHW = caffe_img_resized.transpose((2, 0, 1))
            net_24c.blobs['data'].reshape(1, *caffe_img_resized_CHW.shape)
            net_24c.blobs['data'].data[...] = caffe_img_resized_CHW
            net_24c.forward()
    
            prediction = net_24c.blobs['prob'].data
    
            confidence = prediction[0][1]
    
            if confidence > 0.9:#0.05:
                cur_rectangle[4] = confidence
                result.append(cur_rectangle)
    
        return result
    def cal_face_24c_net(net_24_cal, img_forward, rectangles):
        '''
        Adjusts net to take one crop of image only during test time
        :param caffe_image: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
        height, width, channels = img_forward.shape
        result = []
        for cur_rectangle in rectangles:
    
            original_x1 = cur_rectangle[0]
            original_y1 = cur_rectangle[1]
            original_x2 = cur_rectangle[2]
            original_y2 = cur_rectangle[3]
            original_w = original_x2 - original_x1
            original_h = original_y2 - original_y1
    
            cropped_caffe_img = img_forward[original_y1:original_y2, original_x1:original_x2] # crop image
    
            caffe_img_resized = cv2.resize(cropped_caffe_img, (24, 24))
            caffe_img_resized_CHW = caffe_img_resized.transpose((2, 0, 1))
            net_24_cal.blobs['data'].reshape(1, *caffe_img_resized_CHW.shape)
            net_24_cal.blobs['data'].data[...] = caffe_img_resized_CHW
            net_24_cal.forward()
    
            output = net_24_cal.blobs['prob'].data
    
            prediction = output[0]      # (44, 1) ndarray
    
            threshold = 0.1
            indices = np.nonzero(prediction > threshold)[0]   # ndarray of indices where prediction is larger than threshold
    
            number_of_cals = len(indices)   # number of calibrations larger than threshold
    
            if number_of_cals == 0:     # if no calibration is needed, check next rectangle
                result.append(cur_rectangle)
                continue
    
            total_s_change = 0
            total_x_change = 0
            total_y_change = 0
    
            for current_cal in range(number_of_cals):       # accumulate changes, and calculate average
                cal_label = int(indices[current_cal])   # should be number in 0~44
                if (cal_label >= 0) and (cal_label <= 8):       # decide s change
                    total_s_change += 0.83
                elif (cal_label >= 9) and (cal_label <= 17):
                    total_s_change += 0.91
                elif (cal_label >= 18) and (cal_label <= 26):
                    total_s_change += 1.0
                elif (cal_label >= 27) and (cal_label <= 35):
                    total_s_change += 1.10
                else:
                    total_s_change += 1.21
    
                if cal_label % 9 <= 2:       # decide x change
                    total_x_change += -0.17
                elif (cal_label % 9 >= 6) and (cal_label % 9 <= 8):     # ignore case when 3<=x<=5, since adding 0 doesn't change
                    total_x_change += 0.17
    
                if cal_label % 3 == 0:       # decide y change
                    total_y_change += -0.17
                elif cal_label % 3 == 2:     # ignore case when 1, since adding 0 doesn't change
                    total_y_change += 0.17
    
            s_change = total_s_change / number_of_cals      # calculate average
            x_change = total_x_change / number_of_cals
            y_change = total_y_change / number_of_cals
    
            cur_result = cur_rectangle      # inherit format and last two attributes from original rectangle
            cur_result[0] = int(max(0, original_x1 - original_w * x_change / s_change))
            cur_result[1] = int(max(0, original_y1 - original_h * y_change / s_change))
            cur_result[2] = int(min(width, cur_result[0] + original_w / s_change))
            cur_result[3] = int(min(height, cur_result[1] + original_h / s_change))
    
            result.append(cur_result)
    
        result = sorted(result, key=itemgetter(4), reverse=True)    # sort rectangles according to confidence                                                                       # reverse, so that it ranks from large to small
    
        return result
    def detect_face_48c_net(net_48c, img_forward, rectangles):
        '''
        Adjusts net to take one crop of image only during test time
        :param caffe_img: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
        result = []
        for cur_rectangle in rectangles:
    
            x1 = cur_rectangle[0]
            y1 = cur_rectangle[1]
            x2 = cur_rectangle[2]
            y2 = cur_rectangle[3]
    
            cropped_caffe_img = img_forward[y1:y2, x1:x2]     # crop image
    
            caffe_img_resized = cv2.resize(cropped_caffe_img, (48, 48))
            caffe_img_resized_CHW = caffe_img_resized.transpose((2, 0, 1))
            net_48c.blobs['data'].reshape(1, *caffe_img_resized_CHW.shape)
            net_48c.blobs['data'].data[...] = caffe_img_resized_CHW
            net_48c.forward()
    
            prediction = net_48c.blobs['prob'].data
    
            confidence = prediction[0][1]
    
            if confidence > 0.95:#0.1:
                cur_rectangle[4] = confidence
                result.append(cur_rectangle)
    
        result = sorted(result, key=itemgetter(4), reverse=True)    # sort rectangles according to confidence
                                                                            # reverse, so that it ranks from large to small #after sorter and globalnms the best faces will be detected
        return result
    def cal_face_48c_net(net_48_cal, img_forward, rectangles):
        '''
        Adjusts net to take one crop of image only during test time
        :param caffe_image: image in caffe style to detect faces
        :param rectangles:  rectangles in form [x11, y11, x12, y12, confidence, current_scale]
        :return:    rectangles after calibration
        '''
        height, width, channels = img_forward.shape
        result = []
        for cur_rectangle in rectangles:
    
            original_x1 = cur_rectangle[0]
            original_y1 = cur_rectangle[1]
            original_x2 = cur_rectangle[2]
            original_y2 = cur_rectangle[3]
            original_w = original_x2 - original_x1
            original_h = original_y2 - original_y1
    
            cropped_caffe_img = img_forward[original_y1:original_y2, original_x1:original_x2] # crop image
            caffe_img_resized = cv2.resize(cropped_caffe_img, (48, 48))
            caffe_img_resized_CHW = caffe_img_resized.transpose((2, 0, 1))
            net_48_cal.blobs['data'].reshape(1, *caffe_img_resized_CHW.shape)
            net_48_cal.blobs['data'].data[...] = caffe_img_resized_CHW
            net_48_cal.forward()
    
            output = net_48_cal.blobs['prob'].data
    
            prediction = output[0]      # (44, 1) ndarray
    
            threshold = 0.1
            indices = np.nonzero(prediction > threshold)[0]   # ndarray of indices where prediction is larger than threshold
    
            number_of_cals = len(indices)   # number of calibrations larger than threshold
    
            if number_of_cals == 0:     # if no calibration is needed, check next rectangle
                result.append(cur_rectangle)
                continue
    
            total_s_change = 0
            total_x_change = 0
            total_y_change = 0
    
            for current_cal in range(number_of_cals):       # accumulate changes, and calculate average
                cal_label = int(indices[current_cal])   # should be number in 0~44
                if (cal_label >= 0) and (cal_label <= 8):       # decide s change
                    total_s_change += 0.83
                elif (cal_label >= 9) and (cal_label <= 17):
                    total_s_change += 0.91
                elif (cal_label >= 18) and (cal_label <= 26):
                    total_s_change += 1.0
                elif (cal_label >= 27) and (cal_label <= 35):
                    total_s_change += 1.10
                else:
                    total_s_change += 1.21
    
                if cal_label % 9 <= 2:       # decide x change
                    total_x_change += -0.17
                elif (cal_label % 9 >= 6) and (cal_label % 9 <= 8):     # ignore case when 3<=x<=5, since adding 0 doesn't change
                    total_x_change += 0.17
    
                if cal_label % 3 == 0:       # decide y change
                    total_y_change += -0.17
                elif cal_label % 3 == 2:     # ignore case when 1, since adding 0 doesn't change
                    total_y_change += 0.17
    
            s_change = total_s_change / number_of_cals      # calculate average
            x_change = total_x_change / number_of_cals
            y_change = total_y_change / number_of_cals
    
            cur_result = cur_rectangle      # inherit format and last two attributes from original rectangle
            cur_result[0] = int(max(0, original_x1 - original_w * x_change / s_change))
            cur_result[1] = int(max(0, original_y1 - 1.1 * original_h * y_change / s_change))
            cur_result[2] = int(min(width, cur_result[0] + original_w / s_change))
            cur_result[3] = int(min(height, cur_result[1] + 1.1 * original_h / s_change))
    
            result.append(cur_result)
    
        return result
    
    def detect_faces_net(nets, img_forward, min_face_size, stride,
                     multiScale=False, scale_factor=1.414, threshold=0.05):
        '''
        Complete flow of face cascade detection
        :param nets: 6 nets as a tuple
        :param img_forward: image in normal style after subtracting mean pixel value
        :param min_face_size:
        :param stride:
        :param multiScale:
        :param scale_factor:
        :param threshold:
        :return: list of rectangles
        '''
        net_12c_full_conv = nets[0]
        net_12_cal = nets[1]
        net_24c = nets[2]
        net_24_cal = nets[3]
        net_48c = nets[4]
        net_48_cal = nets[5]
    
        rectangles = detect_face_12c_net(net_12c_full_conv, img_forward, min_face_size,
                                     stride, multiScale, scale_factor, threshold)  # detect faces
        rectangles = cal_face_12c_net(net_12_cal, img_forward, rectangles)      # calibration
        rectangles = localNMS(rectangles)      # apply local NMS
        rectangles = detect_face_24c_net(net_24c, img_forward, rectangles)
        rectangles = cal_face_24c_net(net_24_cal, img_forward, rectangles)      # calibration
        rectangles = localNMS(rectangles)      # apply local NMS
        rectangles = detect_face_48c_net(net_48c, img_forward, rectangles)
        rectangles = globalNMS(rectangles)      # apply global NMS
        rectangles = cal_face_48c_net(net_48_cal, img_forward, rectangles)      # calibration
    
        return rectangles

[SouthEast]: /images/20220616/c8dabef738e24385bf74bd55c094aa8a.png
[Link 1]: http://www.cs.stevens.edu/~ghua/publication/CVPR15b.pdf
[github]: https://github.com/anson0910/CNN_face_detection
[github 1]: https://github.com/anson0910/CNN_face_detection_models
[SouthEast 1]: /images/20220616/17cd9c03c388490c9ae6f9e2c85fa4bd.png
[SouthEast 2]: /images/20220616/2776f494b63845e9bccb6919e09cb2ae.png
[SouthEast 3]: /images/20220616/b61af8d7be37471c925a5afe10bc389d.png