Detection of Blink State Based on Fatigued Driving

## Publications

/ Export Citation / / / Text size:

#### International Journal of Advanced Network, Monitoring and Controls

Xi'an Technological University

Subject: Computer Science, Software Engineering

eISSN: 2470-8038

17
58
Visit(s)
0
Comment(s)
0
Share(s)

SEARCH WITHIN CONTENT

FIND ARTICLE

Volume / Issue / page

Archive
Volume 6 (2021)
Volume 5 (2020)
Volume 4 (2019)
Volume 3 (2018)
Volume 2 (2017)
Volume 1 (2016)
Related articles

VOLUME 4 , ISSUE 4 (December 2019) > List of articles

### Detection of Blink State Based on Fatigued Driving

Citation Information : International Journal of Advanced Network, Monitoring and Controls. Volume 4, Issue 4, Pages 24-29, DOI: https://doi.org/10.21307/ijanmc-2019-067

Published Online: 27-January-2020

### ARTICLE

#### ABSTRACT

In recent years, with the improvement of the national economy, the penetration rate of automobiles has been increasing, and traffic accidents have also increased. Fatigue driving is the main factor in many traffic accidents. Fatigue driving can cause the driver’s inattention, slow response, and make wrong decisions on danger signals, which affect the driver’s personal safety. In modern development, driving safety is developing towards intelligence and safety. Therefore, the detection of driver fatigue has become a generally accepted demand. This paper proposes a method to calculate the threshold of blinking, which can detect the blinking state of the driver in real time through video. During the driving process, when the driver is in the closed eye state for a long time, an early warning is issued to avoid the accident. This paper uses Python language to achieve the first, through the digital image technology call Dlib open source library to detect 68 feature points of the face, and then measure the aspect ratio between the length and width of the human eye, and finally through the Kmeans clustering algorithm to collect the ratio The analysis yields the blink threshold. The experimental results show that the recognition rate is 92.5% when the video frame rate is 30, and the recognition accuracy is 92.5%. The experimental results show that the method designed in this paper can quickly detect the fatigue characteristics of the human eye, has a higher recognition rate and accuracy for fatigue driving, and helps reduce the occurrence of traffic accidents.

## I. INTRODUCTION

With the improvement of people’s material living standards, cars have become the main means of transportation for people, but the growing number of vehicles has led to more traffic accidents. According to statistics, fatigue driving is the main cause of traffic accidents[1,2].Under normal circumstances, the medical community believes that there are two reasons for fatigue driving, one is because the driver’s attention is too concentrated, and the other is that the body does not rest well. Because of being in this state for a long time, the body will be fatigued, lose concentration, the driver will snoring, lose concentration, decrease the ability to judge dangerous situations, and cause traffic accidents. At present, there are relatively few applications of fatigue driving equipment in China’s in-vehicle systems. Fatigue detection mainly through facial features, eye and mouth features, human electrical signal characteristics and convolutional neural network characteristics[3,4,5].The detection of facial features is generally based on the frequency of blinking eyes, the degree of mouth opening, and the frequency of head movements due to fatigue. The fatigue of human body electrical signals is generally the measurement of surface EMG signals, because human fatigue can be expressed by muscle physiological information. Surface EMG signals can reflect real-time physiological processes of muscle information and physiological signals on the skin surface. Convolutional neural networks generally extract facial features through image processing methods, and then extract the main features through convolutional layers, pooling layers, and fully connected layers to analyze and determine whether fatigue. Chen[6] uses the ASM algorithm to accurately locate the eyes and mouth area, calculates the eye’s aspect ratio, mouth height value, and black and white pixel ratio near the mouth, and obtains the blink frequency and mouth opening degree. The degree of mouth opening is used as an input to the fuzzy inference engine to obtain three types of fatigue levels to accurately quantify the degree of fatigue

The method proposed in this paper is to judge the driver’s fatigue driving according to the characteristics of the human eye. Because the digital image processing open source visual library OpenCV comes with a human face detection library, but the disadvantage is that the lighting requirements are very high, the lighting slightly changed, it will be difficult to locate or inaccurate positioning[7]. Therefore, this paper chooses Dlib open source library to detect human eye features. Firstly, the 68 face feature points provided by the Dlib open source library are used to accurately calibrate the position of the face and the human eye, and then the aspect ratio between the length and the width of the human eye is measured. Finally, the Kmeans clustering algorithm is used to analyze the collected ratio. The threshold of blinking. Figure 1 below, a is the 68 face feature points marked by Dlib, and b is the feature point on the face of the paper.

##### Figure 1.

Facial feature points

## II. RELATED WORK

### A. Blink detection and threshold analysis methods

This chapter mainly introduces the blink algorithm formula and blink threshold analysis method. The blink threshold analysis method uses the Kmeans clustering algorithm in machine learning. There are many methods for blink detection, such as support vector machine classification, eye movement sequence analysis, convolutional neural network feature extraction, eye feature point analysis, etc. This article uses the eye feature analysis method. Threshold analysis methods in machine learning usually use regression algorithms, decision tree methods, Bayesian methods, and clustering algorithms. This article uses the Kmeans clustering algorithm in machine learning.

##### (1)
$Blinkthreshold=2|p1−p4||p2−p6|+|p3−p5|$

##### Figure 2.

(a) The lateral distance is cd longitudinally ab; (b) dlib human eye calibration features

### C. Kmeans clustering algorithm

The Kmeans algorithm is a relatively common algorithm in clustering algorithms. Its advantage is that it is easy to implement and understand, and the calculation speed is fast. The core idea is to calculate the distance between the sample point and the centroid of the cluster, and divide the calculated result into the same cluster as the sample point with the centroid of the cluster.

The similarity between samples in K-means is determined by the distance between them. The closer the distance is, the higher the similarity is. The common distance calculation methods are Euclidean distance, Euclidean distance and Manhattan distance. European distance. In the cluster analysis, the formulas for two m-dimensional samples xi=(xi1,xi2,xi3…,xim) and xj=(xj1,xj2,xj3…,xjm) are as follows:

##### (2)
$dis⁡ted=∑k=1m(xik−xjk)2$

The steps of the k-means algorithm are as follows:

• 1) First randomly select the centroids of K clusters.

• 2) Calculate the Euclidean distance from each sample point to each centroid, and classify it into the cluster with the smallest center of mass, and then calculate the centroid of each new cluster.

• 3) After all the sample points are divided, recalculate the position of the centroid of each cluster, and then iteratively calculate the distance from each sample point to the centroid of each cluster, and then re-divide the sample points.

• 4) Repeat steps 2 and 3 until after the iteration, the partitioning of all sample points remains unchanged, and K-means gets the optimal solution.

The main problem of the calculation result is to ensure the convergence of the algorithm. Here, the square error is calculated by the following formula, which is used to illustrate that the clustering effect can minimize the sum of squares in each cluster.

##### (3)
$J(c,u)=∑i=1K∥x(i)−uc(i)∥2$

j(c, u) represents the sum of squares of the distance from each sample point to its cluster, uc(i) represents the centroid of the cluster to which the i-th sample belongs, and the smaller j(c, u), all the sample points and their clusters The smaller the distance, the better the quality of the division. The termination condition of the K-means algorithm is that j(c, u) converges to a minimum. In order to achieve clustering, the maximum value of the objective function is obtained. Take a one-dimensional array as an example.

##### (4)
$J=∑i=1k∑xj∈ui(x(i)−uc(i))2$

Transform the above formula to get:

$∂J∂ui=∂∂ui∑i=1k∑xj∈ui(xj−ui)2$

When $(−2)∗∑xj∈ui(xj−ui)=0ui=1|ci|∑xj∈uiXj$ The result of the optimization is to calculate the mean of the cluster.

During the experiment, the algorithm may be too slow to achieve effective results because the data set is too large. Therefore, you can specify the maximum number of convergence times for the K-means algorithm or specify the cluster center transformation threshold. When the algorithm reaches the maximum number of times or When the cluster center rate of change is less than a certain threshold, the algorithm stops updating.

K-means algorithm advantages: easy to understand, easy to implement, high operating efficiency, the disadvantage is that the greedy strategy is used to cluster the sample points, resulting in easy local convergence of the algorithm, slower data processing in big data, and outliers and The noise is very sensitive, and a small number of outliers and noise points can have a significant impact on the averaging of the algorithm.

## III. THE EXPERIMENT

##### Figure 3.

The following is the experimental data of the paper

##### Figure 4.

Public data set sample

The following table a is a comparison of the public data sets provided by Zhejiang University and the experimental results of the text person. Table b is a comparison of other methods with the method of this paper. RenAnhu[9] trained the classifiers of blinking and closed eyes through the Adaboosts algorithm. The person in the video is then tested for blinking. Zhang Wei[14] performed a correlation analysis of the blink of the eye by analyzing the left forehead EEG signals Attention and Meditation and Blink data.

##### TABLE I.

COMPARED WITH PUBLIC DATASETS

##### TABLE II.

COMPARED WITH OTHER LITERATURE

## IV. CONCLUSION

This paper overcomes the shortcomings of digital image processing and OpenCV vision open source library, and combines the existing open source Dlib machine learning library,The data between the vertical and horizontal ratio of blink is calculated by mathematical method, and the threshold value of the vertical and horizontal ratio of blink is analyzed by means of kmeans clustering algorithm in machine learning. According to the analysis of the public data set of Zhejiang University, when the threshold value of the vertical and horizontal ratio of blink is 5.1, the accurate recognition rate of blink is 92.5%. Through the experimental comparison, this algorithm can effectively detect the fatigue state of blink, which is more important This algorithm is fast, efficient and easy to transplant to various devices, and has great practical value in the field of fatigue driving. The shortcomings of the paper: for fatigue monitoring, not only eyes as a reference point, nose tip shaking, mouth opening and so on have an impact on face fatigue, so the fatigue detection algorithm in this paper needs to be improved.

## References

1. M. Hülsmann, D. Donnermeyer, E. Schäfer. A critical appraisal of studies on cyclic fatigue resistance of engine-driven endodontic instruments[J]. International Endodontic Journal, 2019, 52(10).
2. Pierre Thiffault, Jacques Bergeron. Monotony of road environment and driver fatigue: a simulator study[J]. Accident Analysis and Prevention, 2003, 35(3).
3. Liu Longfei, Wu Shizhen, Xu Wangming. Real-time detection method of fatigue driving based on face feature point analysis[J]. Television Technology, 2018, 42(12): 27-30+55.
4. Yan Wang, Rui Huang, Lei Guo. Eye gaze pattern analysis for fatigue detection based on GP-BCNN with ESM[J]. Pattern Recognition Letters, 2019, 123.
5. Driver’s Fatigue Detection Based on Yawning Extraction[J]. Nawal Alioua, Aouatif Amine, Mohammed Rziza, Aboelmagd Noureldin. International Journal of Vehicular Technology. 2014
6. Chen Xin, Li Weixiang, Li Wei, Zhang Wenqing, Zhu Yuan. Multi-feature fusion fatigue detection method based on improved ASM [J]. Computer Engineering and Design, 2019, 40 (11): 3269-3275.
7. Rafael C. Gonzalez, Richard E. Woods. Digital Image Processing, Third Edition[M], 2017
8. Andrej Fogelton, Wanda Benesova. Eye blink completeness detection[J]. Computer Vision and Image Understanding, 2018.
9. Ren Anhu, Liu Bei. Face Recognition Blink Detection Based on Adaboost[J]. Computer and Digital Engineering, 2016, 44(03): 521-524.
10. Zeng Youwen, Feng Zhen, Zhu Yabing, Li Qi. Relationship between the number of blinks and fatigue based on EEG experiment[J]. Journal of Changchun University of Science and Technology(Natural Science Edition), 2017, 40(01):123-126.
11. Tereza Soukupová, Jan Čech, Eye blink detection using facial landmarks[J]. 21st Computer Vision Winter Workshop(CVWW), 2016
12. J. Manikandan, B. Venkataramani. Study and evaluation of a multi-class SVM classifier using diminishing learning technique[J]. Neurocomputing, 2009, 73(10).
13. F. Song, X. Tan, X. Liu and S. Chen, Eyes Closeness Detection from Still Images with Multi-scale Histograms of Principal Oriented Gradients, Pattern Recognition, 2014.
14. Zhang Wei, He Jian, Zhang Yan, Zhou Ming. A wearable fatigue driving detection system based on EEG and blink frequency[J]. Computer Engineering, 2017, 43(02): 293-298+303.

### FIGURES & TABLES

Figure 1.

Facial feature points

Figure 2.

(a) The lateral distance is cd longitudinally ab; (b) dlib human eye calibration features

Figure 3.

The following is the experimental data of the paper

Figure 4.

Public data set sample

### REFERENCES

1. M. Hülsmann, D. Donnermeyer, E. Schäfer. A critical appraisal of studies on cyclic fatigue resistance of engine-driven endodontic instruments[J]. International Endodontic Journal, 2019, 52(10).
2. Pierre Thiffault, Jacques Bergeron. Monotony of road environment and driver fatigue: a simulator study[J]. Accident Analysis and Prevention, 2003, 35(3).
3. Liu Longfei, Wu Shizhen, Xu Wangming. Real-time detection method of fatigue driving based on face feature point analysis[J]. Television Technology, 2018, 42(12): 27-30+55.
4. Yan Wang, Rui Huang, Lei Guo. Eye gaze pattern analysis for fatigue detection based on GP-BCNN with ESM[J]. Pattern Recognition Letters, 2019, 123.
5. Driver’s Fatigue Detection Based on Yawning Extraction[J]. Nawal Alioua, Aouatif Amine, Mohammed Rziza, Aboelmagd Noureldin. International Journal of Vehicular Technology. 2014
6. Chen Xin, Li Weixiang, Li Wei, Zhang Wenqing, Zhu Yuan. Multi-feature fusion fatigue detection method based on improved ASM [J]. Computer Engineering and Design, 2019, 40 (11): 3269-3275.
7. Rafael C. Gonzalez, Richard E. Woods. Digital Image Processing, Third Edition[M], 2017
8. Andrej Fogelton, Wanda Benesova. Eye blink completeness detection[J]. Computer Vision and Image Understanding, 2018.
9. Ren Anhu, Liu Bei. Face Recognition Blink Detection Based on Adaboost[J]. Computer and Digital Engineering, 2016, 44(03): 521-524.
10. Zeng Youwen, Feng Zhen, Zhu Yabing, Li Qi. Relationship between the number of blinks and fatigue based on EEG experiment[J]. Journal of Changchun University of Science and Technology(Natural Science Edition), 2017, 40(01):123-126.
11. Tereza Soukupová, Jan Čech, Eye blink detection using facial landmarks[J]. 21st Computer Vision Winter Workshop(CVWW), 2016
12. J. Manikandan, B. Venkataramani. Study and evaluation of a multi-class SVM classifier using diminishing learning technique[J]. Neurocomputing, 2009, 73(10).
13. F. Song, X. Tan, X. Liu and S. Chen, Eyes Closeness Detection from Still Images with Multi-scale Histograms of Principal Oriented Gradients, Pattern Recognition, 2014.
14. Zhang Wei, He Jian, Zhang Yan, Zhou Ming. A wearable fatigue driving detection system based on EEG and blink frequency[J]. Computer Engineering, 2017, 43(02): 293-298+303.