Multiple Vehicle License Plate Location in Complex Background

## Publications

/ Export Citation / / / Text size:

#### International Journal of Advanced Network, Monitoring and Controls

Xi'an Technological University

Subject: Computer Science, Software Engineering

eISSN: 2470-8038

14
21
Visit(s)
0
Comment(s)
0
Share(s)

SEARCH WITHIN CONTENT

FIND ARTICLE

Volume / Issue / page

Archive
Volume 6 (2021)
Volume 5 (2020)
Volume 4 (2019)
Volume 3 (2018)
Volume 2 (2017)
Volume 1 (2016)
Related articles

VOLUME 3 , ISSUE 1 (May 2018) > List of articles

### Multiple Vehicle License Plate Location in Complex Background

Citation Information : International Journal of Advanced Network, Monitoring and Controls. Volume 3, Issue 1, Pages 62-68, DOI: https://doi.org/10.21307/ijanmc-2018-004

Published Online: 07-May-2018

### ARTICLE

#### ABSTRACT

In order to expand the application range of the intelligent traffic management system, and to solve the problem that the license plate positioning accuracy is low in the changing of the scene. On the basis of the analysis of previous methods advantages and disadvantages, applying deep learning model orientation method is proposed. The image expressed as graph of graph theory. Based on the principle of minimum spanning tree preliminary separate target objects in image of vehicle. Combined with the color, dimension, texture and match the similarity to choose, search and merger area in the image, suspicious area of the license plate is obtained. Using visual word package to express rectangular profile after coarse positioning. Using support vector machine (SVM) to classify and identify rectangular area of license plate. Accurate positioning license plate location is positioned accurately. The method of accuracy is 96.4% for 135 pieces of test sample positioning, strong anti-jamming.

## I. VISUAL WORD PACKAGE MODEL OF LICENSE PLATE LOCALIZATION ALGORITHM

Analysis of previous license plate localization algorithm based on texture feature. After the license plate image by the morphological processing can effectively make the license plate area rectangle connected area. But it is difficult to further improve the accuracy of this method. If the interference area of the image and texture features of license plate is similar, then interference region to form a rectangular connected domain also. So just rely on license plate aspect ratio to determine conditions to distinguish the license plate region and the license plate region is difficult. This makes the poor anti-jamming, misjudgment rate is high. Aimed at the problem in rectangular area filter link using the visual word packet and support vector machine (SVM) to improve filtering accuracy. Basic principle of visual word package features is to use characteristic descriptors to express images, and put the picture as a different set of feature points. Through the statistics of each feature point frequency in the single photo to vectors to a photograph. Namely in the form of a histogram to represent the photograph [7]. As shown in figure 1 package basic process for visual words. Due to the different types of pictures’ vector representation from it’s visual word package are different, so it can choose the appropriate classifier using sample set for training. Then use the trained classifier to classify test images.

##### Figure 1.

Flow chart of BOW model

Considering the license plate images will get affected by Angle, light and other factors. In order to reduce such factors on the characteristics of the descriptor to describe the influence of the license plate area, and can achieve rapid positioning of the target. This paper adopted the surf feature descriptor. The descriptor is characterized by has fast calculation and partial invariance. That is to say, it has the scale of the robustness of a certain range scaling, image rotation, the change of perspective, illumination changes and image blur, can effectively eliminate from light, Angle and other factors. Extraction algorithm of surf feature descriptor is Hessian matrix is used to determine the candidate points. Not greatly restrain and realize the feature point detection. In the image pixel Hessian matrix are defined as follows:

##### (1)
$H(f(x,y))=[∂2f/∂x2∂2f/∂x∂y∂2f/∂x∂y∂2f/∂y2]$

Hessian matrix discriminant such as type (2). The value of the discriminant is the eigenvalues of the Hessian matrix. It classifies all points using symbols of determining structure and according to the discriminant value plus or minus identifying whether this point is the value of the pole.

##### (2)
$det(H)=∂2f∂x2∂2f∂y2−(∂2f∂x∂y)2$

This algorithm use image pixel L (x, y) to replace function f (x, y) and use type (1) through specific nuclear convolution between to computation the second order partial derivative, it is concluded that the Hessian matrix of the matrix elements is Lxx, Lxy, Lyy. According to the type (2) the discriminant, a approximate Hessian determinant map can be obtained. Compared with Hessian matrix of each pixel point and the pyramid image the size of the 26 points in the field of 3 d. The maximum as the characteristics of the initial point. As shown in figure 2. Local eigenvalues of x is greater than the surrounding pixels. It is the characteristic points of the region.

##### Figure 2.

Feature points

In order to guarantee the rotation invariance, we need to calculate Centered on feature points and the sum of the corresponding Harr wavelet for 60 degrees within the sector is all points in the direction of the x and y. Close to feature points of the weight is big, away from is small. To get the main direction of each feature point. This process is shown in figure 3.

##### Figure 3.

Determine the main direction

Although can Surf characteristics describe a image, an image contains a large number of Surf feature points. If the training directly used in classification, calculation will be very big. We need through the clustering algorithm to cluster the vector data. Using a cluster of clusters represent visual words in a visual word, and then map the surf feature points to the visual word package generated code. In this paper, using K means clustering algorithm in constructing the visual word package. The principle of algorithm is simple and easy to implement and there is a good clustering effect. Such as type (3) the calculation formula. E is sum of square error for all clustering objects. P is the clustering objects. |Ci| is the number of Ci class of clustering objects.

##### (3)
$E=∑∑|p−∑p∈Cip|Ci||$

The problem of License plate location only need to classify an outline of the license plate and the plate profile picture. We chose SVM, which is the classifier of dichotomy. SVM is a kind of machine learning methods based on the theory of VC dimension and the theory of structural risk minimization. It is outstanding in solving nonlinear problems and it was originally designed for binary classification problems [8]. Principle of SVM is put data map to high-dimensional space, then finding largest classification interval hyperplane in high dimensional space, and using the hyperplane to classify. Such as type (4) calculation formula. (xi, yi) is the training sample. l for the sample. n for the dimensions of the input. C is punish parameters. as though laser coefficient. k(xi, xj) for the selection of kernel function.

##### (4)
$max∂∑i=1n∂i−12∑i, j=1n∂i∂iyiyjk(xi,xj)s.t. 0≤∂i≤C, i=1,…,n∑i=1n∂iyi=0$

## II. THE REALIZATION OF THE ALGORITHM

### A. The suspicious area extraction of license plate

Image contains abundant information. The object shape, size, color, texture, and other characteristics. The image is defined as the graph of graph theory, are defined as follows:

##### (5)
$G=(V,E)vi,vj∈V,ei=(vi,vj)∈E$

Among them, vi is a picture point in the image and a vertex of graph in the graph theory. ei is the weight of an edge of graph in the graph theory, is vi and vj between the picture points of the gray level difference or the distance between the. Using Graph Cuts algorithm [] to divided graph G into a series of disjoint independent subgraphs G′. The algorithm is mainly using the minimum Spanning Tree (MST, Minimun Spanning Tree), realizing regional segmentation. For segmentation region contains the largest edge weights are defined as follows:

##### (6)
$Int(C)=maxe∈MST(C,E)ω(e)$

Among them, C is a segmented regions. w() is weight. The minimum edge weights of vertices are connected between the two regional segmentation are defined as follows:

##### (7)
$Dif(C1,C2)=minvi∈C1,vj∈C2,(vi,vj)∈Eω((vi,vj))$

For no edge connection between segmentation part, is S. For the segmentation boundary judgment are defined as follows:

##### (8)
$D(C,C)={trueif, Dif(C1,C2)>MInt(C1,C2)falseotherwise$

The smallest division of internal difference are defined as follows:

##### (9)
$MInt(C1,C2)=min(Int(C1)+τ(C1),Int(C2)+τ(C2))$

Among them, τ() as the threshold function control segmentation area of consolidation degree, defined as τ(C) = K/|C|, |C| is segmentation part of the number of vertices, k is coarsness of parameter control image segmentation area. Using Graph Cuts algorithm to Conduct a preliminary regional segmentation for image, the segmentation result is shown in figure 4.

##### Figure 4.

Image segmentation

In the image object is hierarchical. After initial segmented regions, using similarity to calculate diversification and region merging. For area merger adopt four similarity in the image as follows:

1) Color similarity. The normalized image obtains 25 bins in the histogram for each color channel. Each area can be expressed into a 75 - dimensional vector. Regional color of the computation formula is as follows:

##### (10)
$Scolour(ri,rj)=∑k=1nmin(cik,cjk)Ct=size(ri)×Ci+size(rj)×Cjsize(ri)+size(rj)$

2) Texture similarity. To calculate gaussian differential for each color channel of eight different direction. The normalized image acquires 10 bins in the histogram for each channel and for each color. Making areas represent a 240 - dimensional vector. computation formula is as follows:

##### (11)
$Stexture(ri,rj)=∑k=1nmin(tik,tjk)Ssize(ri,rj)=1−size(ri)+size(rj)size(im)$
##### (12)
$Ssize(ri,rj)=1−size(ri)+size(rj)size(im)$

3) Size similarity. Refers to the number of pixels of similarity in the region. Small area for priority merger. Computation formula is as follows:

4) Consistent with similarity. Refers to the marked rectangle the smaller alignment is higher for the combined area. The calculation is as follows:

##### (13)
$fill(ri,rj)=1−size(BBij)−size(ri)−size(ri)size(im)$

To combine the calculated four kinds of similarity. Similar set is S. Formula is as follows:

##### (14)
$s(ri,rj)=a1scolour(ri,rj)+a2stexture(ri,rj)+a3ssize(ri,rj)+a4sfill(ri,rj)$

The ai ∈ {0,1}. Finding the similarity of the two biggest area is ri and rj from S that is similarity of collection. merging it into a region rt, as well as dividing calculation of similarity that is in ri and rj between neighboring areas in S that is a collection. Calculate rt generated area and its adjacent area of similarity, and added to the similarity of set S. rt is added to the collection area R and use rectangular box to mark area. The suspicious area is extracted by using the algorithm for vehicle image as shown in figure 5.

##### Figure 5.

The suspicious area extraction

### B. The location of license plate

1) Putting the pixels in the image into type (1). The Hessian matrix on dimension can be got such as type (15). Among is convolution results that is gaussian filtering second derivative. and above meaning. Gaussian filter function such as type (6);

##### (15)
$H=[Lxx(x,σ^)Lxy(x,σ^)Lxy(x,σ^)Lyy(x,σ^)]$

2) On the original image, by expanding the size of the box to form the different scales of image pyramid. For example 9 * 9 box value of the filter template as shown in figure 6. The grey part template value is 0. Corresponding to the second order gauss filter is=1.2. is the scale of the corresponding values. After the convolution of the value of, ,. Get F of Hessian matrix expression is as follows:

##### (16)
$ΔH=DxxDxy−(0.9Dxy)2$
##### Figure 6.

9*9 filtering template

3) Building scale image pyramid. In every order, The four layers of the scale of the image is chosen. The four order of building of parameters as shown in figure 7. The grey number is the size of the filter template the box. If the image size is greater than the template size, we need to continue to increase the order. Such as the size of the filter template for N * N. The scale of the corresponding is N/s = 1.2 * 9;Using Hessian matrix to calculate the extremum, and then in the 3 * 3 * 3 within the territory of three-dimensional take non-maximum suppression. When has compared to last scale, the next scale and the scale of around 26 field values are big or hours, there are candidate features. In the scale space and image space take a interpolation arithmetic. Get the stable feature points location and the dimension values.

##### Figure 7.

The filter size of Octave

4) With feature points as the center, s is the scale value of the feature points. For calculating radius of 6 s points in the field in the x and y direction Harr wavelet response. And giving the response value assigned a gaussian weight coefficient, this allows the response contribution near the feature point is big, away from the feature point is small. Adding the 60 degrees within the scope of the response to form a new vector. Traverse the entire circle area, selecting the longest vector for the feature points in the direction of the principal direction. As shown in figure 3.

5) Rotating coordinate axis to the main direction. According to the main direction to choose side length of 20 s square area. The window area was divided into 4 * 4 sub area. Computing wavelet response in each area 5 s * 5 s within the scope. For and. As shown in figure 8. Each subregion of response and the absolute value of response together to form Σdx, Σdy, Σ|dx|, Σ|dy. In each subdomain form 4 d weight of vector = (Σdx, Σdy, Σ|dx|, Σ|dy). So each feature point is 4 * 4 * 4 = 64 dimension of description of the vector. And then to vector normalization. The extraction results as shown in figure 9.

##### Figure 8.

Generate surf feature points

##### Figure 9.

Surf feature points

Surf feature points are extracted. Using the k - means clustering algorithm to generate visual word package. [10] All surf feature points’ samples extracted in the training library are {x(t),…,x(m)}, each x(t)64. the generation process is as follows:

• a) Then select 1000 cluster centroid point are μ1, μ1,μ2…,μk ∈. 64,;

• b) To calculate each sample i should belong to the class that is $Ci:=argminj‖x(i)−μj‖2$, recalculating each type μj of the centroid $μj:=∑i−1m{c(i)=j}x(i)∑i−1m{c(i)=j}$ is until convergence.

The training of the classifier. Using visual word package to express the picture of the training set. That is to say, extracting surf feature points of images and mapping it to the corresponding word package. To generate the code of the picture. Inputting into SVM to take training, the process is as follows:

• a) Gaving each sample x(i)1000 mark yi. License plate sample tagged value is 1. The license tag of value is -1;

• b) To select the gaussian kernel function as a conversion function that cast onto the n dimensional vector. Selecting its σ value control the dimension of the projection. Selection of penalty factor C is to optimize decision surface;

• c) All the sample are substituted into type (4) and to calculation, get the classification of the decision function in the form below:

##### (17)
$minω,ξ{12‖ω‖2+C∑i=1nξi},yiω⋅xi−b≥1−ξi,ξi≥0$

In the recognition phase, through visual word package said the image into vector form. Using the trained SVM classifier to classify its license plate images. Locating the license plate location.

## III. CONCLUSION

Aimed at the phenomenon of laminated object in the image, using based on graph search algorithm to obtain the suspicious area of vehicle license plate in the image. Extracting its feature points of surf for coarse positioning of the rectangular profile area. According to generated bag of Visterms represented the candidate images as codebook. Using decisions classification function are obtained by training SVM to classify rectangular area, locating the license plate. This method have higher recognition rate and anti-jamming is strong under complex background. The collection of 140 photos of the result: its accuracy is 135 pieces, accuracy is 96.4%, has the strong robustness.

## References

1. ZhiQiang Li, Yongbin Li. The development and research status of license plate recognition technology. Science & Technology Information, 2012, 5:110–125.
2. Geng Qing-Tian, Zhao Hong-Wei. License plate recognition based on fractal and hidden Markov feature. Optics and Precision Engineering, 2013, 21(12):3198–3203
[CROSSREF]
3. Qin Zhong, Shi Sheng Li, Xu Jianmin et al. Method of license plate location on comer feature[C]// Proceeding of the 6th World Congress on Intelligent Control and Automation Piscataway, NJ, USA: IEEE, Press, 2006: 8645–8649
[CROSSREF]
4. Guo Z Q, Wang Y J, Dong Z, et al. License plate detection method base on paired morphological operator[C]// Proceeding of the 16th National Conference on Image and Graphics. Beijing: Tsinghua University Press, 2011, 292–295.
5. Zhang Yin, Pan Yunhe. A new approach for vehicle license plate locating from color image[J]. Journal of Image and Graphics. 2001, 6(4):374–377.
6. Zheng L H, He X J, Samali B, et al. An algorithm for accuracy enhancement of license plate recognition[J]. Journal of Computer and System Sciences, 2013, 79(2):245–255.
[CROSSREF]
7. Qiwei Wang, Shouhong Wan, Lihua Yue, Che Wang. Visual Attention based Bag-of-Words Model for Image Classification. Sixth International Conference on Digital Image Processing. 2014
8. Corinna Cortes, Vladimir Vapnik. Support-Vector Networks. Machine Learning, 1995, 20(3):273–297
9. He Chunhua, Zhang Xuefei, Hu Yingchun. A study on the improved algorithm for Sobel on image edge detection. OPTICAL TECHNIQUE, 2013, 38(3):324
10. A. Bolovinou, I. Pratikakis, S. Perantonis. Bag of spatio-visual words for context inference in scene classification[J]. Pattern Recognition. 2013, 43(3):1039–1053
[CROSSREF]

### FIGURES & TABLES

Figure 1.

Flow chart of BOW model

Figure 2.

Feature points

Figure 3.

Determine the main direction

Figure 4.

Image segmentation

Figure 5.

The suspicious area extraction

Figure 6.

9*9 filtering template

Figure 7.

The filter size of Octave

Figure 8.

Generate surf feature points

Figure 9.

Surf feature points

### REFERENCES

1. ZhiQiang Li, Yongbin Li. The development and research status of license plate recognition technology. Science & Technology Information, 2012, 5:110–125.
2. Geng Qing-Tian, Zhao Hong-Wei. License plate recognition based on fractal and hidden Markov feature. Optics and Precision Engineering, 2013, 21(12):3198–3203
[CROSSREF]
3. Qin Zhong, Shi Sheng Li, Xu Jianmin et al. Method of license plate location on comer feature[C]// Proceeding of the 6th World Congress on Intelligent Control and Automation Piscataway, NJ, USA: IEEE, Press, 2006: 8645–8649
[CROSSREF]
4. Guo Z Q, Wang Y J, Dong Z, et al. License plate detection method base on paired morphological operator[C]// Proceeding of the 16th National Conference on Image and Graphics. Beijing: Tsinghua University Press, 2011, 292–295.
5. Zhang Yin, Pan Yunhe. A new approach for vehicle license plate locating from color image[J]. Journal of Image and Graphics. 2001, 6(4):374–377.
6. Zheng L H, He X J, Samali B, et al. An algorithm for accuracy enhancement of license plate recognition[J]. Journal of Computer and System Sciences, 2013, 79(2):245–255.
[CROSSREF]
7. Qiwei Wang, Shouhong Wan, Lihua Yue, Che Wang. Visual Attention based Bag-of-Words Model for Image Classification. Sixth International Conference on Digital Image Processing. 2014
8. Corinna Cortes, Vladimir Vapnik. Support-Vector Networks. Machine Learning, 1995, 20(3):273–297
9. He Chunhua, Zhang Xuefei, Hu Yingchun. A study on the improved algorithm for Sobel on image edge detection. OPTICAL TECHNIQUE, 2013, 38(3):324
10. A. Bolovinou, I. Pratikakis, S. Perantonis. Bag of spatio-visual words for context inference in scene classification[J]. Pattern Recognition. 2013, 43(3):1039–1053
[CROSSREF]