---
abstract: 'We propose an attention-injective deformable convolutional network called for crowd understanding that can address the accuracy degradation problem of highly congested noisy scenes. contains two concatenated networks. An attention-aware network called Attention Map Generator (AMG) first detects crowd regions in images and computes the congestion degree of these regions. Based on detected crowd regions and congestion priors, a multi-scale deformable network called Density Map Estimator (DME) then generates high-quality density maps. With the attention-aware training scheme and multi-scale deformable convolutional scheme, the proposed achieves the capability of being more effective to capture the crowd features and more resistant to various noises. We have evaluated our method on four popular crowd counting datasets (ShanghaiTech, UCF\_CC\_50, WorldEXPO’10, and UCSD) and an extra vehicle counting dataset TRANCOS, and our approach beats existing state-of-the-art approaches on all of these datasets.'
author:
- |
    Ning Liu$^{1, 2}$     Yongchao Long$^{1, 2}$     Changqing Zou$^3$     Qun Niu$^{1, 2}$     Li Pan$^4$     Hefeng Wu$^{1, 5, 6\,}$[^1]\
    $^1$School of Data and Computer Science, Sun Yat-sen University\
    $^2$Guangdong Key Laboratory of Information Security Technology $^3$University of Maryland\
    $^4$Shanghai Jiao Tong University $^5$Guangdong University of Foreign Studies $^6$WINNER Technology\
    [liuning2@mail.sysu.edu.cn, {longych3, niuqun}@mail2.sysu.edu.cn, ]{}\
    [cqzou@umiacs.umd.edu, panli@sjtu.edu.cn, wuhefeng@gmail.com]{}
bibliography:
- 'mybib.bib'
title: 'ADCrowdNet: An Attention-Injective Deformable Convolutional Network for Crowd Understanding'
---

Introduction {#sec:intro}
============

Crowd understanding has attracted much attention recently because of its wide range of applications like public safety, congestion avoidance, and flow analysis. The current research trend for crowd understanding has developed from counting the number of people to displaying distribution of crowd through density map. Generally, generating accurate crowd density maps and performing precise crowd counting for highly congested noisy scenes is challenging due to the complexity of crowd scenes caused by various factors including background noises, occlusions, and diversified crowd distributions.

![image](f1.png){width="79.00000%"}

Researchers recently have leveraged deep neural networks (DNN) for accurate crowd density map generation and precise crowd counting. Although these DNNs-based methods [@zhang2016single; @sam2017switching; @sindagi2017generating; @liu2018crowd; @CaoWZS18eccv; @li2018csrnet] have made significant success in solving the above issues, they still have the problem of accuracy degradation when applied in highly congested noisy scenes. As shown in Figure \[fig:sample\], the state-of-the-art approach [@li2018csrnet], which has achieved much lower Mean Absolute Error (MAE) than the previous state-of-the-art methods, is still severely affected by background noises, occlusions, and non-uniform crowd distributions.

In this paper, we aim at an approach which is capable of dealing with highly congested noisy scenes for the crowd understanding problem. To achieve this, we designed an attention-injective deformable convolutional neural network called which is empowered by a visual attention mechanism and a multi-scale deformable convolution scheme. The visual attention mechanism is delicately designed for alleviating the effects from various noises in the input. The multi-scale deformable convolution scheme is specially introduced for the congested environments. The basic principle of visual attention mechanism is to use the pertinent information rather than all available information in the input image to compute the neural response. This principle of focusing on specific parts of the input has been successfully applied in various deep learning models for images classification [@hu2017squeeze], semantic segmentation [@ren2017end], image deblurring [@qian2018attentive], and visual pose estimation [@chu2017multi], which also suits our problem where the interest regions containing the crowd need to be recognized and highlighted out from noisy scenes. The multi-scale deformable convolution scheme takes as input the information of the dynamic sampling locations, other than evenly distributed locations, which has the capability of modeling complex geometric transformation and diverse crowd distribution. This scheme fits well the nature of the distortion caused by the perspective view of the camera and diverse crowd distributions in real world, therefore guaranteeing more accurate crowd density maps for the congested scenes.

To incorporate the visual attention mechanism and deformable convolution scheme, we leverage an architecture consisting of two neural networks as shown in Figure \[fig:overviewArch\]. Our training contains two stages. The first stage generates an attention map for a target image via a network called Attention Map Generator (AMG). The second stage takes the output of AMG as input and generates the crowd density map via a network called Density Map Estimator (DME). The attention map generator AMG mainly provides two types of priors for the DME network: 1) candidate crowd regions and 2) the congestion degree of crowd regions. The former prior enables the multi-scale deformable convolution scheme empowered DME network to pay more attention to those regions having people crowds, and thus improving the capacity of being resistant to various noises. The latter prior indicates each crowd region with congestion degree (i.e., how crowded each crowd region is), which provides fine-grained congestion context prior for the subsequent DME network and boosts the performance of the DME network on the scenes containing diverse crowd distribution.

The main contributions of this paper are summarized as follows. First, a novel attention-injective deformable convolutional network framework is proposed for crowd understanding. Second, our AMG model that attends the crowd regions in images, is innovatively formulated as a binary classification network by introducing third party negative data (i.e., background images with no crowds). Third, our DME model can estimate the crowds effectively by using the proposed structure of aggregating multi-scale deformable convolution representations. Furthermore, extensive experiments conducted on all popular datasets demonstrate the superior performance of our approach over existing leading ones. In particular, the proposed model outperforms the state-of-the-art crowd counting solution CSRNet [@li2018csrnet] with 3.0%, 18.8%, 3.0%, 13.9% and 5.1% lower Mean Absolute Error (MAE) on ShanghaiTech Part\_A, Part\_B, UCF\_CC\_50, WorldExpo’10, UCSD datasets, respectively. Apart from crowd counting, is also general for other counting tasks. We have evaluated on a popular vehicle counting dataset named TRANCOS [@guerrero2015extremely], and achieves 32.8% lower MAE than CSRNet.

![image](overview.pdf){width="99.00000%"}

\[fig:overviewArch\]

Related Work {#sec:related}
============

**Counting by detection:** Early approaches of crowd understanding mostly focus on the number of people in crowds [@DollarWSP12]. The major characteristics of these approaches are the sliding window based detection scheme and hand crafted features extracted from the whole human body or particular body parts with low-level descriptors like Haar wavelets [@viola2004robust] and HOG [@dalal2005histograms]. Generally, approaches in these groups deliver accurate counts when their underlying assumptions are met but are not applicable in more challenging congested scenes.

**Counting by regression:** Counting by regression approaches differs depending on the target of regression: object count [@ChenLGX12; @ChenGXL13], or object density [@lempitsky2010learning]. This group of approaches avoid solving the hard detection problem. Instead, they deploy regression model to learn the mapping between image characteristics (mainly histograms of lower level or middle level features) and object count or density. These approaches that directly regress the total object count discard the information of the location of the objects and only use 1-dimensional object count for learning. As a result, a large number of training images with the supplied counts are needed in training. Lempitsky  [@lempitsky2010learning] propose a method to solve counting problem by modeling the crowd density at each pixel and cast the problem as that of estimating an image density whose integral over any image region gives the count of objects within that region. Since the ideal linear mapping is hard to obtain, Pham  [@pham2015count] use random forest regression to learn a non-linear mapping instead of the linear one.

**Crowd understanding by CNN:** Inspired by the great success in visual classification and recognition, literature also focuses on the CNN-based approaches to predict crowd density map and count the number of crowds [@walach2016learning; @onoro2016towards; @LiHH2018arXiv; @KangC18bmvc; @qiu2019crowd]. Walach  [@walach2016learning] use CNN with a layered training structure. Shang [@shang2016end] adapt an end-to-end CNN which uses the entire images as input to learn the local and global count of the images and ultimately outputs the crowd count. A dual-column network combining shallow and deep layers is used in [@boominathan2016crowdnet] to generate density maps. In [@zhang2016single], a multi-column CNN is proposed to estimate density map by exacting features at different scales. Similar idea is used in [@onoro2016towards]. Marsden  [@marsden2016fully] try a single-column fully convolutional network to generate density map while Sindagi  [@sindagi2017cnn] present a CNN that uses high-level prior to boost accuracy.

More recently, Sindagi  [@sindagi2017generating] propose a multi-column CNN called CP-CNN that uses context at various levels to improve generate high-quality density maps. Li  [@li2018csrnet] propose a model called CSRNet that uses dilated convolution to enlarge receptive fields and extract deeper features for boosting performance. These two approaches have achieved the state-of-the-art performances.

Attention-Injective Deformable Convolutional Network {#sec:detail}
====================================================

The architecture of the proposed method is illustrated in Figure \[fig:overviewArch\]. It employs two concatenated networks: AMG and DME. AMG is a classification network based on fully convolutional architecture for attention map generation, while DME is a multi-scale network based on deformable convolutional layers for density map generation. Before training DME, we train the AMG module with crowd images (positive training examples) and background images (negative training examples). We then use the well-trained AMG to generate the attention map of the input image. Afterward, we train the DME module using the pixel-wise product of input images and the corresponding attention maps. In the following sections, we will detail the architectures of the AMG and DME netwroks.

\[fig:ddlevel\]

Attention Map Generator
-----------------------

### Attention map

Attention map is an image-sized weight map where crowd regions have higher values. In our work, attention map is a feature map from a two-category classification network AMG which classifies an input image into crowd image or background image. The idea of using feature map to find the crowd regions in the input is motivated by an object localization work [@zhou2016learning] which points out that the feature maps of classification network contain the location information of target objects.

The pipeline of the attention map generation is shown in Figure \[fig:AMG\]. $F_c$ and $F_b$ are the feature maps from the last convolution layer of AMG. $W_c$ and $W_b$ are the spatial average of the $F_c$ and $F_b$ after global average pooling (i.e., GAP in Figure \[fig:AMG\]). $P_c$ and $P_b$ are confidence scores of the predicted two class. They are generated by softmax from $W_c$ and $W_b$. The attention map is obtained by up-sampling the linear weighted fusion of the two feature maps $F_c$ and $F_b$ (i.e., $F_c \cdot P_c + F_b \cdot P_b$) to the same size as the input image. We also normalize the attention map such that all element values fall in the range $[0, 1]$.

The attention map highlights the regions of crowds. In addition, it also indicates the degree of congestion in individual regions, i.e., higher congestion degree values indicate more congested crowds and lower values indicate less congested ones. Figure \[fig:ddlevel\] illustrates the effect of attention maps at different density levels. The pixel-wise product between the attention map and the input image produces the input data used by the DME network.

### Architecture of attention map generator

The architecture of AMG is shown in Figure \[fig:AMG\], we use the first 10 layers of trained VGG-16 model [@simonyan2014very] as the front end to extract low-level features. We build the back end by adopting multiple dilated convolution layers of different dilation rates with an architecture similar to the inception module in [@szegedy2015going]. The multiple dilated convolution architecture is motivated from [@wei2018revisiting]. It has the capability of localizing people clusters with enlarged receptive fields. The inception module was originally proposed in [@szegedy2015going] to process and aggregate visual information of various scales. We use this module to deal with the diversified crowd distribution in congested scenes.

![Architecture of AMG. All convolutional layers use padding to maintain the previous size. The convolutional layers’ parameters are denoted as “Conv-(kernel size)-(number of filters)-(dilation rate)”, max-pooling layers are conducted over a 2$\times$2 pixel window, with stride 2.](AMG.pdf){width="46.00000%"}

\[fig:AMG\]

Density Map Estimator
---------------------

The DME network consists of two components: the front end and the back end. We remove the fully-connected layers of VGG-16 [@simonyan2014very] and leave 10 convolutional layers to as the front end of the DME. The back end is a multi-scale deformable convolution based CNN network [@dai2017deformable]. The architecture of DME is shown in Figure \[fig:DMEArchi\]. The front end uses the first 10 layers of trained VGG-16 model [@simonyan2014very] to extract low-level features. The back end uses multi-scale deformable convolutional layers with a structure similar to the inception module in [@szegedy2015going], which enables DME to cope with various occlusion, diversified crowd distribution, and the distortion caused by perspective view.

![Architecture of DME. The convolutional layers’ parameters are denoted as “Conv-(kernel size)-(number of filters)-(stride)”, max-pooling layers are conducted over a 2$\times$2 pixel window, with stride 2. The deformable convolutional layers’ parameters are denoted as [“Dconv-(kernel size)-(number of filters)-(stride)”]{}.[]{data-label="fig:DMEArchi"}](DME.pdf){width="46.00000%"}

The deformable convolution scheme was originally proposed in [@dai2017deformable]. Beneficial from the adaptive (deformable) sampling location selection scheme, deformable convolution has shown its effectiveness on various tasks, such as object detection, in the wild environment. The deformable convolution treats the offsets of sampling locations as learning parameters. Rather than uniform sampling, the sampling locations in the deformable convolution can be adjusted and optimized via training (see Figure \[fig:dcov\_example\] for the deformed sampling points by the deformable convolution on an example form ShanghaiTech Part\_B dataset  [@zhang2016single]). Compared to the uniform sampling scheme, this kind of dynamic sampling scheme is more suitable for the crowd understanding problem of congested noisy scenes. We will show the comparative advantages of the deformable convolution in our experimental section.

![Illustration of the deformed sampling locations. Left: standard convolution; right: deformable convolution; top: activation units on the feature map; bottom: the sampling locations of the $3 \times 3$ filter.[]{data-label="fig:dcov_example"}](DCNN.pdf){width="40.00000%"}

Experiments {#sec:exp}
===========

Datasets and Settings
---------------------

We evaluate on four challenging datasets for crowd counting: ShanghaiTech dataset [@zhang2016single], the UCF\_CC\_50 dataset [@idrees2013multi], the WorldExpo’10 dataset [@zhang2015cross], and the UCSD dataset [@chan2008privacy].

**ShanghaiTech dataset** [@zhang2016single]. The ShanghaiTech dataset contains 1,198 images with a total of 330,165 people. It is divided into two parts: Part\_A and Part\_B. Part\_A contains 482 pictures of congested scenes, in which 300 are used as training dataset and 182 are used as testing dataset; Part\_B contains 716 images of sparse scene, 400 of which are used as training dataset and 316 are used as testing dataset.

**UCF\_CC\_50 dataset** [@idrees2013multi]. This dataset contains 50 images downloaded from the Internet. The number of persons per image ranges from 94 to 4543 with an average of 1280 individuals. It is a very challenging dataset with two problems: the limited number of the images and the large span in person count between images. We used 5-fold-cross-validation setting described in [@idrees2013multi].

**WorldExpo’10 dataset** [@zhang2015cross]. It contains 3980 from 5 different scenes. Among 3980 images, 3380 images are used as training dataset and the remaining 600 images are used as testing dataset. Region-of-Interest (ROI) regions are provided in this dataset.

**UCSD dataset** [@chan2008privacy]. The UCSD dataset contains 2000 images in sparse scene. The dataset also provides ROI region information. We created the ground truth in the same way as we did for the WorldExpo’10 dataset. Since the size of each image is too small to support the generation of high-quality density maps, we therefore enlarge each image to 952$\times$632 size by bilinear interpolation. Among the 2000 images, 800 images were used as training dataset, and the rest were used as testing dataset. Region-of-Interest (ROI) regions are also provided in this dataset.

We show a representative example for each crowd counting dataset in Figure \[fig:examples\]. These four crowd counting datasets have their own characteristics. In general, the scenes in ShanghaiTech Part\_A dataset are congested and noisy. Examples in ShanghaiTech Part\_B are noisy but not highly congested. The UCF\_CC\_50 dataset consists of extremely congested scenes which have hardly any background noises. Both WorldExpo’10 dataset and UCSD dataset provide example with sparse crowd scenes in the form of ROI regions. Scenes in the ROI regions of the WorldExpo’10 dataset are generally noisier than the only one scene in the UCSD dataset.

Following [@sindagi2017generating; @li2018csrnet], we use the mean absolute error (MAE) and the mean square error (MSE) for quantitative evaluation of the estimated density maps. PSNR (Peak Signal-to-Noise Ratio) and SSIM [@wang2004image] are used to measure the quality of the generated density map. For fair comparison, we follow the measurement procedure in [@li2018csrnet] and resize the density map and ground truth to the size of the original input image by linear interpolation.

![image](datasets.pdf){width="90.00000%"}

Training
--------

### AMG Training

Training data for the binary classification network AMG consists of two groups of samples: positive and negative samples. The positive samples are from the training sets of the four crowd counting datasets. The negative samples are 650 background images downloaded from the Internet. These negative samples are shared by the training of each individual dataset. These 650 negative samples contain various outdoor scenes where people appear, such as streets, squares, etc., ensuring that the biggest difference between positive sample and negative samples is whether the image contains people. Adam [@kingma2014adam] is selected as the optimization method with the learning rate at 1e-5 and Standard cross-entropy loss is used as the loss function.

### DME training

We simply crop 9 patches from each image where each patch is 1/4 of the original image size. The first four patches contain four quarters of the image without overlapping. The other five patches are randomly cropped from the image. After that, we mirror the patches so that we double the training dataset. We generate the ground truth for DME training following the procedure in [@li2018csrnet]. We select Adam [@kingma2014adam] as the optimization method with the learning rate at 1e-5. As previous works [@zhang2016single; @sam2017switching; @li2018csrnet], we use the euclidean distance to measure the difference between the generating density map and ground truth and define the loss function as

$$L(\Theta)=\frac{1}{2N}\sum_{i=1}^N ||F(X_i;\Theta)-F_i||_{2}^2\eqno(2),$$ where $N$ is the batch size, $F(X_i;\Theta)$ is the estimated density map generated by DME with the parameter$\Theta$, $X_i$ is the input image, and $F_i$ is the ground truth of $X_i$.

Results and Analyses
--------------------

In this section, we first study several alternative network design of . After that, we evaluate the overall performance of and compare it with previous state-of-the-art methods.

### Alternative study

\[fig:DMEVSAMG\]

**DME**. Our first study is to investigate the influence of the AMG network, we compared two network designs on all the four datasets. The first one named AMG-DME has the architecture shown in Figure \[fig:overviewArch\]. The other one named DME uses the only DME network. Our quantitative experimental results in Table \[tab:ablation\] show that AMG-DME is significantly superior than DME on the those datasets which are characteristic of noisy scenes: ShanghaiTech Part\_A, Part\_B and WorldExpo’10. In Figure \[fig:DMEVSAMG\], we illustrate two representative samples from the testing set of ShanghaiTech Part\_A. On the top example which contains a congested noisy scene, estimated people number of AMG-DME is 198 that is much closer to the groud truth 171 than that estimated by DME. From the density map in the 3rd column of Figure \[fig:DMEVSAMG\], we can see the trees in the distance have been recognized as people by the single DME model. However, AMG-DME does not suffer this problem due to the help from the AMG network. On the middle-row example containing a noisy and more congested scene, the performances of AMG-DME and DME agree with those on the top example. The comparison results indicate that AMG-DME is more effective than DME on those noisy examples.

On the UCF\_CC\_50 dataset, AMG-DME has approximate performance (slightly higher AME but lower MSE) with DME. It may due to the fact most of examples in the UCF\_CC\_50 dataset have a large regions of congested crowds while rarely have background noises. On the UCSD dataset where scenes are neither congested nor noisy, both MSE and MAE of AMG-DME is slightly higher than DME. This might because the examples in the UCSD dataset have already provide the accurate information of ROI regions. The attention map generated by the AMG network may destroy the ROI regions, which degrades the performance of the DME network since some ROI regions may be erased from its input.

**AMG-bAttn-DME**. Since the AMG network has shown its strength in coping with noise background of scenes, our second study is to explore if a hard binary attention mask is more effective than the soft attention employed by AMG-DME. We therefore set up an variant of AMG-DME called AMG-bAttn-DME in Table \[tab:ablation\]. AMG-bAttn-DME has the same architecture as AMG-DME while differing with AMG-DME on the attention map (i.e., the attention maps of AMG-bAttn-DME contain either 0 or 1, other than a floating point within $[0,1]$ in the attention maps of AMG-DME). We first conducted the experiments on the ShanghaiTech dataset to find out the optimal binarization threshold for AMG-bAttn-DME. We set three different threshold attention values,$\{0.2, 0.1, 0.0\}$, for the binarization of attention maps. The ROI regions are gradually enlarged with the decreasing the threshold values as shown in Figure \[fig:partA\_seg\]. The results shown in Table \[tab:sh\_thre\] indicates AMG-bAttn-DME with attention threshold of 0.1 achieved the best performance. We then evaluated AMG-bAttn-DME with this optimal attention threshold on the rest three datasets and reported the results in Table \[tab:ablation\]. It is observed that AMG-bAttn-DME is superior than AMG-DME only on ShanghaiTech Part\_A while AMG-DME outperforms AMG-bAttn-DME on all other datasets. It may be due to the AMG network can learn more accurate attention maps on ShanghaiTech Part\_A and the binarization process does not destroy too much information of the crowd regions.

  ----------- ------ ------- ----- ------
   Threshold   MAE     MSE    MAE   MSE
    t = 0.2    68.0   104.1   9.2   17.8
    t = 0.1    63.2   98.9    8.2   15.7
    t = 0.0    63.2   100.6   8.6   15.0
  ----------- ------ ------- ----- ------

  : Performance of AMG-bAttn-DME under different binarization thresholds on the ShanghaiTech dataset.[]{data-label="tab:sh_thre"}

![Architecture of AMG-attn-DME in both training and testing phases.](model_v4.png){width="45.00000%"}

\[fig:model\_AMG-attn-DME\]

**AMG-attn-DME**. Complement to the above experiments, we stretched the design choice exploration to studying an alternative way of injecting the learned attention from the AMG network to the DME network. In our proposed architecture, the DME network directly takes the crowd images as input. An alternative architecture is to weigh intermediate the feature map of a certain layer of the DME network with the attention map from the AMG network. In our implementation, we inject the attention map into the output of the front end of the DME network as shown in Figure \[fig:model\_AMG-attn-DME\]. Following the same training procedures as those in Table \[tab:ablation\], this alternative architecture, named AMG-attn-DME, performs slightly worse than AMG-DME on the datasets with congested noisy scenes like ShanghaiTech Part\_A and ShanghaiTech Part\_B. This may be due to some non-crowd pixels in the attention map from the AMG network having an attention value of zero, which, during the injection, would make convolution features at those corresponding locations vanish, reducing the feature information learned by previous convolutional lays from the input. On the UCF\_CC\_50 dataset and UCSD dataset, AMG-attn-DME is worse than the the only DME network as AMG-bAttn-DME and AMG-DME. This is because the scenes of these two datasets have less noisy background, AMG-attn-DME may reduce the information of the ROI regions through the injected attention map. On the UCSD and WorldExpo’10 datasets, AMG-attn-DME achieved higher effectiveness. Maybe it is because the convolution feature vanishing problem has been alleviated by the black regions around the ROI regions in the input.

### Quantitative results

In this section, we study the overall performance of and compare it with existing methods on each individual crowd counting dataset.

**Comparison on MAE and MSE**. We first compare the variants of the proposed network with the state-of-the-art work CSRNet [@li2018csrnet] along with several previous methods including CP-CNN [@sindagi2017generating], MCNN [@zhang2016single], Cascaded-MTL [@sindagi2017cnn], Switching-CNN [@sam2017switching] on the ShanghaiTech dataset and the UCF\_CC\_50 dataset. These two datasets are characteristic of congested and/or noisy scenes. The comparison results were summarized in Table \[tab:SHTPA\]. On the ShanghaiTech dataset, two of our approach variants (AMG-DME) and (AMG-bAttn-DME) achieved better performances than existing approaches. The only DME network achieved the performance generally close to the state-of-the-art approach CSRNet [@li2018csrnet].

On the two relatively less challenging datasets WorldExpo’10 and UCSD, we compared with recent state-of-the-art recent approaches including Switching-CNN [@sam2017switching], MCNN [@zhang2016single], and CSRNet [@li2018csrnet]. The comparison results are shown in Table \[tab:UCSD\]. Our method achieved the best accuracy in scenes 1, 4, 5 as well as the best average accuracy on the WorldExpo’10 dataset . On the UCSD dataset, our DME model achieved the best accuracy on terms of both MAE and MSE.

**Comparison on PSNR and SSIM.** To study the quality of the density maps generated by , another experiment was conducted on all the five datasets for both and the state-of-the-art method CSRNet [@li2018csrnet]. The comparison results are shown in Table \[tab:qulaityCSR\]. Our method outperforms CSRNet on all the five datasets. On UCF\_CC\_50 dataset, our method improves 7.03% on PSNR and 55.76% on SSIM. On USCD dataset, our method improves 31.81% on PSNR and 8.13% on SSIM.

**[Evaluation on vehicle counting dataset.]{}** We conducted experiments on the TRANCOS [@guerrero2015extremely] dataset for vehicle counting to evaluate the generalization capability of the proposed approach. The positive samples for training are from the training set of TRANCOS [@guerrero2015extremely]. The negative samples use 250 background images downloaded from the Internet, including various road scenes without vehicle. As previous work CSRNet [@li2018csrnet], we use the Grid Average Mean Absolute Error (GAME) to measure the counting accuracy. The comparison results are shown in Table \[tab:TRANCOS\]. It clearly shows that the approach achieved the best performance at all levels of GAMEs.

### Qualitative results

In this section, we further investigate the general performance of the proposed by qualitative results. We mainly compared with the state-of-the-art approach CSRNet [@li2018csrnet] which have demonstrated the best performance on the datasets including the ShanghaiTech, UCF\_CC\_50, the WorlExpo’10, and UCSD datasets. In general, CSRNet has a front-end and back-end architecture as the DME network of the proposed . It is empowered by a dilated convolution design in the back-end of its architecture. Apart from the additional AMG netwok, differs from CSRNet by two additional features in its DME network: 1) the multiple-scale convolution scheme different from the single scale scheme of CSRNet, and 2) the deformable sampling scheme different from the evenly fixed-offset sampling in the dilated convolution of CSRNet.

Figure \[fig:Qualitative\] shows some qualitative comparisons between the proposed (the variant AMG-DME is used) and the state-of-the-art approach CSRNet [@li2018csrnet]. Through visualization, it is observed that CSRNet is much less effective on those examples with various noises than . We can see the evidence from the noise regions marked by red boxes of the 1st column where noises exist in the background regions, as well as the marked regions of the 3rd column where noises can be found in the crowd regions. This may be due to CSRNet directly takes the crowd image as input while the DME network of takes as the input the crowd information highlighted by its AMG network. On the example of the 2nd column where there is not much noise but a significantly non-uniform crowd distribution, also clearly outperforms CSRNet. This indicates that the multi-scale deformable convolution scheme in is more effective than the single-scale fixed-offset dilated convolution scheme in CSRNet.

On the rightmost example of Figure \[fig:Qualitative\] which have highly occluded crowd regions (see the regions within the two green dotted bordered rectangle), only recognized part of the severely occluded crowd regions. It may because the AMG network of cannot highlight out the whole occluded crowd regions for the DME network. Nevertheless, still achieved better performance in terms of all the measurement parameters: estimated number, PSNR and SSIM.

![ From top to bottom: representative samples from the testing set of the ShanghaiTech dataset, ground truth density maps, estimated density maps generated by the state-of-the-art approach CSRNet [@li2018csrnet] and (AMG-DME) respectively. []{data-label="fig:Qualitative"}](quanlity_result.png){width="49.00000%"}

Conclusion {#sec:conclusion}
==========

We propose a convolutional neural network based architecture named for crowd understanding of congested noisy scenes. Benefiting from the multi-scale deformable convolutional layers and [[attention-aware]{}]{} training scheme, generally achieved more accurate crowd counting and density map estimation than existing methods by suppressing the problems caused by noises, occlusions, and diversified crowd distributions commonly presented in highly congested noisy environments. On four popular crowd counting datasets (ShanghaiTech, UCF\_CC\_50, WorldEXPO’10, UCSD) and an extra vehicle counting dataset TRANCOS, achieved significant improvements over recent state-of-the-art approaches.

[^1]: The corresponding author is Hefeng Wu. This research is supported by the National Natural Science Foundation of China (Grant No. 91746204 and 61876045), and Opening Project of Guangdong Province Key Laboratory of Information Security Technology (Grant No. 2017B030314131).
