  Aggregating neighbor features is essential for point cloud classification. In
the existing work, each point in the cloud may inevitably be selected as the
neighbors of multiple aggregation centers, as all centers will gather neighbor
features from the whole point cloud independently. Thus each point has to
participate in the calculation repeatedly and generates redundant duplicates in
the memory, leading to intensive computation costs and memory consumption.
Meanwhile, to pursue higher accuracy, previous methods often rely on a complex
local aggregator to extract fine geometric representation, which further slows
down the classification pipeline. To address these issues, we propose a new
local aggregator of linear complexity for point cloud classification, coined as
APP. Specifically, we introduce an auxiliary container as an anchor to exchange
features between the source point and the aggregating center. Each source point
pushes its feature to only one auxiliary container, and each center point pulls
features from only one auxiliary container. This avoids the re-computation
issue of each source point. To facilitate the learning of the local structure
of cloud point, we use an online normal estimation module to provide the
explainable geometric information to enhance our APP modeling capability. Our
built network is more efficient than all the previous baselines with a clear
margin while still consuming a lower memory. Experiments on both synthetic and
real datasets demonstrate that APP-Net reaches comparable accuracies to other
networks. It can process more than 10,000 samples per second with less than
10GB of memory on a single GPU. We will release the code in
https://github.com/MCG-NJU/APP-Net.
