Many construction activities generate fine particles and severely threaten the physical health of construction workers. Although many dust control measures are implemented in the industry, the occupational health risks still exist. In order to improve the occupational health level, this study proposes a new method of reducing the construction dust pollution through a reasonable site layout plan. This method is based on the field measurement and dust diffusion law. The dust diffusion law can be fitted based on the field monitoring data. With diffusion law, the average dust concentration exposed to workers of different site layouts can be simulated. In addition, the cost of the dust control method is a concern for site managers. Therefore, the total transportation cost reduction is another optimization objective. Finally, the multi-objective particle swarm optimization (MOPSO) algorithm is used to search for an optimized site layout that can reduce dust pollution and transportation cost simultaneously. The result shows that average dust concentration exposed to workers and total transportation cost are significantly reduced by 60.62% and 44.3%, respectively. This paper quantifies the construction dust pollution and provides site managers with a practical solution to reduce the construction dust pollution at low cost.