Abstract: The population structure and spectral structures of plant communities are influenced by both the spatial heterogeneity of abiotic environmental factors and the interactions among organisms within the community. However, the relative importance of different influencing factors remains to be further understood. This study focused on the evergreen broad-leaved forest communities on the north and south slopes of Mt. Guniujiang in the West Huangshan Mountain range. Two 1-hectare permanent plots were established using a total station to spatially locate all tree individuals and determine microenvironmental variables such as elevation and slope. In combination with a single-tree competition model, neighborhood competition index variables were obtained. By fitting generalized linear mixed-effects models and general linear models, multiple factors from two sets of variables (microenvironmental effects and neighborhood competition effects) were used as explanatory variables to measure differences in forest population structure and spectral structure on the north and south slopes. The results show: (1) Within this region, environmental differences between the north and south slopes had a certain impact on the population structure and spectral structure of plant communities, with the north slope community being more species-rich and having closer spectral relationships. (2) The population structure and spectral structure of plant communities were jointly influenced by habitat filtering and neighborhood competition, which together drove the formation of species β diversity within the community. (3) The impact of environmental factors and neighborhood competition factors on population structure and spectral structure varied with life history stages. Trees are mainly influenced by neighborhood effects in the early stages of growth and by microenvironmental factors in the later stages, indicating that their resource acquisition strategies and responses to neighborhood competition effects and environmental factors are also changing. The results of this study can provide a scientific reference for the biodiversity maintenance mechanisms in this region.