For the application of deep learning in the field of Chinese herbal medicine， especially in the field of southern Chinese medicine， the complex background will reduce the accuracy of recognition.If a complex network structure is used， high computing power is required to support training and detection， but the actual embedded or mobile devices are difficult to meet， which affects the effect of on-site detection.This article proposed to improve the EfficientNetv2 network model to classify and identify 8 kinds of southern Chinese medicine leaves in the complex background collected in the field.The network structure was redesigned.The scope of the Fused-MBConv and MBConv architectures was adjusted.Some 3×3 convolutional kernels were replaced with 5×5 convolutional kernels to increase the perceptual field size， reduce the number of convolutional layers of the network， and to further reduce the network complexity.The transfer learning was introduced to train the model.The adaptive moment estimation optimization algorithm was used to optimize the hyperparameters with multiple tests to determine the learning rate.MultiMarginLoss was selected as the loss function to solve the problem of complex background information affecting the accuracy of recognition.The diversity of the data set was increased and the problem of model over-fitting was avoided by adopting data augmentation methods including affine transformation and Gaussian blur and other methods to the experimental data set to improve the stability of the model training process.The results showed that the model improved achieved 99.12% accuracy in recognizing the image of southern Chinese medicine leaves with complex backgrounds， 1.17% more accurate than the baseline model EfficientNetv2-S.The size of parameters was reduced by 85% approximately.The average training time was reduced by 47.62%.The improved model had significant advantages in model storage space， accuracy， and training time， comparing with lightweight models including DenseNet121， ShuffleNet， and RegNet.It is indicated that the model proposed performs well in classifying leaves of multi-variety southern Chinese medicine.The lightweight degree and performance of the model are further improved.