A improved Faster RCNN algorithm was proposed to solve the difficulties of identifying tea leaf diseases under the complex background of tea gardens. VGG-16 and MobileNetV2 networks were extracted by the features of optimized regional recommendation boxes and compared with ResNet50 network. ResNet50 with good performance of identification was selected as the backbone network to enhance the model's ability to extract features of tea leaf diseases under the complex background of tea gardens. The feature pyramid network （FPN） was integrated to improve the problem of missing detection of small targets and multi-scale lesions. The Rank and Sort （RS） Loss function was used to replace the loss function in the original Faster RCNN， which alleviated the impact of uneven sample distribution on the performance of model and further improved the accuracy of identification. The results showed that the mean average precision P_mA and the identification speed of the model improved was 88.06% and 19.1 frames/s. The average precision value of identifying algal spot， white scab， anthracnose and sooty mold was 75.54%， 86.84%， 90.42% and 99.45%， respectively. The average precision value of identifying with improved Faster RCNN algorithm was 40.98， 44.16， 13.9 and 2.43 percentages points higher than that with Faster RCNN algorithm. It is indicated that the method for identifying leaf diseases of tea based on the improved Faster RCNN algorithm can weaken the interference of complex background of tea garden and accurately identify leaf disease target of tea under the complex background of tea garden.