A method of detecting diseases and insect pests in litchi based on improved Faster R-CNN was proposed to solve the problems of detecting small targets of diseases and pests in complex backgrounds of litchi orchards. Swin Transformer with superior capabilities of extracting feature was used to replace the original backbone network VGG16 based on Faster R-CNN. The feature pyramid network （FPN） was used to enhance capability of the multi-scale feature fusion in the Faster R-CNN model， thereby improving the precision of identifying each type of diseases and insect pests in litchi in a balanced manner. The ROI Align strategy was introduced to refine the precision of the candidate box localization in the model， leading to the enhancement in the performance of overall detection in the model. The result showed that the average accuracy of the improved model was 92.76%， 30.08 percentage points higher than that of the original Faster R-CNN detector. The precision of detecting images of five types of diseases and insect pests including algal leaf spot， anthracnose， sooty mold， felt disease， and leaf gall in litchi was 93.05%， 94.81%， 96.57%， 87.03%， and 92.34%， respectively. The average precision was improved by 20.50， 5.70， 13.08， and 3.26 percentage points compared with that of SSD512， RetinaNet， EfficientDet-d0， and YOLOv5s model， respectively. It is indicated that the improved Faster R-CNN model can accurately detect diseases and insect pests in litchi with complex backgrounds， and has high value of application. It will provide a reference for studying the rapid and accurate identification of diseases and insect pests in crops.