Manual sorting of citrus fruit with visual defects on the production line is time-consuming and cost-expensive.This article proposes a sorting solution based on machine vision and CNN-Transformer.The system can be directly implemented on various citrus processing lines for online sorting.For the citrus fruits randomly rotating on the conveyor，a detection algorithm Mobile-citrus based on convolutional neural network （CNN） was developed to detect and temporarily classify the defective one.A tracking algorithm Tracker-citrus was used to record the classification information along the path.The real category of the fruit was identified using the historical information，with tracking accuracy of 98.4% and classification accuracy of 92.8%.In addition，a trajectory prediction algorithm based on Transformer was used to predict the future path of fruit with the average prediction error of 2.98 pixels，which can be used to guide the robot arm to sort defective citrus in real time.The results showed that the method proposed can be applied to citrus production lines for online sorting.