The yield of rice is closely related to the number of panicle of rice and the weight of panicle, and accurate prediction of rice yield can accelerate the breeding speed. In order to study the relationship between rice yield and plant phenotypic characteristics, this experiment took potted rice as the research object, using visible light images combined with image processing technology for feature extraction, and obtained 51 phenotypic traits of whole rice. Combined with deep learning technology, the Faster R-CNN convolutional neural network training model was used to detect the number of rice spikes. At the same time, the Rice-PanicleNet model was trained using the SegNet network framework to segment the rice spikes to obtain the binary image of the rice spikes. , Combined with image processing technology to extract 33 phenotypic feature data of the panicle. A total of 85 phenotypic features were extracted from the image. The artificial measurement data in the experiment included the fresh weight and dry weight of potted rice panicle. Normalize all the data to build a prediction model of fresh weight and dry weight of potted rice panicles. Finally, select the most according to the model's decision coefficient R2, average relative error (MAPE) and standard deviation of relative absolute value (SAPE) Excellent prediction model. The prediction results show that the panicle characteristics prediction effect is the best, and the decision coefficients R2 of the predicted value and the real value of the model with the best effect are 0.787±0.051 and 0.840±0.054, respectively. In this study, combined with deep learning, the number of panicle and panicle characteristics that are difficult to obtain automatically by traditional methods are extracted, which provides a new idea and method for rice panicle weight prediction, and further improves the accuracy of rice panicle weight prediction.