Robotic crop phenotyping has emerged as a key technology for assessing crops’ phenotypic traits at scale, which is essential for developing new crop varieties with the aim of increasing productivity and adapting to the changing climate. However, developing and deploying crop phenotyping robots faces many challenges, such as complex and variable crop shapes that complicate robotic object detection, dynamic and unstructured environments that confound robotic control, and real-time computing and managing big data that challenge robotic hardware/software. This work specifically addresses the first challenge by proposing a novel Digital Twin(DT)/MARS-CycleGAN model for image augmentation to improve our Modular Agricultural Robotic System (MARS)’s crop object detection from complex and variable backgrounds. The core idea is that in addition to the cycle consistency losses in the CycleGAN model, we designed and enforced a new DT/MARS loss in the deep learning model to penalize the inconsistency between real crop images captured by MARS and synthesized images generated by DT/MARS-CycleGAN. Therefore, the synthesized crop images closely mimic real images in terms of realism, and they are employed to fine-tune object detectors such as YOLOv8. Extensive experiments demonstrate that the new DT/MARS-CycleGAN framework significantly boosts crop/row detection performance for MARS, contributing to the field of robotic crop phenotyping. We release our code and data to the research community (https://github.com/UGA-BSAIL/DT-MARS-CycleGAN).
Three-dimensional (3D) plant phenotyping techniques measure organ-level traits effectively and provide detailed plant growth information to breeders. In apple tree breeding, architectural traits can determine photosynthesis efficiency and characterize the developmental stages of trees. The overall goal of this study was to develop a deep learning-based organ-level instance segmentation method to quantify the 3D architectural traits of apple trees. This study utilized PointNeXt for the semantic segmentation of apple tree point clouds, classifying them into trunks and branches, and benchmarked its performance against several competitive models, including PointNet, PointNet++, and Point Transformer V2 (PTv2). A cylinder-based constraint method was introduced to refine the semantic segmentation results. Next, the branches were identified with the density-based spatial clustering of applications with noise (DBSCAN) algorithm. The type of 3D skeleton vertices determined whether a cluster represented a single branch or multiple branches. If multiple, a graph-based technique further separated them. This study also directly applied the instance segmentation model SoftGroup++ to the apple tree point clouds and analyzed the segmentation results on the apple tree dataset. Finally, seven architectural traits of apple trees were extracted, including height, volume, and crown width of the tree, as well as height and diameter for the trunk, and length and count for the branches. The experimental results showed that the post-processed mIoU values for PointNet, PointNet++, PTv2, and PointNeXt were 0.8495, 0.8535, 0.9500, and 0.9481, respectively. The final instance segmentation results based on SoftGroup++ and PointNeXt achieved mAP_50 of 0.815 and 0.842, respectively. For traits such as tree height, trunk length and diameter, branch length, and branch count, the method based on PointNeXt achieved R2 values of 0.987, 0.788, 0.877, 0.796, and 0.934, with mean absolute percentage errors of 0.86 %, 2.17 %, 5.93 %, 10.24 %, and 13.55 %, respectively. The segmentation results of PTv2 and SoftGroup++ were also used to extract the phenotypic traits of apple trees, achieving results comparable to those of PointNeXt. The proposed method demonstrates a cost-effective and accurate approach for extracting the architectural traits of apple trees, which will benefit apple breeding programs as well as the precision management of apple orchards.
Thank you to all our lab members who continue to do great work!
No responses yet