NRI: INT: COLLAB: TREE FRUIT HARVESTING WITH ARRAYS OF VISION-GUIDED LINEAR ROBOT ARMS

Research Goal 1: Development of Linear Robot Arm ArraysThis goal includes the development of a simulator for the mult-arm harvester using digitized trees and fruits and of algorithms to compute the optimal coordinated motion trajectories for the robot arms. An economic model for robotic harvesting will also be developed.Objective 1.1: Modeling and Simulation of LRA HarvesterThis technical objective aims at developing: a model for LRAs with several arms in each cell; an integrated model for an LRA on a carrier platform, and a harvesting model of the system operating in a digitized orchard. The model will capture the essential kinematics and dynamics of the system.Objective 1.2: Fruit-Picking Order Assignments for LRAsThis objective aims at developing algorithms to compute the motion trajectories of the multiple arms, which are dynamic; incorporate uncertainty about fruit location and presence; minimize picking cycle times, and execute in real-time.Objective 1.3: Economic analysisThe goal of this objective is to develop an economic model of robotic harvesting that will be used to guidethe design and functional parameters of the machine but will also be available for other researchers, startup companies and investors to use when exploring commercial robotic harvest technologies.Objective 1.4: Engineering Design and Development of LRAIn addition to significant design issues, this technical objective covers the detailed CAD design of the LRA system (telescopic arm, mobile arm base module, air nozzles - with CMU, frame); fabrication of needed components; integration of machine, actuators, pneumatic, electrical and electronic systems (hardware); development and tuning of the low-level motion control software; integration of perception system, fruit- picking assignment (high-level control) and motion control system. As the last step, the LRA will be installed on the orchard platform. Given our budgetary constraints, the goal is to build 15 arms and compare their performance in orchards against the predicted PCT vs. number-of-arms curve.Research Goal 2: Increase Fruit Visibility and DetectionA combination of novel approaches that include active foliage agitation via controlled air streams, deep learning-based partially occluded fruits detection, and multi-view imaging system are formulated under this goal to significantly increase fruit visibility in canopies of trellised and hedged trees.Objective 2.1: Fruit detection systemRobust detection and localization of fruits is an essential component for an automated fruit harvester. We plan to use the current state- of-the-art, Mask R-CNN as a multi-class detector to instantly detect and segment not only clearly visible and partially occluded fruits but also the foliage in the scene.Objective 2.2: Improved Fruit Visibility via Multi-View GeometryThis objective aims at detecting occluded fruits using a multi-view approach that accounts for occlusion and multiple fruit count. Additionally, dynamic changes in the set of fruits to be picked will be tracked.Objective 2.3: Increased Fruit Visibility via Foliage AgitationThis objective aims at designing, building and testing a novel tree foliage agitation system that increases fruit visibility in a sequence of images or a video stream. More specifically, by causing leaves to move it is expected that different parts of fruit surface will become visible over a series of successive image frames.Research Goal 3: System Integration and EvaluationThis goal covers the integration of hardware and software used for the perception and actuation systems, as well as the evaluation of the functional system in laboratory and real-world conditions in commercial orchards.Objective 3.1: Integration of actuation and perception systemsThe information flows between the perception and actuation subsystems will be defined and implemented to achieve their integration. The computational pipeline for feeding target 3D points (fruit centers) - with corresponding detection probabilities - from the cameras to the LRA's picking order assignment algorithm will be established.Objective 3.2: Evaluation of LRA actuation systemEvaluations of the LRA actuation system (without perception) will be performed in laboratory conditions.Objective 3.3: Evaluation of LRA perception and actuation in commercial orchardsThe LRA with integrated actuation and perception systems will be tested in an orchard with V-trellised Fuji apple trees, and V-trellised and hedged high-density Bartlett pear trees.