UC Merced Robotics

Precision Agriculture · Mission Planning

An End-to-End Software Architecture Integrating Natural Language Mission Planning and Behavior Trees for Precision Agriculture

Marcos Abel Zuzuárregui  ·  Stefano Carpin

UC Merced Robotics Lab

Paper (coming soon) Code (coming soon)
Amiga mobile manipulator navigating row waypoints in a pistachio orchard
The Amiga mobile manipulator in a commercial pistachio orchard, following row-based waypoints while staying aware of each tree's canopy.

Overview

From a sentence like "Sample leaves from tree number 5" to a robot autonomously approaching the canopy and grasping a leaf for hyperspectral sensing.

  • LLM planning meets closed-loop control. Natural-language requests become XML mission specs, which the robot parses into fault-tolerant Behavior Trees with built-in retry logic.
  • No tree database needed. An orchard-management layer interpolates tree positions and row waypoints from a single user-drawn GPS polygon. That keeps GPS lists out of the LLM context.
  • Perception decides the last meter. 3D LiDAR picks the closest reachable canopy point, so the robot adapts to each tree's shape instead of trusting a fixed GPS offset.
  • Field-validated. We tested on ROS2 (Nav2, MoveIt2) in commercial pistachio and citrus orchards and in a lab setting.

Architecture

The mission is planned once, off-board. After that the robot needs no network connection.

1 · Input
Natural language
Operator describes the mission in plain words.
→
2 · Plan
LLM → XML
Mission spec that conforms to a robot capability schema.
→
3 · Transfer
TCP to robot
After transfer, the robot needs no connection.
→
4 · Execute
Behavior Tree
Validated on board and run with retries on ROS2.
Mission planning architecture: user NL input, LLM, retry loop, mission tree
Mission planning architecture. Users give NL input, the LLM produces a mission, and the robot executes the resulting BT in the field. The architecture is adapted from our GPT Mission Planner.

On-robot software stack

The ROS2 Humble stack keeps mission orchestration separate from hardware execution. Compute-heavy perception runs in its own nodes so it can't starve navigation or manipulation.

  • Nav2 handles global and local planning over the Amiga's gRPC/CAN drivers.
  • MoveIt 2 handles motion planning and IK for the Kinova Gen3.
  • A ROS1–ROS2 bridge streams point clouds from the legacy 3D LiDAR drivers.
  • Our BT engine, orchard DB, and collision-aware approach nodes are the core contributions.
System diagram of ROS2 nodes and custom components
System diagram. Off-the-shelf ROS2 frameworks alongside the components we developed.

Platform

A field-deployable mobile manipulator for close-range leaf sensing. All autonomy runs on board.

Labeled mobile manipulator platform
  • 1IMU (VectorNav VN-100)
  • 2RTK GPS (Emlid Reach RS2)
  • 3RGB-D camera (OAK-D W PoE)
  • 46-DOF arm (Kinova Gen3)
  • 5Two-finger gripper (Robotiq)
  • 6Nano spectrometer (350–1010 nm)
  • 7Fiber-optic cable
  • 83D LiDAR (Ouster OS0-128)
  • 9VIS-NIR light source
  • 10Mobile platform (Bonsai Amiga)

Compute: NVIDIA Jetson AGX Orin (64 GB, ROS2 Humble). A separate NUC hosts the LiDAR sensor tower.

Methods

The mission gives an abstract plan, the orchard layer grounds it in the field, and perception handles the final approach.

Mission text decomposed into a behavior tree with retry nodes
From request to Behavior Tree. A multi-tree pistachio mission. The BT includes implicit tasks the user never asked for, such as entering and exiting rows and retrying up to 3 times, which the mission needs to succeed in practice.

Orchard management abstraction

Earlier versions passed the LLM a full list of GPS points. That doesn't scale and can overflow the context window. Now the user provides one polygon and the grid size (rows × columns). The system then interpolates every tree position and its adjacent row waypoints.

The LLM can then plan with relative references like "the third tree down the row", and the robot resolves them into concrete waypoints on board.

Diagram of LiDAR-based closest canopy point selection
Inside-canopy navigation. The robot sweeps an azimuth window around the expected tree direction and a height band, then targets the closest canopy point at a safety distance d.
Pcanopy = { p ∈ P | atan2(py, px) ∈ [φtree ± Δφ] ∧ hmin ≤ pz ≤ hmax }
pgoal = argminp ∈ Pcanopy ‖p − probot‖ − d·np

Experiments used hmin = 1.0 m, hmax = 1.5 m, and d = 1 m. The goal comes from the canopy as the LiDAR sees it, so tree and robot GPS estimates don't have to be exact.

Interpolated tree GPS points and row waypoints over satellite imagery
Interpolated tree locations (red) and adjacent row waypoints (green) over satellite imagery of a 144-tree (8 × 18) pistachio block, all generated from a single polygon.
Robotiq gripper with custom tip grasping a citrus leaf
Leaf grasp. A custom end-effector tip on the Robotiq gripper holds a citrus leaf still for a repeatable spectral reading. Leaf detection uses an existing YOLOv8-based RGB-D pipeline, integrated as-is behind a pose interface.
Amiga robot next to a potted citrus tree in the lab setup
Lab setup. A potted tangerine tree used to validate approach and sampling from four robot orientations before field trials.

Results

We ran lab trials, then field trials in commercial pistachio and citrus orchards.

30/30
Lab trials that reached a sampling attempt, across 4 orientations and 2 lighting conditions
5/5
Multi-tree pistachio missions with successful navigation and approach
7/8
Single-tree citrus trials that succeeded, with 4/5 BT recoveries
5/5
Human-in-the-loop "follow me" trees sampled with no recoveries
Robot approaching a citrus tree
Approaching a citrus tree.
Robot approaching a leafless pistachio tree
Approaching a leafless (winter) pistachio canopy.
GPS paths of pistachio experiments over aerial imagery
GPS paths of pistachio missions 2 (blue), 4 (red), and 5 (orange). The aerial image was taken in summer.

Table 1 · Lab, single tree

Approach from multiple robot orientations

OrientationTrialsRecoveriesSuccessLighting
0°52/25/5Sunny
+90°52/25/5
−90°51/15/5
180°53/35/5
+90°50/05/5Cloudy
−90°50/05/5
Overall308/830/30

Table 2 · Citrus, single tree

Prompt: "Sample leaves from tree number X"

Tree #TrialsRecoveriesSuccessLighting
222/22/2Dusk
310/10/1
520/02/2
710/01/1
1122/22/2
Overall84/57/8

Table 3 · Multi-tree, multi-row missions

Prompt: "Sample leaves from tree numbers X, Y, and/or Z"

Tree #TrialsRecoveriesSuccessLighting
11, 1310/01/1Sunny
11, 1410/01/1
11, 5310/01/1
11, 13, 1410/01/1
11, 13, 2910/01/1
Pistachio*50/05/5
4, 952/43/5Dawn
4, 950/50/5Sunny
Citrus102/93/10

* Pistachio trees had no leaves during winter testing, so success means successful navigation and approach only. A recovery counts a BT retry sequence that eventually succeeded. Success means the hyperspectral sensor was triggered through motion planning and the mission completed.

Hyperspectral measurements

When a leaf was grasped, the gripper-mounted spectrometer recorded VIS–NIR signatures usable for water stress, nutrient, disease, and pest assessment.

The sensing payload is decoupled from navigation and approach, so the platform is sensor-agnostic. The payload can be swapped for thermal or chemical sensing without touching the BT or mission planner.

Leaf transmittance versus wavelength for three leaves
Hyperspectral signatures from 3 trials with successful leaf contact.

Limitations & lessons learned

What broke in the field, and what we'd change.

Lighting sensitivity

The third-party leaf segmentation struggled in direct sun. Six of the seven multi-tree citrus failures were lighting-related. The module sits behind a pose interface, so it can be swapped out.

Canopy shape

Protruding branches ("suckers") can dominate the closest-point estimate. The robot then stops safely but out of arm reach. Sparse canopies can also invite Nav2 to cut between trees.

Grasp variability

Wind, small curved citrus leaves, and dense clusters lead to partial or multi-leaf grasps. These degrade how consistent the spectra are.

Citrus tree with sucker branches highlighted
(A) Sucker shoots from rapid early-spring growth. (B) Area of interest causing approach problems. (C) Side view of the highlighted areas.

Takeaways

  • Nav2 near canopies is highly sensitive to tuning of costmap inflation, footprint, and controller parameters.
  • GPS degrades under canopy, right when precision matters most. Fusing RTK GPS with a calibrated IMU and wheel odometry through an EKF is the more robust path.
  • Pure velocity control was unsafe with anomalous LiDAR returns. A collision-aware velocity controller plus BT retries worked best.