FARM: Find Anything using Relational Spatial Memory

// teaser

FARM grounding relational queries in indoor houses and outdoor construction sites

Figure 1. FARM grounds relational language queries in real time, from indoor multi-floor houses to outdoor construction sites. It can find one lamp among 42 near-identical distractors using its spatial relations to landmarks.

// overview

Robots in large, object-rich spaces need a persistent spatial memory and the ability to retrieve a specific object instance from natural language that refers to it through relations to landmarks, such as "the tall lamp below the dartboard and left of the poster." Prior systems require offline processing, fixed vocabularies, or struggle with compositional spatial reasoning.

FARM builds an online, object-centric scene graph in real time (5–10 Hz), storing geometry, vision-language descriptors, and viewpoint evidence per object. At query time, an LLM parses the instruction into a symbolic specification with spatial predicates; soft evaluators score candidate bindings against memory; and a VLM reranks the top candidates by inspecting projected views, giving structured reasoning over noisy, metric data.

// key contributions

Real-time, scale-invariant mapping Online scene-graph construction from indoor rooms to 15,000 m² outdoor sites under one fixed hyperparameter setting, with no offline post-processing.
Predicate + VLM retrieval Structured spatial-predicate evaluation combined with VLM verification, for +164% Recall@5 and +224% Recall@10 over prior methods.
FARM-Scenes benchmark Seven large indoor/outdoor scenes (1,800–15,000 m²) with 44k annotated relational language queries.
Closed-loop robot deployment Demonstrated on a quadrupedal robot using onboard sensors and compute.

// method

FARM online memory construction pipeline

Figure 3. Online memory construction: RGB-D detection, lifting to 3D Gaussians, cross-view association, and asynchronous VLM captioning & embedding.

Figure 4. Relational retrieval: parse the query into a query graph, soft-score predicates against memory, and rerank candidates with a VLM on projected views.

// results & impact

+164%

Recall@5 over prior work

+224%

Recall@10 over prior work

5–10 Hz

online mapping rate

15,000 m²

largest mapped scene

FARM real-world relational retrieval on a Boston Dynamics Spot robot

Figure 5. Real-world relational retrieval on a Boston Dynamics Spot in indoor and warehouse scenes, using spatial predicates and anchor objects.

FARM works across ScanNet, HM3D, and outdoor construction, warehouse, and museum environments, pointing toward robots that can take an everyday relational instruction and act on it in the messy, open world.

// technologies & tools

LLMs VLMs Scene Graphs 3D Gaussians RGB-D Perception Real-time Mapping PyTorch ROS 2 Quadruped (Spot)

// team

A collaboration between Stanford University (Autonomous Systems Lab) and UC Berkeley.

Siming He

★ First author

UC Berkeley

Leo Huang · Adam Lilja · Jonas Frey

Co-authors

UC Berkeley & Stanford

Fabio Hübel

Co-author

Stanford University · ASL

Marco Pavone

PI

Stanford University · ASL

S. Shankar Sastry · Jitendra Malik · Claire Tomlin

PIs

UC Berkeley

// reference

He, Huang, Lilja, Hübel, Frey, Pavone, Sastry, Malik, Tomlin. "FARM: Find Anything using Relational Spatial Memory." Under review. arXiv:2606.15476.