Teleoperation capture best practices for imitation learning

Teleoperation best practices for imitation learning are the decisions that make demonstrations trainable: action space, camera placement, time sync, episode boundaries, operator diversity, metadata, and QA gates. Set them before capture scales. Operant applies these practices on custom programs so the episodes match your robot and the pipeline that will train on them.

Imitation learning copies what the operator did. If the recorded actions, cameras, or episode cuts do not match the policy you intend to train, more hours of capture will not fix it. The teleoperation capture service treats these choices as scope, not as cleanup after the files land.

Terms used here are defined in the robotics data collection glossary, including teleoperation, embodiment, and QA gates.

Lock the action space first

The action space is what the operator commands and how those commands map onto the robot. Joint targets, end-effector poses, gripper width, base velocity, and whole-body commands are different learning problems. Pick the space your policy will use, then record that space directly.

Write the mapping down before the first session:

• Which joints, end effectors, or base axes the operator controls.
• Control rate and the units stored in the log.
• What a no-motion command looks like.
• How gripper, hand, or tool state is encoded.
• Which signals are observations and which are actions.

A pilot that logs a different mapping than production training will look successful in review and fail on hardware. If you are still choosing the device the operator holds, settle that in how to choose a teleoperation interface, then return here for the capture process.

Place cameras and synchronize sensors

Cameras should see the contact, the object, and enough scene context for the policy inputs you will actually train on. Wrist views, overhead views, and side views answer different questions. Occlusion at the gripper is a capture design problem, not a labeling problem.

For each rig, specify:

• Camera roles, resolution, and frame rate.
• Extrinsic and intrinsic calibration, and how often they are rechecked.
• Proprioception, gripper state, force/torque, and any other required streams.
• The clock source and the sync tolerance observations must meet.

Time alignment belongs in the capture plan. Pair the rig with multi-sensor synchronization so frames, commands, and contact signals refer to the same moment. A calibration file that is missing from the delivery is a failed episode, even when the video looks fine.

Define episode boundaries before capture

An episode is the unit your training loop will sample. Inconsistent starts, ends, and resets are one of the most expensive teleoperation mistakes, because they are hard to see in a single clip and obvious once you try to batch the data.

Agree on:

• What event starts the episode.
• What counts as success, failure, and a discarded attempt.
• When a recovery stays inside the episode and when it becomes a new one.
• How the operator resets the scene, and what gets logged during reset.
• Whether idle time at the start or end is trimmed.

These rules should match the imitation learning data collection plan. If evaluation needs slips, regrasps, or aborted attempts, those outcomes are part of the episode definition, not clips to delete later.

Track operator diversity

A single fluent operator can make a pilot look better than the behavior you can collect at scale. Style, speed, grasp choice, and recovery habits all transfer into the policy. Diversity is a scoping target: how many operators, which tasks each person covers, and how operator identity is stored.

Track operator ID on every accepted episode. Review early episodes for one-person habits, such as a single approach angle or a reset that only one person can perform. Add operators before you add hours.

Log metadata you can filter on

Metadata is how your team slices the data after delivery. Without it, every review becomes a video search. At minimum, log task, operator, scene, outcome, and the rig or calibration version. Add the fields your evaluation already uses: object class, clutter, lighting, failure type, or recovery path.

Keep the schema stable from the pilot onward. A field that appears only in later batches cannot be used to balance the full set.

Set QA gates that stop a bad pilot

QA should fail an episode while the operator is still on site. Useful gates:

• Required sensor streams are present.
• Timestamps fall inside the agreed tolerance.
• Calibration files match the rig used for that episode.
• Episode boundaries and outcome labels are filled in.
• Operator, task, and scene metadata are present.
• A sample review confirms the behavior is visible and the action log matches the motion.

If early episodes show occlusion, action-space mismatch, or sync drift, stop and fix the pipeline. Scaling a broken setup is how teleoperation programs become expensive. The same gates should appear in the statement of work for the robotics data collection program this capture belongs to.

When another method fits better

Teleoperation is the method when the policy needs actions in the robot's control space. A human demonstration fits when you need a person performing the task and the robot cannot yet be driven through it. Those movements still have to be mapped onto the embodiment. Egocentric capture fits when the training input is a first-person view of the behavior. The comparison is summarized in teleoperation versus a human demonstration, and the capture options are scoped under robot demonstration data.

Scripted or autonomous logs fit when the behavior is already reliable and you need volume of the robot's own execution, including failures, rather than expert corrections.

Checklist before you scale

1. The recorded action space matches the policy you will train.
2. Camera roles, calibration, and sync tolerance are written down.
3. Episode start, end, success, failure, and reset rules are agreed.
4. Operator identity and diversity targets are in the metadata plan.
5. Filter fields match the evaluation slices you already use.
6. QA gates can reject an episode during the session.
7. The pilot delivery uses the same formats as production.

To scope a teleoperation program around your robot, book a discovery call.