Beyond 1:1: Multi-participant robotics with LiveKit

In real robot deployments, we are seeing more and more LiveKit customers move beyond a single peer-to-peer connection. Whether it involves serving multiple models, supporting teleoperation, or bringing a human operator into the loop, our SFU architecture supports multi-participant connections with the reliability, low latency, and overall developer experience LiveKit is known for.

With LiveKit, a session is a Room: adding a camera, model, operator, or downstream service simply means adding another participant, not redesigning the transport architecture. Within a Room, LiveKit uses a pub/sub model. Each participant can publish video, audio, and data tracks, while every other participant selectively subscribes to the individual tracks it needs. A policy can consume wrist video and joint state, an operator can watch the overhead camera, and a recorder can subscribe to everything, all from the same session.

This makes it practical to compose robotics systems around a shared realtime session, where each participant publishes its own media, data, and control signals. Below are several architectures we have helped customers enable today.

Human-in-the-loop data recording#

A LiveKit Room supports this workflow as a shared realtime session with four participants: the robot, the policy, the human operator, and a recorder.

The challenge appears when the robot, learned policy, and human operator are all on different networks. For example, the robot and policy may be in the US while the operator is in Mexico. The network round trip between sites may be 60 to 80 ms, while a 30 Hz controller has only 33 ms per tick.

This means network transport has to be kept out of the way of control as much as possible. The robot needs to publish its latest observation directly to both the policy and the operator. The policy needs to return its action to the robot on the shortest available path, while the operator needs an equally direct path to intervene.

A client-server topology forces an undesirable choice: route the operator through the policy, or route the policy through the operator. Either choice adds another network hop for one control path and leaves one participant with a stale or indirect view of the robot's current state.

With a LiveKit Room, the robot, policy, operator, and recorder are independent participants in the same realtime session. The robot's observations fan out simultaneously to the policy, operator, and recorder. Policy actions and human interventions travel independently back to the robot. The recorder captures the synchronized streams without participating in the control path.

VLM-guided skill orchestration#

A robotics system often needs to execute a task that is easy to express in natural language but difficult to implement as a single control policy. Consider an instruction such as: "Sort the blue connectors into the left tray and discard the rest." Completing it may require several specialized policies, including grasping, placement, insertion, inspection, and recovery.

A vision-language model can act as the task-level orchestrator. It interprets the instruction together with a periodic view of the workspace, maintains progress toward the goal, and selects the next skill to invoke. Each selected skill is executed by a separate policy participant that controls the robot at its required rate.

This separation matches the computational characteristics of the components. Low-level policies may consume wrist-camera images, force signals, and joint state at tens of hertz. A VLM generally needs a wider but slower view, such as an overhead camera frame every second and a compact representation of task progress. Its inference may take hundreds of milliseconds or longer, which is appropriate for choosing the next manipulation primitive, but not for generating motor commands.

LiveKit lets these components share a realtime session without requiring every participant to consume every stream. The robot publishes its sensor streams once. Each policy subscribes to the observations it requires for control, while the VLM subscribes only to the lower-rate visual and state inputs relevant to planning.

Each skill policy joins as its own participant. Adding a capability means adding another policy participant, without changing the robot connection or rebuilding the media and data topology. Multiple candidate policies can also subscribe to the same observations for evaluation, while only the active policy publishes commands to the robot.

Policy escalation and recovery#

The VLM selects the next skill, but it is not on the control path. It may update at 1 Hz, while an insertion policy and its reward model operate at 30 Hz or higher.

For insertion, the wrist camera is often ambiguous at the moment of contact. The connector, gripper, and socket occlude the visual evidence needed to distinguish seated, nearly seated, and stuck states. A reward model can classify those states from signals the policy does not necessarily consume: joint current, force-torque readings, end-effector velocity, commanded versus measured pose, and an overhead camera.

The acting policy and reward model are independent Room participants. The policy subscribes to the minimum observations needed to generate actions, such as wrist video and robot proprioception. The reward model additionally subscribes to high-rate telemetry and diagnostic streams. It evaluates the active policy's execution at control rate, producing a compact status such as in_progress, success, stuck, or unsafe.

That status drives policy escalation. A specialized insertion policy runs close to the robot and holds control authority during normal execution. A larger generalist policy is already connected to the same Room and subscribed to the robot's current observations, but cannot publish commands accepted by the robot.

When the reward model reports stuck or unsafe, the robot revokes authority from the specialist and grants it to the generalist. This is a control-plane update, not a new inference deployment. The generalist already has current visual, state, and task context, so it can begin producing recovery actions without loading weights, establishing a transport session, or waiting to rebuild a history window.

Classical and end-to-end model orchestration#

Robotics systems rarely replace classical methods with a single end-to-end model. They combine components with different strengths: deterministic state estimation, motion planning, collision checking, learned perception, and learned control policies.

Each component can join the Room as an independent participant. A classical planner may consume a map, robot pose, and goal, then publish a collision-free trajectory. An end-to-end policy may consume camera images and proprioception to execute a contact-rich local maneuver that is difficult to model analytically. A safety participant can independently check joint limits, collision distance, and workspace constraints before the robot accepts a command.

The robot publishes its observations once, and each participant subscribes to the data it needs. The global planner may need low-rate localization and map updates. A local policy may need high-rate wrist video and joint state. Neither must be placed in the other's request path.

Control authority can also follow the task phase. A classical planner may own navigation through known space, then hand control to an end-to-end policy for grasping or insertion. If the learned policy reports completion, the planner receives the updated robot state and chooses the next route or task step. The handoff is between participants in the same Room, so the transition does not require reconnecting the robot or creating a separate transport path.

This gives teams a way to adopt learned models where they add the most value while retaining classical components for the parts of the system that benefit from predictability, explicit constraints, and debugging tools.

Building on LiveKit Portal#

The architectures above can be built directly on LiveKit Rooms. If you do not want to implement the robotics transport layer yourself, we have LiveKit Portal.

Portal provides the common contract between a robot and its operators. The Robot publishes camera frames and typed state, then receives actions. An Operator can be a policy, teleoperation UI, recorder, or service. It receives synchronized observations and can return actions. Multiple Operators can join the same Room, but the Robot accepts commands only from the active one.

Portal also handles observation synchronization. Video and state arrive independently over different transport paths, even when captured at the same instant. Portal stamps both with the robot's monotonic clock, buffers them at the receiver, and emits an aligned (frames, state, timestamp) bundle. Both sides declare the same schema before connecting, so policy code does not need to manage clock synchronization, track matching, or state serialization.

See the technical overview for the synchronization and multi-operator model.

Working across local and cloud#

Using LiveKit Portal does not require sending every robot stream through the cloud. For deployments that need local control latency, data locality, or an isolated robot network, teams can run the open-source LiveKit SFU alongside the robot.

Here's how it works: The robot and local policies join a Room on the local SFU. A bridge participant joins that local Room and a Room in LiveKit Cloud, selectively relaying the tracks needed by remote participants. A teleoperator, cloud-hosted policy, fleet dashboard, or recorder can then join through LiveKit Cloud without becoming part of the local control path.

One reason to keep the primary Room local is raw video. A 640 × 480 RGB stream at 30 fps is about 221 Mbit/s before framing and transport overhead. Two cameras are already about 440 Mbit/s. That is normal traffic on a switched local network, but not a useful default WAN payload.

The local Room can therefore publish raw RGB or PNG tracks to local policies and recorders. The bridge subscribes to a separate MJPEG or WebRTC video track and republishes that representation to the cloud Room. The local recording remains pixel-exact; remote viewers receive the version sized for the WAN.

The same split applies to control. A local policy subscribes and publishes actions only through the local SFU. The bridge is not on that action path. A remote policy or teleoperator joins the cloud Room when needed, and the bridge relays only the observations and commands required for that participant.

From an application perspective, nothing changes: every component is still a Room participant with explicit publications and subscriptions. The bridge is simply the participant that has connections to both Rooms.

Come get candy from us at Actuate 2026#

We are bringing a candy machine to Actuate.

Inside is an SO-Frame: an open evaluation frame for an SO-101 arm with the LeSlider add-on, a wrist camera, an overhead camera, and the compute needed to run it. Scan the QR code, ask for a candy by voice, and it will plan the task, select the necessary skills, and hand it to you.

Every architecture in this post is running inside that box. A VLM turns the request into a sequence of skills and selects the policy that drives the robot. A reward model detects whether each skill has succeeded or is stuck. The rail uses position control, while a learned policy handles grasping. A human operator can take over during an episode.

Find us at booth #7, August 18–19, at Fort Mason.