# RL Policies Execution

Alongside the model-based controllers, Kangaroo can be driven by locomotion policies trained with reinforcement learning (see [Reinforcement Learning](../reinforcement_learning/intro.md)). These policies are trained in simulation with [`pal_mjlab`](https://github.com/pal-robotics/pal_mjlab), exported to ONNX, and deployed on the real robot through the [`pal_policy_deployer`](https://github.com/pal-robotics/pal_policy_deployer) framework.

## The `pal_policy_deployer` Framework

[`pal_policy_deployer`](https://github.com/pal-robotics/pal_policy_deployer) is a modular ROS 2 framework for deploying trained machine-learning policies (e.g. ONNX models) on a robot. The two packages relevant to Kangaroo are:

| Package | Responsibility |
|---|---|
| `pal_policy_deployer` | Installation entry point. Contains the launch files and the trained models, and ties the other packages together. |
| `pal_policy_inferer` | Runs the policy inference on the observation vector and publishes the resulting commands. |

Trained models are placed under `pal_policy_deployer/models/`, together with their metadata (observation order, joint structure) and the YAML configuration for the robot-specific settings.

### The `kang_mjlab` Inferer Node

`pal_policy_inferer` provides an implemented node — [`kang_mjlab_node`](https://github.com/pal-robotics/pal_policy_deployer/blob/humble-devel/pal_policy_inferer/src/kang_mjlab_node.cpp) — that is specially fitted for the policies trained on [`pal_mjlab`](https://github.com/pal-robotics/pal_mjlab). Most of the information the node needs to run a policy is **read directly from the metadata embedded in the ONNX model** and then configured automatically. This includes fields such as `joint_names`, `default_joint_pos`, `action_scale`, `observation_names`, `joint_stiffness`, and `joint_damping`, so a policy can be swapped simply by pointing the node at a different model.

At runtime the node builds its observation vector from the robot data published on `/controller_manager/introspection_data` (together with commands such as `/cmd_vel` for locomotion policies), runs inference, and writes the resulting joint commands to the controller command topics.

### ROS 2 Interfaces

The tables below list the topics, actions, and services exposed by the [`kang_mjlab_node`](https://github.com/pal-robotics/pal_policy_deployer/blob/humble-devel/pal_policy_inferer/src/kang_mjlab_node.cpp).

**Subscriptions** (inputs used to build the observation vector):

| Topic | Purpose |
|---|---|
| `/controller_manager/introspection_data/names` | Names of the introspected robot signals (published once / on change). |
| `/controller_manager/introspection_data/values` | Values of the introspected signals; their arrival triggers a new inference step. |
| `/cmd_vel` | Commanded base velocity for locomotion policies. |
| `/odom` | Odometry, subscribed only when the observation set includes a base linear-velocity term. |

**Publishers** (outputs):

| Topic | Type | Purpose |
|---|---|---|
| `/subscriber_controller/desired_state` | `control_msgs/msg/MultiDOFCommand` | Joint commands produced by the policy. Remapped by the launch file to `/rl_impedance_controller/reference`. |
| `pal_policy_deployer_statistics` | `pal_statistics_msgs` (introspection) | Diagnostics such as inference elapsed times and command values. |

**Actions:**

| Action | Type | Purpose |
|---|---|---|
| `~/switch_policy` (i.e. `/pal_policy_deployer/switch_policy`) | `pal_policy_inferer/action/SwitchPolicy` | Switches the active policy by `policy_index`. See [Multi-Policy Architecture](#multi-policy-architecture). |

## Launching a Policy

A policy is launched by passing its model tag to the deployer launch file:

```bash
ros2 launch pal_policy_deployer kang_mjlab_policy_deployer.launch.py model:=v636
```

By default, the **second index** (index `1`) is the jumping (imitation) policy, and additional policies loaded alongside it take the following indices.

### Launch Arguments

The [`kang_mjlab_policy_deployer.launch.py`](https://github.com/pal-robotics/pal_policy_deployer/blob/humble-devel/pal_policy_deployer/launch/kang_mjlab_policy_deployer.launch.py) file exposes one argument per policy slot, so both the primary (locomotion) policy and the secondary (imitation) policy can be selected without editing any file:

| Argument | Default | Description |
|---|---|---|
| `task` | `loco` | Task name of the **primary** policy (index `0`). |
| `model` | `v315` | Model version of the primary policy. |
| `imitation_task` | `imitation` | Task name of the **secondary** policy (index `1`). |
| `imitation_model` | `v719` | Model version of the secondary (imitation) policy. |
| `imitation_frames` | `147` | Frame budget the imitation policy runs for before automatically reverting to the primary policy. |
| `robot_type` | `lower_body` | Hardware configuration; selects the controller config `config/<robot_type>.yaml`. |
| `use_sim_time` | `false` | Use the simulation clock. |

For example, launching with `model:=v636` keeps the imitation slot at its default (`v719`) and only replaces the primary policy with the forward stair-climbing model.

### How the Launch File Defines the Policies

Internally the launch file turns those arguments into a **list of policies** and a set of **per-policy parameters** passed to the `kang_mjlab_node`. Each policy is declared by a name in a `policies` list, and its configuration is nested under that name:

```yaml
policies: ["loco", "imitation"]   # order defines the switch index (0, 1, ...)

loco:
  task:  loco
  model: v315

imitation:
  task:   imitation
  model:  v719
  frames: 147
```

The position of a name in the `policies` list is exactly the `policy_index` used by the [switch action](#multi-policy-architecture): `loco` → `0`, `imitation` → `1`.

Before the node starts, the launch file resolves each policy to a model directory of the form:

```
<pal_policy_deployer_share>/models/<task>/<task>_<model>_mjlab/
```

and verifies that the directory contains at least one `.onnx` file, so a missing or mistyped model fails fast at launch time. It also loads the `rl_impedance_controller` from `config/<robot_type>.yaml` and remaps the node's `desired_state` output onto `/rl_impedance_controller/reference`.

### Mandatory Parameters per Task Type

The parameters that **must** be defined depend on the task type of the policy:

| Task type | Mandatory parameters | Purpose |
|---|---|---|
| `loco` (locomotion) | `task`, `model` | Selects the task family and the ONNX model to run. |
| `imitation` | `task`, `model`, `frames` | In addition to task/model, `frames` sets the frame budget after which the policy auto-reverts to the primary (locomotion) policy so the robot can stabilize. |

In other words, `task` and `model` are required for **every** policy regardless of type, while `frames` is required only for **imitation** tasks — it is what gives the imitation policy its bounded, self-terminating behavior.

## Available Policies

The policies are grouped by the task they solve. Each is identified by its version tag.

### Walking on Flat Terrain

Flat-terrain locomotion is handled by policies **v315** and **v534**. These are the default policies for commanding Kangaroo to walk on level ground.

### Climbing Stairs

Stair climbing is split by direction of travel:

| Policies | Task |
|---|---|
| **v619**, **v630** | Climb stairs **backward** |
| **v636**, **v639** | Climb stairs **forward** |

### Jumping (Imitation)

Jumping is realized through an imitation policy, **v719**.

## Multi-Policy Architecture

The deployer can load more than one policy at a time and switch between them at runtime. This is what enables combining a task policy with a stabilizing policy.

For **imitation tasks** (such as jumping with v719), it is recommended to launch the imitation policy **together with a flat-terrain policy**. The imitation policy performs the task, and once the task is finished the deployer switches back to the flat-terrain policy so it can stabilize the robot after the maneuver.

Policies are switched through an action. Each loaded policy has an **index**, and the goal selects which policy to activate by that index:

```bash
ros2 action send_goal /pal_policy_deployer/switch_policy \
  pal_policy_inferer/action/SwitchPolicy "policy_index: 1"
```

Here `policy_index` defines the index of the policy to switch to among the loaded policies. In the imitation setup above, sending the goal switches into the imitation policy to perform the task; after it completes, control returns to the flat-terrain policy for stabilization.