rlcli train: one-shot SL, RL, Harbor agent, and self-distillation training

rlcli provides four training subcommands under train: sl for supervised fine-tuning, rl for reinforcement learning on math benchmarks, harbor for sandboxed agent RL, and opsd for on-policy self-distillation from a prompts file. Each subcommand sends requests to the Tinker API at --base-url and automatically sets TINKER_API_KEY=tml-dummy.

sl
rl
harbor
opsd

train sl

Supervised fine-tune a model on a messages JSONL dataset.

rlcli train sl --dataset <PATH> --model <MODEL> [OPTIONS]

Flags

string

required

Path to a JSONL file, or - to read from stdin.

string

required

HuggingFace model name or path.

string

default:"http://localhost:8000"

Tinker API server URL.

string

Renderer to use. Defaults to the recommended renderer for the model.

int

default:"8"

Training batch size.

float

default:"1e-4"

Learning rate.

int

default:"2048"

Maximum sequence length for training examples.

int

default:"1"

Number of training epochs.

int

default:"32"

LoRA rank.

int

default:"20"

Save a checkpoint every N steps.

float

default:"0"

Fraction of data to hold out for evaluation.

string

default:"~/.rlcli/runs/sl-<timestamp>"

Directory for training logs.

flag

Print the training plan without executing.

Example

rlcli train sl --dataset clean.jsonl --model Qwen/Qwen3-4B-Instruct-2507 \
  --batch-size 16 --lr 2e-4 --epochs 2

train rl

Run reinforcement learning on GSM8K.

rlcli train rl --model <MODEL> [OPTIONS]

Flags

string

required

HuggingFace model name or path.

string

default:"http://localhost:8000"

Tinker API server URL.

choice

default:"importance_sampling"

RL loss to use. All losses are available; the server must support the chosen one.

JSON string

Raw JSON overrides for the loss function configuration.

choice

Backend hint for the loss guard. Choices: jax, fsdp, megatron. Auto-detected for managed servers.

choice

default:"gsm8k"

Dataset to train on. The only supported choice is gsm8k.

string

Renderer to use. Defaults to the recommended renderer for the model.

int

default:"64"

Training batch size.

int

default:"8"

Group size for grouped RL.

float

default:"4e-5"

Learning rate.

int

default:"256"

Maximum tokens to generate per response.

int

default:"32"

LoRA rank.

int

default:"20"

Save a checkpoint every N steps.

int

default:"0"

Run evaluation every N steps. 0 disables evaluation.

int

Maximum number of training steps.

string

Directory for training logs.

flag

Print the training plan without executing.

Example

rlcli train rl --model Qwen/Qwen3-4B-Instruct-2507 --loss ppo \
  --batch-size 32 --group-size 4 --max-steps 100

train harbor

Run sandboxed Harbor agent RL on terminal-bench or your own task dataset.

rlcli train harbor --model <MODEL> [OPTIONS]

Flags

string

required

HuggingFace model name or path.

string

default:"http://localhost:8000"

Tinker API server URL.

choice

default:"importance_sampling"

RL loss to use.

JSON string

Raw JSON overrides for the loss function configuration.

choice

Backend hint for the loss guard. Choices: jax, fsdp, megatron. Auto-detected for managed servers.

string

default:"[email protected]"

Task dataset. Defaults to [email protected] and auto-downloads via uvx harbor datasets download if missing.

choice

default:"docker"

Sandbox backend. Choices: docker, modal.

string

Filter tasks by name or pattern.

int

Maximum number of tasks to use.

string

Renderer to use. Defaults to the recommended renderer for the model.

int

default:"4"

Group size for grouped RL.

int

default:"4"

Number of groups per batch.

float

default:"1e-5"

Learning rate.

int

default:"4096"

Maximum tokens to generate per response.

int

default:"10"

Maximum turns per agent episode.

int

default:"32"

LoRA rank.

int

default:"5"

Save a checkpoint every N steps.

int

default:"0"

Run evaluation every N steps. 0 disables evaluation.

int

Maximum number of training steps.

int

default:"3600"

Timeout for the sandbox environment in seconds.

int

default:"120"

Timeout for individual commands inside the sandbox in seconds.

string

Directory for training logs.

flag

Print the training plan without executing.

Example

rlcli train harbor --model Qwen/Qwen3-4B-Instruct-2507 \
  --dataset [email protected] --sandbox docker --max-steps 50

train opsd

On-policy self-distillation on a prompts JSONL. Each step the student samples --group-size completions per prompt, a teacher scores the student’s own tokens, and the negative reverse KL (student || teacher) becomes the per-token advantage for an importance_sampling update. There is no reward: the teacher is the only signal. Drives the pinned cookbook’s distillation.train_on_policy recipe.

rlcli train opsd --dataset <PATH> --model <MODEL> [OPTIONS]

Flags

string

required

Path to a JSONL file, or - to read from stdin. Each line is {"prompt": "..."} or {"messages": [...]}; for messages, the last user turn is the prompt and earlier system/user/assistant text turns are context (trailing assistant turns are dropped).

string

required

HuggingFace model name of the student.

string

default:"http://localhost:8000"

Tinker API server URL.

string

Teacher: a base model name, or a tinker://... checkpoint of --model on the same server. Defaults to --model itself (self-distillation).

string

Privileged text the teacher sees prepended to every prompt (hint, blank line, prompt). The student never sees it, so with the default teacher this is “same weights + hint” self-distillation.

float

default:"1.0"

Weight of the negative reverse KL in the per-token advantages.

float

default:"0.0"

Discount for summing future per-token KL into each advantage. 0 keeps it per-token.

int

Stop after N steps. Defaults to one pass over the dataset.

string

Renderer to use. Defaults to the recommended renderer for the model.

int

default:"32"

Prompts per step.

int

default:"4"

Samples per prompt.

float

default:"1e-4"

Learning rate. --learning-rate is an alias.

int

default:"1024"

Maximum tokens to generate per response.

int

default:"32"

LoRA rank.

int

default:"20"

Save a checkpoint every N steps.

string

default:"~/.rlcli/runs/opsd-<timestamp>"

Directory for training logs.

flag

Load the prompts and print the resolved config without contacting the server.

Example

rlcli train opsd --model Qwen/Qwen3-4B-Instruct-2507 --dataset prompts.jsonl \
  --teacher-hint "Think step by step and double-check your arithmetic." \
  --batch-size 16 --group-size 4 --steps 50

The teacher scores the student’s tokens as rendered with the student’s chat template; a --teacher with a different template is scored under the student’s rendering (the cookbook’s assumption too). With --teacher-hint, rlcli builds the hinted prompt with the same renderer and wraps the teacher client so its logprob calls see hint + prompt while the student’s sequence is unchanged.

Common notes

TINKER_API_KEY is set to tml-dummy automatically if unset.
TINKER_BASE_URL is forwarded automatically.
The loss guard runs before training begins and fails fast with a clear message if the selected loss is unsupported by the backend.

gspo has a known upstream packing-order bug on multi-GPU and Megatron setups (NovaSky-AI/SkyRL#2043). Single-GPU FSDP is verified clean. rlcli prints a warning to stderr when --loss gspo is used.

Overview

Commands

rlcli train: one-shot SL, RL, Harbor agent, and self-distillation training

train sl

Flags

Example

train rl

Flags

Example

train harbor

Flags

Example

train opsd

Flags

Example

Common notes

​train sl

​Flags

​Example

​train rl

​Flags

​Example

​train harbor

​Flags

​Example

​train opsd

​Flags

​Example

​Common notes

train sl

Flags

Example

train rl

Flags

Example

train harbor

Flags

Example

train opsd

Flags

Example

Common notes