Evaluation — rl_mind.evaluation

Evaluating agents.

During training, we periodically evaluate the current actor on a separate set of environments (using its deterministic behavior, actor.act), and we keep a copy of the best actor seen so far. The Evaluator handles this: call run_if_needed(steps, actor) inside the training loop; it does nothing until every new steps have been performed since the last evaluation.

All the evaluations are kept in history (for plots and statistical tests), and visualize_best() plays an episode of the best actor (in a notebook widget, or as a video file when run as a script).

class rl_mind.evaluation.EvalResult(step: int, rewards: Tensor, is_best: bool)

Bases: object

The result of one evaluation

Parameters:

• step – The number of training steps when the evaluation was run

• rewards – The cumulated reward of each evaluation episode, shape [num_envs]

• is_best – True if this evaluation is the best one so far

property mean: float

The mean cumulated reward

class rl_mind.evaluation.Evaluator(env: VecEnv, every: int, run_dir: Path | None = None, writer: SummaryWriter | None = None)

Bases: Generic[ActionT]

Periodically evaluates an actor, keeping track of the best one

The evaluation environments are deliberately separate from the training ones (their own seed, their own episode count, no same-step reset), so that what is measured does not depend on the data the agent is training on. Evaluation uses Actor.act, the deterministic behavior.

env

The evaluation environments

every

The number of training steps between two evaluations

run_dir

Where videos are recorded (see visualize_best)

writer

The tensorboard writer the mean reward is logged to, if any

history: list[EvalResult]

All the evaluations so far

best_actor: Actor[ActionT] | None

A copy of the best actor so far

best_reward

The mean reward of the best actor

run_if_needed(steps: int, actor: Actor[ActionT]) → EvalResult | None

Evaluate actor if it is time to do so

The call to make from the training loop: it does nothing until every steps have been performed since the last evaluation. When it does evaluate, the result is appended to history, logged to writer, and a deep copy of the actor is kept in best_actor if it improves on best_reward.

Parameters:

• steps – The number of environment steps performed so far (typically a collector’s steps)

• actor – The actor to evaluate

Returns:

The EvalResult, or None when no evaluation was run

run(actor: Actor[ActionT]) → Tensor

Run one evaluation episode in each environment

Steps every environment until all of them are done, ignoring the rewards of an environment once its episode has ended. Does not update history nor best_actor (see run_if_needed).

Parameters:

actor – The actor to evaluate (its act, deterministic, behavior is used)

Returns:

The cumulated reward of each episode, shape [num_envs]

visualize_best(video: bool = False, seed: int = 0, trimmer: Trimmer | None = None, fps: int = 30, width: str = '400px')

Play one episode of the best actor

By default, in a notebook, the episode is shown in the same player as the one of rl_mind.rendering (rollout_frames + animate_frames: play/pause and a slider to scrub through it). The player only works while the kernel runs: with video=True (and always when run as a script), a video is recorded under <run_dir>/videos (the working directory if no run_dir was given) and embedded in the notebook, where it is kept when the notebook is saved (a script prints its path). Either way, the environment has the same wrappers and arguments as the evaluation environments, and only actor.act (deterministic) is used.

Parameters:

• video – Record a video instead of showing the player

• seed – The seed of the episode (player only)

• trimmer – Bounds the frames kept in memory (player only; None keeps every frame, ~10 KB each once encoded)

• fps – The playback speed (player only)

• width – The CSS width of the player (player only)

Returns:

An IPython.display.Video when a video is shown in a notebook, None otherwise

Raises:

AssertionError – If no evaluation has been run yet

rl_mind.evaluation.actor_policy(actor: Actor, action_space: Space)

Turn an actor into a single-environment policy obs -> action

The observation (a numpy array, no batch dimension) is batched, given to actor.act (the deterministic behavior), and the action converted back to what gymnasium expects for action_space. This is what rl_mind.rendering.rollout_frames needs to play a trained actor.

Parameters:

• actor – The actor

• action_space – The action space of the (single) environment

Returns:

A callable obs -> action

rl_mind.evaluation.record_video(actor: Actor, env_name: str, directory: Path, wrappers=(), **kwargs) → Path | None

Record a video of one episode played by actor (deterministic mode)

A fresh single environment is created with render_mode="rgb_array" and played until the episode ends; only actor.act is used, so the video shows the behavior the actor has learned, without exploration noise.

Parameters:

• actor – The actor to film

• env_name – The gymnasium environment name

• directory – Where to write the video file

• wrappers – gymnasium wrappers applied to the environment (must match the ones used during training)

• kwargs – Extra arguments passed to gymnasium.make

Returns:

The path to the recorded video, or None if no file was written