Collectors — `rl_mind.collectors`¶

Collecting interaction data.

The collectors run an actor in a vectorized environment and package the result into the containers of rl_mind.data. They deal with the auto-reset bookkeeping so that the learning code never has to.

TransitionCollector.collect(n_steps) performs n_steps steps in every environment and returns the corresponding batch of Transitions (for replay-buffer based algorithms). The environments keep their state between calls.
EpisodeCollector.collect(n_episodes) resets the environments and returns at least n_episodes complete episodes (Episode), for episodic algorithms, which need on-policy full episodes.
RolloutCollector.collect(n_steps) returns a [T, B] Rollout (for on-policy algorithms such as A2C and PPO). It requires an environment with same_step_reset=True, so that every row of the rollout is a valid transition. The environments keep their state between calls.

All count the total number of recorded environment steps in steps. Actions are computed with torch.no_grad(): if the learning algorithm needs gradients (e.g. of action log-probabilities), it must recompute them from the stored observations and actions.

	`TransitionCollector`	`EpisodeCollector`	`RolloutCollector`
Returns	`Transitions` — flat `[N, …]`	`list[Episode]`	`Rollout` — `[T, B, …]`
`collect(...)` argument	`n_steps`	`n_episodes`	`n_steps`
For	off-policy (DQN, DDPG, TD3, SAC)	episodic (REINFORCE)	on-policy (A2C, PPO)
`VecEnv` reset mode	next-step (default)	next-step (default)	requires `same_step_reset=True`
Environment state across calls	kept	reset at each call	kept

class rl_mind.collectors.TransitionCollector(env: VecEnv, actor: Actor[ActionT])[source]¶

Bases: Generic[ActionT]

Collect batches of transitions by running an actor in a VecEnv

The collector for off-policy algorithms (DQN, DDPG, TD3, SAC, TQC): the transitions it returns are meant to be pushed into a rl_mind.data.ReplayBuffer, so their order does not matter and the (invalid) auto-reset steps are simply dropped.

env¶: The environments being stepped

actor¶: The actor being run

steps¶: Total number of recorded transitions

collect(n_steps: int) → Transitions[ActionT][source]¶

Perform n_steps steps in every environment

The environments are reset on the first call only: successive calls continue the same episodes, so a training loop can interleave collect(1) with gradient steps.

Parameters:: n_steps – The number of steps performed in each of the env.num_envs environments
Returns:: The recorded transitions, a flat batch of at most n_steps * num_envs of them (the steps where an environment was auto-resetting are skipped, and steps is increased by the number actually recorded)

class rl_mind.collectors.EpisodeCollector(env: VecEnv, actor: Actor[ActionT])[source]¶

Bases: Generic[ActionT]

Collect full episodes by running an actor in a VecEnv

The collector for episodic algorithms (REINFORCE), which need complete on-policy episodes: the environments are therefore reset at the beginning of every collect.

env¶: The environments being stepped

actor¶: The actor being run

steps¶: Total number of recorded steps

collect(n_episodes: int) → list[Episode[ActionT]][source]¶

Collect at least n_episodes complete episodes

The environments are reset at the beginning of the call, so that all the returned episodes are collected with the current actor.

Parameters:: n_episodes – The minimum number of episodes to return
Returns:: The completed episodes, of varying lengths — more than n_episodes when several environments finish during the same step (the episodes still running are discarded)

class rl_mind.collectors.RolloutCollector(env: VecEnv, actor: Actor[ActionT])[source]¶

Bases: Generic[ActionT]

Collect fixed-length [T, B] rollouts by running an actor in a VecEnv

The collector for on-policy algorithms (A2C, PPO, GAIL): the time axis is preserved, since the advantages are computed by a backward recursion over it. This is why the environment must use same_step_reset=True — every row of the [T, B] block is then a valid transition.

env¶: The environments being stepped

actor¶: The actor being run

steps¶: Total number of recorded steps

collect(n_steps: int) → Rollout[ActionT][source]¶

Perform n_steps steps in every environment

The environments are reset on the first call only: successive calls continue where the previous one stopped.

Parameters:: n_steps – The number of steps performed in each environment, i.e. the length T of the rollout
Returns:: A [n_steps, num_envs] rollout (every row is a valid transition, episode boundaries being marked by terminated / truncated)

Collectors — rl_mind.collectors¶

Collectors — `rl_mind.collectors`¶