Extra environments — rl_mind.envs, rl_mind.mazemdp_env

Both modules register their environment in gymnasium when imported, which is why they are never imported by rl_mind itself:

import rl_mind.envs          # CartPoleContinuous-v1
import rl_mind.mazemdp_env   # MazeMDP-v0

Extra environments.

CartPoleContinuous-v1 is the classic cart-pole environment with a continuous action in \([-1, 1]\) (the force applied to the cart), used by the DDPG/TD3/SAC notebooks. Adapted from bbrl_gymnasium (MIT license, Sorbonne Université).

class rl_mind.envs.ContinuousCartPoleEnv(**kwargs)

Bases: CartPoleEnv

Continuous-action version of the CartPole-v1 environment

Same observation space, reward (+1 per step) and termination condition as CartPole-v1, but the action is the signed fraction of the maximum force applied to the cart, a Box(-1, 1, shape=(1,)). Registered as CartPoleContinuous-v1 (500 steps at most) when rl_mind.envs is imported.

step(action)

Apply a force to the cart and advance the simulation by one step

Parameters:

action – The force, as a fraction of force_mag in \([-1, 1]\) (clipped); a scalar or a one-element array

Returns:

The gymnasium 5-tuple (obs, reward, terminated, truncated, info), with a reward of 1.0 at every step

The Simple Maze MDP environment (MazeMDP-v0).

A small tabular Markov Decision Process on a grid, used by the dynamic-programming and tabular-RL notebooks. Wraps the mazemdp package and exposes the transition model (P, r, terminal_states, gamma) for planning, plus gymnasium-style reset / step for learning.

Importing this module registers MazeMDP-v0 in gymnasium. Adapted from bbrl_gymnasium (MIT license, Sorbonne Université).

class rl_mind.mazemdp_env.MazeMDPEnv(render_mode=None, **kwargs)

Bases: Env

A tabular grid-world MDP, with its transition model exposed

Observations are state indices (a Discrete space) and there are four actions (north, south, east, west). Unlike a usual gymnasium environment, the whole model is available as attributes (P, r, terminal_states, gamma), which is what the dynamic-programming notebooks need; the draw_* methods render a value function and/or a policy on the grid.

Created through gymnasium, either random or from an explicit wall list:

import rl_mind.mazemdp_env  # registers MazeMDP-v0
env = gym.make("MazeMDP-v0", kwargs={"width": 5, "height": 5,
                                     "ratio": 0.2, "seed": 0})

Parameters:

• render_mode – "rgb_array", "human", or None

• kwargs – A single kwargs= dictionary describing the maze: width, height, hit (reward for hitting a wall), then either ratio (fraction of walls, with an optional seed for a reproducible random maze) or walls (explicit wall indices, with optional terminal_states), plus optional start_states. With no argument at all, a random 10x10 maze with 20% walls is built.

nb_states

The number of states (walls excluded)

coord_x

The grid column of each state

coord_y

The grid row of each state

observation_space

The observation space: a state index in [0, nb_states)

action_space

The action space: 0 = north, 1 = south, 2 = east, 3 = west

terminal_states

The indices of the terminal states

P

The transition model \(P(s' \mid s, a)\), shape [nb_states, 4, nb_states]

gamma

The discount factor \(\gamma\) of the MDP

r

The reward model \(r(s, a)\), shape [nb_states, 4]

state

The current state index. Reading it is cheating: a learning agent only knows the states reset / step return.

set_title(title)

Set the title drawn above the maze

Parameters:

title – The new title

set_render_func(render_func: Callable, *args, **kwargs)

Set the function render calls

Parameters:

• render_func – The drawing function (the mazemdp renderer by default); the current render_mode is passed as mode=

• args – Positional arguments bound to render_func

• kwargs – Keyword arguments bound to render_func

seed(seed=None)

Seed the environment’s random generator

Parameters:

seed – The seed, or None to draw one

Returns:

A one-element list holding the seed actually used

step(action)

Perform one action in the maze

Parameters:

action – The action index (0 = north, 1 = south, 2 = east, 3 = west)

Returns:

The gymnasium 5-tuple (state, reward, terminated, truncated, info), where state is a state index

reset(*, seed: int | None = None, options: dict[str, Any] | None = None) → tuple[ObsType, dict[str, Any]]

Start a new episode

Parameters:

• seed – Unused (the maze itself is fixed at construction time; seed it there for a reproducible random maze)

• options – Passed to mazemdp’s reset, e.g. {"uniform": True} to start from a state drawn uniformly

Returns:

The initial state index and an (empty) info dictionary

sample_transition(**kwargs)

Draw a transition from the model, without stepping the environment

Parameters:

kwargs – Passed to mazemdp’s sample_transition

Returns:

What mazemdp returns, a sampled (state, action, reward, next state) transition

draw_v_pi_a(v, policy, agent_pos, title='MDP studies', recorder=None)

Draw the value function, the policy and the agent’s position

Parameters:

• v – The value of each state, shape [nb_states]

• policy – The action chosen in each state, shape [nb_states]

• agent_pos – The state the agent stands on

• title – The title drawn above the maze

• recorder – A video recorder to capture the frame into, instead of displaying it

Returns:

What the renderer returns (an [H, W, 3] array in rgb_array mode), or None when recording

draw_v_pi(v, policy, title='MDP studies', recorder=None)

Draw the value function (colors) and the policy (arrows)

Parameters:

• v – The value of each state, shape [nb_states]

• policy – The action chosen in each state, shape [nb_states]

• title – The title drawn above the maze

• recorder – A video recorder to capture the frame into, instead of displaying it

Returns:

What the renderer returns, or None when recording

draw_v(v, title='MDP studies', recorder=None)

Draw a value function (or the Q-values of each action)

Parameters:

• v – The value of each state (shape [nb_states]), or of each state-action pair (shape [nb_states, 4], drawn as four triangles per cell)

• title – The title drawn above the maze

• recorder – A video recorder to capture the frame into, instead of displaying it

Returns:

What the renderer returns, or None when recording

draw_pi(policy, title='MDP studies', recorder=None)

Draw a policy (one arrow per state)

Parameters:

• policy – The action chosen in each state, shape [nb_states]

• title – The title drawn above the maze

• recorder – A video recorder to capture the frame into, instead of displaying it

Returns:

What the renderer returns, or None when recording

init_draw(title, recorder=None)

Open (or reset) the figure the maze is drawn in

Parameters:

• title – The title drawn above the maze

• recorder – A video recorder to capture the frame into

Returns:

What the renderer returns, or None when recording

render()

Render the maze with the current render function

Returns:

An [H, W, 3] array in rgb_array mode, None in human mode

render_mdp(func, *args, **kwargs)

Render with a custom drawing function (legacy; prefer set_render_func followed by render)

Note that this replaces the set_render_func method with a partial of func, so later calls to it fail; nothing in the practicals uses this method.

Parameters:

• func – The drawing function

• args – Positional arguments bound to func

• kwargs – Keyword arguments bound to func

set_no_agent()

Stop drawing the agent (for pure dynamic-programming figures)

set_timeout(timeout)

Set the maximum number of steps of an episode

Parameters:

timeout – The step limit after which the episode is truncated