worker.collector.base_serial_evaluator

base_serial_evaluator

Please Reference ding/worker/collector/base_serial_evaluator.py for usage

BaseSerialEvaluator

class ding.worker.collector.base_serial_evaluator.BaseSerialEvaluator(cfg: dict, env: ding.envs.env_manager.base_env_manager.BaseEnvManager = None, policy: collections.namedtuple = None, tb_logger: SummaryWriter = None)

Overview:

Base class for serial evaluator.

Interfaces:

__init__, reset, reset_policy, reset_env, close, should_eval, eval

Property:

env, policy

__del__()

Overview:

Execute the close command and close the evaluator. __del__ is automatically called to destroy the evaluator instance when the evaluator finishes its work

__init__(cfg: dict, env: ding.envs.env_manager.base_env_manager.BaseEnvManager = None, policy: collections.namedtuple = None, tb_logger: SummaryWriter = None) → None

Overview:

Init method. Load config and use self._cfg setting to build common serial evaluator components, e.g. logger helper, timer. Policy is not initialized here, but set afterwards through policy setter.

Arguments:

• cfg (EasyDict)

close() → None

Overview:

Close the evaluator. If end_flag is False, close the environment, flush the tb_logger and close the tb_logger.

classmethod default_config() → easydict.EasyDict

Overview:

Get evaluator’s default config. We merge evaluator’s default config with other default configs and user’s config to get the final config.

Return:

cfg: (EasyDict): evaluator’s default config

eval(save_ckpt_fn: Optional[Callable] = None, train_iter: int = - 1, envstep: int = - 1, n_episode: Optional[int] = None) → Tuple[bool, float]

Overview:

Evaluate policy and store the best policy based on whether it reaches the highest historical reward.

Arguments:

• save_ckpt_fn (Callable): Saving ckpt function, which will be triggered by getting the best reward.

• train_iter (int): Current training iteration.

• envstep (int): Current env interaction step.

• n_episode (int): Number of evaluation episodes.

Returns:

• stop_flag (bool): Whether this training program can be ended.

• eval_reward (float): Current eval_reward.

reset(_policy: Optional[collections.namedtuple] = None, _env: Optional[ding.envs.env_manager.base_env_manager.BaseEnvManager] = None) → None

Overview:

Reset evaluator’s policy and environment. Use new policy and environment to collect data. If _env is None, reset the old environment. If _env is not None, replace the old environment in the evaluator with the new passed in environment and launch. If _policy is None, reset the old policy. If _policy is not None, replace the old policy in the evaluator with the new passed in policy.

Arguments:

• policy (Optional[namedtuple]): the api namedtuple of eval_mode policy

• env (Optional[BaseEnvManager]): instance of the subclass of vectorized env_manager(BaseEnvManager)

reset_env(_env: Optional[ding.envs.env_manager.base_env_manager.BaseEnvManager] = None) → None

Overview:

Reset evaluator’s environment. In some case, we need evaluator use the same policy in different environments. We can use reset_env to reset the environment. If _env is None, reset the old environment. If _env is not None, replace the old environment in the evaluator with the new passed in environment and launch.

Arguments:

• env (Optional[BaseEnvManager]): instance of the subclass of vectorized env_manager(BaseEnvManager)

reset_policy(_policy: Optional[collections.namedtuple] = None) → None

Overview:

Reset evaluator’s policy. In some case, we need evaluator work in this same environment but use different policy. We can use reset_policy to reset the policy. If _policy is None, reset the old policy. If _policy is not None, replace the old policy in the evaluator with the new passed in policy.

Arguments:

• policy (Optional[namedtuple]): the api namedtuple of eval_mode policy

should_eval(train_iter: int) → bool

Overview:

Determine whether you need to start the evaluation mode, if the number of training has reached the maximum number of times to start the evaluator, return True

VectorEvalMonitor

class ding.worker.collector.base_serial_evaluator.VectorEvalMonitor(env_num: int, n_episode: int)

Overview:

In some cases, different environment in evaluator may collect different length episode. For example, suppose we want to collect 12 episodes in evaluator but only have 5 environments, if we didn’t do any thing, it is likely that we will get more short episodes than long episodes. As a result, our average reward will have a bias and may not be accurate. we use VectorEvalMonitor to solve the problem.

Interfaces:

__init__, is_finished, update_info, update_reward, get_episode_reward, get_latest_reward, get_current_episode, get_episode_info

__init__(env_num: int, n_episode: int) → None

Overview:

Init method. According to the number of episodes and the number of environments, determine how many episodes need to be opened for each environment, and initialize the reward, info and other information

Arguments:

• env_num (int): the number of episodes need to be open

• n_episode (int): the number of environments

get_current_episode() → int

Overview:

Get the current episode. We can know which episode our evaluator is executing now.

get_episode_info() → dict

Overview:

Get all episode information, such as total reward of one episode.

get_episode_reward() → list

Overview:

Get the total reward of one episode.

get_latest_reward(env_id: int) → int

Overview:

Get the latest reward of a certain environment.

Arguments:

• env_id: (int): the id of the environment we need to get reward.

is_finished() → bool

Overview:

Determine whether the evaluator has completed the work.

Return:

• result: (bool): whether the evaluator has completed the work

update_info(env_id: int, info: Any) → None

Overview:

Update the information of the environment indicated by env_id.

Arguments:

• env_id: (int): the id of the environment we need to update information

• info: (Any): the information we need to update

update_reward(env_id: int, reward: Any) → None

Overview:

Update the reward indicated by env_id.

Arguments:

• env_id: (int): the id of the environment we need to update the reward

• reward: (Any): the reward we need to update