The implementation of some reinforcement learning algorithms refers to many open source projects on GitHub, and the framework design mainly refers to the implementation of rllib. At present, the following algorithms and game environments have been implemented:
Algorithms | Framework | Discrete Actions | Continuous Action |
---|---|---|---|
PPO | tf | Yes | Yes |
GameEnv | Demo ALgorithms |
---|---|
Gym CartPole-v1 | PPO |
Gym Atari BeamRiderNoFrameskip-v4 | PPO |
Gfootball | PPO |
TODO
run PYTHONPATH=./ python tests/test_ppotrainer_gym.py
run PYTHONPATH=./ python tests/test_ppotrainer_atari.py
To reproduce the training result of deepmind/rllib, some of the following tricks are important:
-
DeepMind atari enviroment wrappers, see
envs/wrappers/atari_wrappers.py
copid from rllib in facta. MonitorEnv for recording statistical variabels like _episode_reward * _episode_length
b. NoopResetEnv for doing some No-Operation after environment reset
c. MaxAndSkipEnv for Returning only every 4-th frame
d. WarpFrame for warrper the observation to 84x84x1
e. FireResetEnv for doing the fire-reseting
f. EpisodicLifeEnv for making one episode for one life
g. FrameStack for multi-frame-stack to observation
-
Set clip_reward True, clip the reward to [-1, 1]
-
Learning rate: 5e-5, epsilon: 1e-7
-
Other hparam according to config
Important reference indicators
- Episode Reward: According to your reward engine, the bigger the better
- explained_variance: It reflects the difference between your critic VF_OUT and VALUE_TARGET; In general, the closer to 1, the better
TODO