OpenAI Retro Contest – Compilation of Reinforcement Learning Write-Ups

Compilation of all the Write-Ups which i found on Discord/Internet for the OpenAI Reinforcement Learning Challenge.

Leaderboard status after the contest has ended.

1. World Models applied to Sonic

https://dylandjian.github.io/world-models/

Github link: https://github.com/dylandjian/retro-contest-sonic

Write-Up based on the World Models paper: https://arxiv.org/pdf/1803.10122.pdf

2. Train a Reinforcement Learning agent to play custom levels of Sonic the Hedgehog with Transfer Learning

https://flyyufelix.github.io/2018/06/11/sonic-rl.html

Write-Up on Transfer Learning. This implementation is for the 5th place in leaderboard with a score of 5522.

Read More

OpenAI Retro Contest – Everything I know about JERK agent

The first approach in the OpenAI Retro Contest which I started to implement, test and modify was the JERK approach. Jerk agent is one of the baseline scripts for this contest.

You can find it here: https://github.com/openai/retro-baselines

I think it is the easiest algorithm to understand for programmers who doesn’t have any Machine Learning experience.

The pseudo-code for the JERK algorithm looks like this:

Why is this the easiest approach? Because this algorithm is based on rewards. But not the same kind of rewards like rainbow or ppo2. JERK algorithm has the moves already scripted before. It doesn’t learn the same way like the two others. Sonic runs forward and jumps and if it scores points or progresses on the level further it gets rewarded. It learns based on rewards and tries to not make the mistakes again, because making a mistake will cost him “reward points”. It’s somehow like with us, humans. We are motivated to do something if we get a possible reward at the end.

Read More

OpenAI Retro Contest – Render .bk2 files on Windows

How to render .bk2 files on Windows.

This tutorial is an follow-up on Docker Settings tutorial:

https://www.noob-programmer.com/openai-retro-contest/docker-settings/

After creating and running our local environment we want to see the results of our agent playing the game.

I am not sure if there is another possibility playing .bk2 files on Windows than through python script. I’ve tried different .bk2 players or codecs but none of them seemed to be working.

After I tried to follow this guide on playback and rendering video : https://github.com/openai/retro

import retro

movie = retro.Movie('SonicTheHedgehog-Genesis-GreenHillZone.Act1-0000.bk2')
movie.step()

env = retro.make(game=movie.get_game(), state=retro.STATE_NONE, use_restricted_actions=retro.ACTIONS_ALL)
env.initial_state = movie.get_state()
env.reset()

while movie.step():
    keys = []
    for i in range(env.NUM_BUTTONS):
        keys.append(movie.get_key(i))
    _obs, _rew, _done, _info = env.step(keys)

and calling with script

python scripts/playback_movie.py SonicTheHedgehog-Genesis-GreenHillZone.Act1-0000.bk2

This solution didn’t really work.

I can guess that this solution could work on mac or Linux.

After some research i found this tutorial:

View story at Medium.com

This solution I guess would work on mac, but if you are a Windows user we get this error:

    assert not pass_fds, "pass_fds not supported on Windows."
AssertionError: pass_fds not supported on Windows.

The thing is we actually don’t need to export it to .mp4 file. We just can run the script and run it as playback.

So… what works?

Copy this script and call it “render.py”

#!/usr/bin/python

import sys
import retro
from os import listdir
from os.path import isfile, join, isdir


def render(file):
    movie = retro.Movie(file)
    movie.step()

    env = retro.make(game=movie.get_game(), state=retro.STATE_NONE, use_restricted_actions=retro.ACTIONS_ALL)
    env.initial_state = movie.get_state()
    env.reset()
    frame = 0
    framerate = 2
    while movie.step():
        if frame == framerate:
            env.render()
            frame = 0
        else:
            frame += 1

        keys = []
        for i in range(env.NUM_BUTTONS):
            keys.append(movie.get_key(i))
        _obs, _rew, _done, _info = env.step(keys)
    env.close()
if isdir(sys.argv[1]):
    onlyfiles = [f for f in listdir(sys.argv[1]) if isfile(join(sys.argv[1], f))]
    onlyfiles.sort()
    for file in onlyfiles:
        if ".bk2" in file :
            print('playing', file)
            render(sys.argv[1]+file)
else:
    print('playing', sys.argv[1])
    render(sys.argv[1])

Put it in scripts folder.

And than call this in your bash:

python ./scripts/render.py ./results/bk2/SonicTheHedgehog-Genesis-GreenHillZone.Act1-0001.bk2

PS: You get the results folder and .bk2 files after you run your local evaluation.

If everything works fine you get your script running:

This way we rendered only one file.

If you want to render whole folder on one run just run it this way:

python ./scripts/render.py ./results/bk2/

What else?

The script is running on frame rate with value 2. If you want to make the playback slower or faster just change the frame rate value to 1 (slower) or 5 (faster).

Is there a possibility to render the .bk2 files into .mp4 files to upload on YouTube?

Maybe.

I didn’t find the solution yet. But I can guess that if you edit this script:

https://github.com/openai/retro/blob/master/retro/scripts/playback_movie.py

and find a way to render it without using pass_fds than you can render the .bk2 files to .mp4 or other format.

That’s all for rendering .bk2 files with windows. Thanks for reading!