OpenAI Retro Contest – Compilation of Reinforcement Learning Write-Ups

Compilation of all the Write-Ups which i found on Discord/Internet for the OpenAI Reinforcement Learning Challenge.

Leaderboard status after the contest has ended.

1. World Models applied to Sonic

https://dylandjian.github.io/world-models/

Github link: https://github.com/dylandjian/retro-contest-sonic

Write-Up based on the World Models paper: https://arxiv.org/pdf/1803.10122.pdf

2. Train a Reinforcement Learning agent to play custom levels of Sonic the Hedgehog with Transfer Learning

https://flyyufelix.github.io/2018/06/11/sonic-rl.html

Write-Up on Transfer Learning. This implementation is for the 5th place in leaderboard with a score of 5522.

Read More

OpenAI Retro Contest – Everything I know about JERK agent

The first approach in the OpenAI Retro Contest which I started to implement, test and modify was the JERK approach. Jerk agent is one of the baseline scripts for this contest.

You can find it here: https://github.com/openai/retro-baselines

I think it is the easiest algorithm to understand for programmers who doesn’t have any Machine Learning experience.

The pseudo-code for the JERK algorithm looks like this:

Why is this the easiest approach? Because this algorithm is based on rewards. But not the same kind of rewards like rainbow or ppo2. JERK algorithm has the moves already scripted before. It doesn’t learn the same way like the two others. Sonic runs forward and jumps and if it scores points or progresses on the level further it gets rewarded. It learns based on rewards and tries to not make the mistakes again, because making a mistake will cost him “reward points”. It’s somehow like with us, humans. We are motivated to do something if we get a possible reward at the end.

Read More

OpenAI Retro Contest – Docker Settings

Installing Docker on Windows 10 for Education

What we did before

We installed our gym retro and got our sonic game running with simple algorithm.

Last post: https://www.noob-programmer.com/machine-learning/how-to-install-gym-retro/

PS: It’s actually computer playing sonic by running this script:

import gym_remote.exceptions as gre
import gym_remote.client as grc

from retro import make
def main():
    print('connecting to remote environment')
    env = make(game='SonicTheHedgehog-Genesis', state='GreenHillZone.Act1')
    print('starting episode')
    env.reset()
    while True:
        env.render()
        action = env.action_space.sample()
        action[7] = 1
        ob, reward, done, _ = env.step(action)
        if done:
            print('episode complete')
            env.reset()


if __name__ == '__main__':
    try:
        main()
    except gre.GymRemoteError as e:
        print('exception', e)

We want now to upload our agent to the server and we can do it through Docker.

PS2: The above example wont work on server because we use local enviroment!

from retro import make and env = make(game=’SonicTheHedgehog-Genesis’, state=’GreenHillZone.Act1′)

Intro

Follow the instructions on page:

https://contest.openai.com/details

Because of Personal Settings i would suggest to register account and log in first.

This way you get auto-fill of this form:

export DOCKER_REGISTRY=<docker registry url>
docker login >$DOCKER_REGISTRY \
    --username <docker registry username> \
    --password <docker registry password>

For test purposes i would first curl the files:

mkdir simple-agent
cd simple-agent
curl -O https://contest.openai.com/static/simple-agent.py
curl -O https://contest.openai.com/static/simple-agent.docker
docker build -f simple-agent.docker -t $DOCKER_REGISTRY/simple-agent:v1 .

and

docker push $DOCKER_REGISTRY/simple-agent:v1

The next thing are local evaluations.

docker pull openai/retro-env
docker tag openai/retro-env remote-env

and running the agent:

retro-contest run --agent $DOCKER_REGISTRY/simple-agent:v1 \
    --results-dir results --no-nv --use-host-data \
    SonicTheHedgehog-Genesis GreenHillZone.Act1

If you run all this in git-bash, there is a possibility that you can’t close the agent.

I would than open new terminal and wrote:

docker ps # get the id of the running container
docker stop <container> # kill it (gracefully)

This way if you did something wrong you get an error message saying what went wrong.

You can see the scores and logs of the local evaluation if you open results folder and look at the log.csv and monitor.csv

If the local evaluation went good, we can upload the same agent on the page.

Go to “Jobs” on page and submit your agent: simple-agent:v1

Other Things:

If you want to upload your scenario.json with your jerk agent and include it in your agent i was told it is not possible. With the jerk agent from retro/baselines we can only edit the script. We can’t modify the scenario.json file.

To run the docker you need to have docker app installed.

Before configuration docker for windows i would suggest to create an account password. Account password is needed to configure docker app.

Other Blog posts on the same topic:

Jerk.agent configuration

View story at Medium.com

Tensorflow PPO2 Algorithm

View story at Medium.com

Configuring on Mac:

https://docs.google.com/document/d/1IzgkjvtHSYkzwZMhBjzXF3w4BxLBBlPjcku40eu2tEU/edit#heading=h.civ65uq80m51

Building your own Images:

View story at Medium.com

 

OpenAI Retro Contest – How To Install Gym Retro – Possible Problems

If you want to learn more about Machine Learning and AI there is great contest running on https://contest.openai.com/

How to install retro environment you can find on: https://contest.openai.com/details and on github https://github.com/openai/retro

The guide how to install all modules is great, but problems can still arise. After four days of trying i finally made it and it finally works!

System: Windows 10, 64 Bit.

So… what did i do wrong? Why did i took me so long?

– python 3.6.5 but 32- bit version

– installed gcc the wrong way

– tried to install gcc manually

– tried running through pyCharm.

So let’s start with with Python Version:

https://github.com/openai/retro/issues/32

You can find out what kind of version of python you have by looking at the Icon name. If you have 32 Bit Version it won’t work.

https://www.python.org/downloads/windows/

You need to install 64 bit Python. My version is : Python 3.6.5 – 2018-03-28

Installing GCC

One of the requirements to run retro gym is gcc 5 or higher.

I did it through cygwin.

Download 64 bit version from:

https://cygwin.com/install.html

And follow this tutorial:

http://preshing.com/20141108/how-to-install-the-latest-gcc-on-windows/

Possible problems:

You need to copy setup-x86_64.exe from downloads to your cygwin folder.

You run this command thought cmd in your cygwin folder:

(Don’t forget, it is CMD not Cygwin!)

C:\cygwin64>setup-x86_64.exe -q -P wget -P gcc-g++ -P make -P diffutils -P libmpfr-devel -P libgmp-devel -P libmpc-devel won't work. Also if you are currently in the project folder you can just write: setup-x86_64.exe -q -P wget -P gcc-g++ -P make -P diffutils -P libmpfr-devel -P libgmp-devel -P libmpc-devel

If you are getting the gcc package don’t forget to change

wget http://ftpmirror.gnu.org/gcc/gcc-4.9.2/gcc-4.9.2.tar.gz

to:

wget http://ftpmirror.gnu.org/gcc/gcc-7.3.0/gcc-7.3.0.tar.gz

We want the newer version after all. The same thing here:

../gcc-4.9.2/configure --program-suffix=-4.9.2 --enable-languages=c,c++ --disable-bootstrap --disable-shared

replace with:

../gcc-7.3.0/configure --program-suffix=-7.3.0 --enable-languages=c,c++ --disable-bootstrap --disable-shared

At the end you should test the compiler. If it works we are ready to go!

Install from binary

Next you can install environment through pip3:

pip3 install gym-retro

Running through pyCharm

On the details page: https://contest.openai.com/details

There is this script:

import retro


def main():
    env = retro.make(game='Airstriker-Genesis', state='Level1')
    obs = env.reset()
    while True:
        obs, rew, done, info = env.step(env.action_space.sample())
        env.render()
        if done:
            obs = env.reset()


if __name__ == '__main__':
    main()

But if you run it through IDE you get this:


C:\Users\programmer\PycharmProjects\untitled1\venv\Scripts\python.exe C:/Users/programmer/PycharmProjects/untitled1/retro-agent.py
Traceback (most recent call last):
File "C:/Users/programmer/PycharmProjects/untitled1/retro-agent.py", line 1, in <module>
import retro
ModuleNotFoundError: No module named 'retro'

Process finished with exit code 1

The thing is you need to run this script through terminal. Just run your script in terminal :

python retro-agent.py

And you should get the ai playing the game Airstriker-Genesis on our screen.

Other problems:

Don’t forget to Add Python to your PATH!

GCC Needs to be 5 or Higher!

64 Bit versions of programs!

Run the retro-agent.py throught terminal!

Installing from source, possible problems:

https://github.com/openai/retro/issues/30