OpenAI Retro Contest – Everything I know about JERK agent

The first approach in the OpenAI Retro Contest which I started to implement, test and modify was the JERK approach. Jerk agent is one of the baseline scripts for this contest.

You can find it here: https://github.com/openai/retro-baselines

I think it is the easiest algorithm to understand for programmers who doesn’t have any Machine Learning experience.

The pseudo-code for the JERK algorithm looks like this:

Why is this the easiest approach? Because this algorithm is based on rewards. But not the same kind of rewards like rainbow or ppo2. JERK algorithm has the moves already scripted before. It doesn’t learn the same way like the two others. Sonic runs forward and jumps and if it scores points or progresses on the level further it gets rewarded. It learns based on rewards and tries to not make the mistakes again, because making a mistake will cost him “reward points”. It’s somehow like with us, humans. We are motivated to do something if we get a possible reward at the end.

Read More

OpenAI Retro Contest – Render .bk2 files on Windows

How to render .bk2 files on Windows.

This tutorial is an follow-up on Docker Settings tutorial:

https://www.noob-programmer.com/openai-retro-contest/docker-settings/

After creating and running our local environment we want to see the results of our agent playing the game.

I am not sure if there is another possibility playing .bk2 files on Windows than through python script. I’ve tried different .bk2 players or codecs but none of them seemed to be working.

After I tried to follow this guide on playback and rendering video : https://github.com/openai/retro

import retro

movie = retro.Movie('SonicTheHedgehog-Genesis-GreenHillZone.Act1-0000.bk2')
movie.step()

env = retro.make(game=movie.get_game(), state=retro.STATE_NONE, use_restricted_actions=retro.ACTIONS_ALL)
env.initial_state = movie.get_state()
env.reset()

while movie.step():
    keys = []
    for i in range(env.NUM_BUTTONS):
        keys.append(movie.get_key(i))
    _obs, _rew, _done, _info = env.step(keys)

and calling with script

python scripts/playback_movie.py SonicTheHedgehog-Genesis-GreenHillZone.Act1-0000.bk2

This solution didn’t really work.

I can guess that this solution could work on mac or Linux.

After some research i found this tutorial:

View story at Medium.com

This solution I guess would work on mac, but if you are a Windows user we get this error:

    assert not pass_fds, "pass_fds not supported on Windows."
AssertionError: pass_fds not supported on Windows.

The thing is we actually don’t need to export it to .mp4 file. We just can run the script and run it as playback.

So… what works?

Copy this script and call it “render.py”

#!/usr/bin/python

import sys
import retro
from os import listdir
from os.path import isfile, join, isdir


def render(file):
    movie = retro.Movie(file)
    movie.step()

    env = retro.make(game=movie.get_game(), state=retro.STATE_NONE, use_restricted_actions=retro.ACTIONS_ALL)
    env.initial_state = movie.get_state()
    env.reset()
    frame = 0
    framerate = 2
    while movie.step():
        if frame == framerate:
            env.render()
            frame = 0
        else:
            frame += 1

        keys = []
        for i in range(env.NUM_BUTTONS):
            keys.append(movie.get_key(i))
        _obs, _rew, _done, _info = env.step(keys)
    env.close()
if isdir(sys.argv[1]):
    onlyfiles = [f for f in listdir(sys.argv[1]) if isfile(join(sys.argv[1], f))]
    onlyfiles.sort()
    for file in onlyfiles:
        if ".bk2" in file :
            print('playing', file)
            render(sys.argv[1]+file)
else:
    print('playing', sys.argv[1])
    render(sys.argv[1])

Put it in scripts folder.

And than call this in your bash:

python ./scripts/render.py ./results/bk2/SonicTheHedgehog-Genesis-GreenHillZone.Act1-0001.bk2

PS: You get the results folder and .bk2 files after you run your local evaluation.

If everything works fine you get your script running:

This way we rendered only one file.

If you want to render whole folder on one run just run it this way:

python ./scripts/render.py ./results/bk2/

What else?

The script is running on frame rate with value 2. If you want to make the playback slower or faster just change the frame rate value to 1 (slower) or 5 (faster).

Is there a possibility to render the .bk2 files into .mp4 files to upload on YouTube?

Maybe.

I didn’t find the solution yet. But I can guess that if you edit this script:

https://github.com/openai/retro/blob/master/retro/scripts/playback_movie.py

and find a way to render it without using pass_fds than you can render the .bk2 files to .mp4 or other format.

That’s all for rendering .bk2 files with windows. Thanks for reading!

OpenAI Retro Contest – Docker Settings

Installing Docker on Windows 10 for Education

What we did before

We installed our gym retro and got our sonic game running with simple algorithm.

Last post: https://www.noob-programmer.com/machine-learning/how-to-install-gym-retro/

PS: It’s actually computer playing sonic by running this script:

import gym_remote.exceptions as gre
import gym_remote.client as grc

from retro import make
def main():
    print('connecting to remote environment')
    env = make(game='SonicTheHedgehog-Genesis', state='GreenHillZone.Act1')
    print('starting episode')
    env.reset()
    while True:
        env.render()
        action = env.action_space.sample()
        action[7] = 1
        ob, reward, done, _ = env.step(action)
        if done:
            print('episode complete')
            env.reset()


if __name__ == '__main__':
    try:
        main()
    except gre.GymRemoteError as e:
        print('exception', e)

We want now to upload our agent to the server and we can do it through Docker.

PS2: The above example wont work on server because we use local enviroment!

from retro import make and env = make(game=’SonicTheHedgehog-Genesis’, state=’GreenHillZone.Act1′)

Intro

Follow the instructions on page:

https://contest.openai.com/details

Because of Personal Settings i would suggest to register account and log in first.

This way you get auto-fill of this form:

export DOCKER_REGISTRY=<docker registry url>
docker login >$DOCKER_REGISTRY \
    --username <docker registry username> \
    --password <docker registry password>

For test purposes i would first curl the files:

mkdir simple-agent
cd simple-agent
curl -O https://contest.openai.com/static/simple-agent.py
curl -O https://contest.openai.com/static/simple-agent.docker
docker build -f simple-agent.docker -t $DOCKER_REGISTRY/simple-agent:v1 .

and

docker push $DOCKER_REGISTRY/simple-agent:v1

The next thing are local evaluations.

docker pull openai/retro-env
docker tag openai/retro-env remote-env

and running the agent:

retro-contest run --agent $DOCKER_REGISTRY/simple-agent:v1 \
    --results-dir results --no-nv --use-host-data \
    SonicTheHedgehog-Genesis GreenHillZone.Act1

If you run all this in git-bash, there is a possibility that you can’t close the agent.

I would than open new terminal and wrote:

docker ps # get the id of the running container
docker stop <container> # kill it (gracefully)

This way if you did something wrong you get an error message saying what went wrong.

You can see the scores and logs of the local evaluation if you open results folder and look at the log.csv and monitor.csv

If the local evaluation went good, we can upload the same agent on the page.

Go to “Jobs” on page and submit your agent: simple-agent:v1

Other Things:

If you want to upload your scenario.json with your jerk agent and include it in your agent i was told it is not possible. With the jerk agent from retro/baselines we can only edit the script. We can’t modify the scenario.json file.

To run the docker you need to have docker app installed.

Before configuration docker for windows i would suggest to create an account password. Account password is needed to configure docker app.

Other Blog posts on the same topic:

Jerk.agent configuration

View story at Medium.com

Tensorflow PPO2 Algorithm

View story at Medium.com

Configuring on Mac:

https://docs.google.com/document/d/1IzgkjvtHSYkzwZMhBjzXF3w4BxLBBlPjcku40eu2tEU/edit#heading=h.civ65uq80m51

Building your own Images:

View story at Medium.com

 

OpenAI Retro Contest – How To Install Gym Retro – Possible Problems

If you want to learn more about Machine Learning and AI there is great contest running on https://contest.openai.com/

How to install retro environment you can find on: https://contest.openai.com/details and on github https://github.com/openai/retro

The guide how to install all modules is great, but problems can still arise. After four days of trying i finally made it and it finally works!

System: Windows 10, 64 Bit.

So… what did i do wrong? Why did i took me so long?

– python 3.6.5 but 32- bit version

– installed gcc the wrong way

– tried to install gcc manually

– tried running through pyCharm.

So let’s start with with Python Version:

https://github.com/openai/retro/issues/32

You can find out what kind of version of python you have by looking at the Icon name. If you have 32 Bit Version it won’t work.

https://www.python.org/downloads/windows/

You need to install 64 bit Python. My version is : Python 3.6.5 – 2018-03-28

Installing GCC

One of the requirements to run retro gym is gcc 5 or higher.

I did it through cygwin.

Download 64 bit version from:

https://cygwin.com/install.html

And follow this tutorial:

http://preshing.com/20141108/how-to-install-the-latest-gcc-on-windows/

Possible problems:

You need to copy setup-x86_64.exe from downloads to your cygwin folder.

You run this command thought cmd in your cygwin folder:

(Don’t forget, it is CMD not Cygwin!)

C:\cygwin64>setup-x86_64.exe -q -P wget -P gcc-g++ -P make -P diffutils -P libmpfr-devel -P libgmp-devel -P libmpc-devel won't work. Also if you are currently in the project folder you can just write: setup-x86_64.exe -q -P wget -P gcc-g++ -P make -P diffutils -P libmpfr-devel -P libgmp-devel -P libmpc-devel

If you are getting the gcc package don’t forget to change

wget http://ftpmirror.gnu.org/gcc/gcc-4.9.2/gcc-4.9.2.tar.gz

to:

wget http://ftpmirror.gnu.org/gcc/gcc-7.3.0/gcc-7.3.0.tar.gz

We want the newer version after all. The same thing here:

../gcc-4.9.2/configure --program-suffix=-4.9.2 --enable-languages=c,c++ --disable-bootstrap --disable-shared

replace with:

../gcc-7.3.0/configure --program-suffix=-7.3.0 --enable-languages=c,c++ --disable-bootstrap --disable-shared

At the end you should test the compiler. If it works we are ready to go!

Install from binary

Next you can install environment through pip3:

pip3 install gym-retro

Running through pyCharm

On the details page: https://contest.openai.com/details

There is this script:

import retro


def main():
    env = retro.make(game='Airstriker-Genesis', state='Level1')
    obs = env.reset()
    while True:
        obs, rew, done, info = env.step(env.action_space.sample())
        env.render()
        if done:
            obs = env.reset()


if __name__ == '__main__':
    main()

But if you run it through IDE you get this:


C:\Users\programmer\PycharmProjects\untitled1\venv\Scripts\python.exe C:/Users/programmer/PycharmProjects/untitled1/retro-agent.py
Traceback (most recent call last):
File "C:/Users/programmer/PycharmProjects/untitled1/retro-agent.py", line 1, in <module>
import retro
ModuleNotFoundError: No module named 'retro'

Process finished with exit code 1

The thing is you need to run this script through terminal. Just run your script in terminal :

python retro-agent.py

And you should get the ai playing the game Airstriker-Genesis on our screen.

Other problems:

Don’t forget to Add Python to your PATH!

GCC Needs to be 5 or Higher!

64 Bit versions of programs!

Run the retro-agent.py throught terminal!

Installing from source, possible problems:

https://github.com/openai/retro/issues/30



							

Tutorial: How to access variable from other class in Unity

I have dealt with this problem very often.

How to get variables from other Class in Unity? For example I’ve got one Class with public Variables and i want to get the same value of the Variable in other Class.

For Example it could be a Class which adds points to overall Score. But in other Class i show it using Unity UI System as Text.

Here is the Class with Variable which is calling the Function in other Class.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class Score : MonoBehaviour {

  public GameObject scoreScript;
  // Use this for initialization

  public void OnTriggerEnter2D(Collider2D node)
  {
    if (node.gameObject.tag == "Apple") {
      Destroy (node.gameObject);
      ScoreMenager scorePointsScript =  scoreScript.GetComponent<ScoreMenager>();
      scorePointsScript.AddScore ();

    }
  }
}

 

And here is the Class which contains the Function where the variable points gets incremented.

using System.Collections.Generic;
using UnityEngine.UI;
using UnityEngine;

public class ScoreMenager : MonoBehaviour {
  public int points;


  Text text;

  void Awake ()
  {
    text = GetComponent<Text> ();

  }
  // Use this for initialization
  void Start () {
  }

  public void AddScore()
  {
    points++;
  }
  // Update is called once per frame
  void Update () {
    

    text.text = "Score: " + points;
  }
}

 

Don’t Forget to Add The “Score” Script to your character! And the same for ScoreMenager, which needs to be added to the UI Element.

And also don’t forget to Create the Text as GUI

Also important is the fact, that you need to change the “Tag” of your element which will be destroyed. After it increases the Score Points value.

Component > UI > Text

 

Other tutorials in the series:

How to make the objects fall in Unity

Making Objects Fall Random On The Screen in Unity

How to Speed Up (Increase Speed) Time in Unity

How to create a simple countdown Timer in Unity

Rendering Crisp Pixelart in Phaser (2017)

I had this problem while building my pixelart game in Phaser. My Pixelart was blurry. It didn’t looked nice. I was searching the web for an answer but i didn’t found one. Many of the tutorials were outdated so i decided to write my solution here.

You can find the working example here: https://www.noob-programmer.com/pixelart_example/

Example above works for Chrome and Mozilla Firefox.

Solution includes editing CSS and one line in phaser script.

Here is the CSS which you need to apply to your html element:

body {
  filter: none;
  image-rendering: -moz-crisp-edges;
  image-rendering: -webkit-crisp-edges;
  image-rendering: pixelated;
  image-rendering: crisp-edges;
}

Used it on body element.

The other line is the Phaser script:
var game = new Phaser.Game(200, 150, Phaser.AUTO, 'gameContainer', {
      preload: preload,
      create: create,
      update: update
    }, null, false, false);

The last parameter (“false”) is for antialiasing. It needs to be set to false.

What about other browsers? What do i need to change in order to get crispy Pixelart? Check this solution:

body {
  -ms-interpolation-mode: nearest-neighbor; // IE 7+ (non-standard property)
  image-rendering: -webkit-optimize-contrast; // Safari 6, UC Browser 9.9
  image-rendering: -webkit-crisp-edges; // Safari 7+
  image-rendering: -moz-crisp-edges; // Firefox 3.6+
  image-rendering: -o-crisp-edges; // Opera 12
  image-rendering: pixelated; // Chrome 41+ and Opera 26+
}

Source: https://builtvisible.com/image-scaling-in-css/

Source: https://developer.mozilla.org/en-US/docs/Games/Techniques/Crisp_pixel_art_look

How to create a simple countdown Timer in Unity

How to make a timer (countdown Clock from 60) as you see in the GIF below:

First you need to create a new Object on the canvas. Choose from the Menu:

GameObject -> UI ->Text

Than remove the “New Text” from the Inspector where the window is labeled “Text”. If you want you can change the size of the font in Inspector by changing the value “Font Size”.

To your object in Unity add a new script called: “Countdown”.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
using System.Threading;

public class CountDown : MonoBehaviour {

  public int timeLeft = 60; //Seconds Overall
  public Text countdown; //UI Text Object

  void Start () {
    StartCoroutine("LoseTime");
    Time.timeScale = 1; //Just making sure that the timeScale is right
  }

  void Update () {
    countdown.text = ("" + timeLeft); //Showing the Score on the Canvas
  }

  //Simple Coroutine
  IEnumerator LoseTime()
  {
    while (true) {
      yield return new WaitForSeconds (1);
      timeLeft--; 
    }

  }}

Don’t forget to add the Text object to our script too!

 

You can change the Font too. In the gif example we used different one, than the standard Arial font.

Other tutorials in the series:

How to make the objects fall in Unity

Making Objects Fall Random On The Screen in Unity

How to Speed Up (Increase Speed) Time in Unity

How to Speed Up (Increase Speed) Time in Unity

How to Increase Speed in your game?

This tutorial can be helpful if you want to create boosters which increase for example the character speed.

All you need to write if you want to increase the speed in your game is this one line of code:

Time.timeScale = 2f;

As you see the value is two, which means the speed will be doubled. You can use any other value. It depends on you how you want your game to react. But remember! The value can’t be lower than 1! If you use as value for example “0.5” the speed will decrease.

In my code it is upon the collision with the Red Timer object.

public float speedUp = 2.5f;

//public variable at the beginning of the code 


if (col.gameObject.tag == "SpeedUp") {
      Time.timeScale = speedUp;
    }

If you want your Time effect last only for 5 seconds you can use Coroutines and function called WaitForSeconds(5f);

if (col.gameObject.tag == "SpeedUp") {
    StartCoroutine( FiveSecondsCooldown());
  }


We just simply call this function using Coroutine.

public IEnumerator FiveSecondsCooldown() {
  Time.timeScale = 2.5f;
  yield return new WaitForSeconds(5f); // 
  Time.timeScale = 1;
}

As you see. The Time.timeScale value has changed to 2.5, than after 5 seconds went back to normal value, which is 1.

Other tutorials in the series:

How to make the objects fall in Unity

Making Objects Fall Random On The Screen in Unity

How to make the objects fall in Unity

There are a lot of ways how to make the objects fall in Unity. In our game we are using two of them.

Why did the left timer hit the ground first? After all the red watch was a little higher than the blue one. Thanks to Physics! And artificial Gravitational pull.

While the left watch was taking on the speed and falling faster in every millisecond (for example like cars do) the right one had a steady linear falling speed. You can use this in a lot of ways.

Why don’t we stick with one falling type? Because we wanted to differentiate the two falling types for different objects.

If you want to use the right way of objects falling (Blue Timer) you can use this script below and assign it to the object.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class TimerFall : MonoBehaviour {

  public float fallSpeed = 40.0f;      //How fast should the object fall

  // Update is called once per frame
  void Update () {

      transform.Translate(Vector3.down * fallSpeed * Time.deltaTime, Space.World);
  }
}

As you see. It’s not much. In fact we need only to write two lines of code. Variable initialization which is responsible for the speed of falling and the transform.Translate function. What does the function do? It just changes the position of an object in the Scene. That’s all.

In the left example (Red Timer) we don’t any code at all!

You just need to use build-in function called Rigidbody 2D. On the object “Add Component” => Physics 2D => Rigidbody 2D

What kind of numbers (variables) do i need to put there to make the physics work? It depends on what do you want to achieve. I am focusing most of the time on “Mass” and Gravity Scale. Sometimes i change the Linear Drag and Angular drag. But not so often.

Here are the variables used in the example above:

Well that’s all you need to know. Thanks for reading.