Playing Swarm Simulator with a BOT in Elixir (part 1)

02 Feb 2016 17 mins read

Hey! Check out this Swarm Simulator. Wouldn’t that be great to have a fully automated solution to grow the swarm for you? How does it feel to be, you know, the coolest farmer in the neighbourhood using the most fancy farming gadgets? I guarantee you that after finishing this tutorial you will know exactly how it feels.

This is a detailed tutorial on how I wrote the code. I’m not very experienced with Elixir therefore you may find some bits weird or overcomplicated. I guess this is natural for learning process. I encurage you to leave comments on what improvements could be done. I will gladly apply them. A Pull Request on Github would be even more welcome!

Full source of the BOT on Github: https://github.com/RadekMolenda/SwarmSimulatorBOT

Technology

Here is what we are going to use:

Elixir
Phantomjs
and… A BOT (we will write one in this article)

Phantomjs we will use Elixir library for browser automation called hound it integrates nicely with phantomjs. Another reason for picking phantomjs is that it’s headless - setting it up on a server should be trivial.

Elixir is the best choice as I just decided to learn this amazing language. We will also benefit from some cool features of erlang like :timer.send_interval/3. It seems like writing it in Ruby wouldn’t be as easy.

Setup

Let’s start with using the Elixir built in build tool mix

mix new swarmsimulatorbot && cd swarmsimulatorbot

This command should output something like:

* creating README.md
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/swarmsimulatorbot.ex
* creating test
* creating test/test_helper.exs
* creating test/swarmsimulatorbot_test.exs
Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:
    cd swarmsimulatorbot
    mix test
Run "mix help" for more commands.

Implementation

We are ready to do some coding now (well almost as we need to install some dependencies before). As I mentioned earlier we will use hound for browser automation. Let’s add it to the codebase, function deps is defined at the end of mix.exs file


1
2
3
4
5
6
7 #file: mix.exs

  defp deps do
    [
      {:hound, "~> 0.8"}
    ]
  end

We can install the library by running the following command in shell

mix deps.get

We should also start hound process when the app starts we will do it by adding :hound to the application function.


1
2
3
4
5 #file: mix.exs

  def application do
    [applications: [:logger, :hound]]
  end

As well as let hound know we will be using phantomjs driver


1
2 #file: config/config.exs
config :hound, driver: "phantomjs"

Starting the hound and finding units

Now when the setup part is done. I would like to write some tests first. In this step I don’t really know what kind of functions and modules I would like to write. Being not experienced in Elixir is also disadvantage as it makes BDD even more difficult. Anyways - it doesn’t really matter we will just start writing and see where we get with it. Let’s open the autogenerated test file test/swarmsimulatorbot_test.exs and change it to look more or less like that:


1
2
3
4
5
6
7
8
9
10
11
12
13 #file: test/swarmsimulatorbot_test.exs

defmodule SwarmsimulatorbotTest do
  use ExUnit.Case
  use Hound.Helpers
  doctest Swarmsimulatorbot

  test "Initial number of units should be three" do
    Swarmsimulatorbot.start
    assert length(Swarmsimulatorbot.units) == 3
    Swarmsimulatorbot.stop
  end
end

You can see there is quite a lot of things going on here. Thanks to use Hound.Helpers we will have access to useful hound helpers functions. You can also see that I want my Swarmsimulatorbot module to respond to three new functions: start/0, units/0 and stop/0.

After looking at hounds simple browser automation readme, I decided that starting hound session and performing some basic steps (like navigating to swarm simulator URL) will be done in start/0. stop/0 will be responsible for stopping the hound session and units/0 should return all swarm units DOM elements after clicking Show all units link. Initialy there will be 3 units available and therefore length(Swarmsimulatorbot.units) should be 3.

There is one thing I don’t like about those tests tho - we will be making a real http request. I was trying to solve this issue by using exvcr library, but integrating it using my current my knowledge is far beyond my skills. I ended up leaving the tests like that but that’s definitely something I would like to improve in next iteration.

Let’s run our tests in shell

mix test

This should give us more or less the following output


  1) test Initial number of units should be three (SwarmsimulatorbotTest)
     test/swarmsimulatorbot_test.exs:6
     ** (UndefinedFunctionError) undefined function Swarmsimulatorbot.start/0
     stacktrace:
       (swarmsimulatorbot) Swarmsimulatorbot.start()
       test/swarmsimulatorbot_test.exs:7



Finished in 0.07 seconds (0.07s on load, 0.00s on tests)
1 test, 1 failure

Let’s fix this and next errors by implementing the correct functions in Swarmsimulatorbot module:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30 #file: lib/swarmsimulatorbot.ex
defmodule Swarmsimulatorbot do
  use Hound.Helpers

  @swarm_url "https://swarmsim.github.io/"

  def start do
    Hound.start_session
    navigate_to(@swarm_url)
    execute_script("localStorage.clear()");
  end

  def stop do
    Hound.end_session
  end

  def units do
    show_all_units
    find_all_elements(:css, ".unit-table tr")
  end

  defp show_all_units do
    click_on_text("More...")
    click_on_text("Show all units")
  end

  defp click_on_text(text) do
    find_element(:link_text, text) |> click
  end
end

Let’s repeat our tests in shell

mix test

The error message should look a bit more serious now

  1) test Initial number of units should be three (SwarmsimulatorbotTest)
     test/swarmsimulatorbot_test.exs:6
     ** (exit) exited in: GenServer.call(Hound.SessionServer, {:change_session, #PID<0.175.0>, :default, %{}}, 60000)
         ** (EXIT) an exception was raised:
             ** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}
                 (hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4
                 (hound) lib/hound/session_server.ex:67: Hound.SessionServer.handle_call/3
                 (stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
                 (stdlib) gen_server.erl:661: :gen_server.handle_msg/5
                 (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
     stacktrace:
       (elixir) lib/gen_server.ex:564: GenServer.call/3
       (swarmsimulatorbot) lib/swarmsimulatorbot.ex:5: Swarmsimulatorbot.start/0
       test/swarmsimulatorbot_test.exs:7

Finished in 0.1 seconds (0.08s on load, 0.1s on tests)
1 test, 1 failure
Randomized with seed 199589
20:17:32.354 [error] GenServer Hound.SessionServer terminating
** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}
    (hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4
    (hound) lib/hound/session_server.ex:67: Hound.SessionServer.handle_call/3
    (stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:661: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {:change_session, #PID<0.175.0>, :default, %{}}
State: %{}

Apparently we forgot about one important element {:error, %HTTPoison.Error{id: nil, reason: :econnrefused}} should give us a clue what went wrong. Taking another look at hound documentation finally gives the explanation:

You’ll need a webdriver server running

Let’s start a webserver then by running the following command in another shell (we need to keep it running all the time from now on)


phantomjs --wd

mix test is finally green


Compiled lib/swarmsimulatorbot.ex
.

Finished in 1.2 seconds (0.07s on load, 1.1s on tests)
1 test, 0 failures

Taking screenshots

We need to check the health of our swarm from time to time. I think the best way to do so is to make screenshots.

The test:


1
2
3
4
5
6
7
8
9 #file: test/swarmsimulatorbot_test.exs
  test "screenshot" do
    Swarmsimulatorbot.start
    f = "screenshots/test.png"
    Swarmsimulatorbot.screenshot("test.png")
    assert File.exists?(f)
    File.rm(f)
    Swarmsimulatorbot.stop
  end

I’m expecting mix test to complain about undefined function Swarmsimulatorbot.screenshot/1. Let’s add a missing function then.


1
2
3
4
5
6 #file: lib/swarmsimulatorbot.ex

  def screenshot(path) do
    show_all_units
    take_screenshot("screenshots/#{path}")
  end

We will only do screenshots on all units page for now as it’s really enough to check our swarm conditions. We don’t want to polute our root directory with some file images - this is why we will keep them in screenshots directory. Hound take_screenshot/1 helper function will take care about the rest.

Make sure you run mkdir screenshots before running tests. mix test should end with 2 tests, 0 failures.

Now we are ready to play with our Swarmsimulatorbot.screenshot/1 function in iex console

iex -S mix
iex(1)> Swarmsimulatorbot.start
nil
iex(2)> Swarmsimulatorbot.screenshot("hello-buggies.png")
"screenshots/hello-buggies.png"
iex(3)> Swarmsimulatorbot.stop
:ok

And if all went well you should endup with the image saved in your screenshots directory similar to this one

Hello Swarm

pretty neat! Looking at swarm is quite entertaining but we obviously want more - we want our BOT to actively grow our Swarm.

Growing the Swarm

If we want to grow our swarm we need to think about the strategy. How about?. Iterate over each unit and try to grow as much units as you can. Seems like a perfect starting strategy. It should be quite easy to implement - on each unit page find the last ‘clickable’ button and click it. We will call our strategy dummy_grow/0

In our test we will expect population of drones would increase after first call of dummy_grow/0 function.


1
2
3
4
5
6
7
8
9
10
11 #file: test/swarmsimulatorbot_test.exs
  test "#dummy_grow grows the swarm" do
    Swarmsimulatorbot.start
    Swarmsimulatorbot.dummy_grow
    drone_text = Swarmsimulatorbot.units
    |> Enum.at(2)
    |> inner_text

    assert drone_text =~ ~r/Drone.*3/
    Swarmsimulatorbot.stop
  end

We will fix test by implementing dummy_grow/0 function the following way.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19 #file: lib/swarmsimulatorbot.ex

  def dummy_grow do
    units_size = length(units) - 1
    Enum.each(0..(units_size), fn(index) ->
      units
      |> Enum.at(index)
      |> find_within_element(:tag, "a")
      |> click

      active_buttons
      |> List.last
      |> click
    end)
  end

  def active_buttons do
    find_all_elements(:css, "a:not(.disabled).btn")
  end

And this change satisfies the tests. The drones are growing!

If you look closer at the dummy_grow/0 function you might have noticed there is some redundancy. It would be so much simpler just to iterate over Swarmsimulatorbot.units/0 and not call units/0 one more time in Enum.each/2 - that’s very true, but apparently angularJS (the framework used for building Swarm simulator) reloads the page after each click. An attempt to iterate over units/0 and clicking it one by one would endup in error due to rest of the units not being present in DOM after first unit click.

Rest of the code seems quite self-explanatory. We have used some new hound helper functions and some standard Elixir programming to implement dummy grow functionality.

Kind of a server

We have written just enough to start iex session call Swarmsimulatorbot.dummy_grow/0 couple of times and take some screenshots to see that the swarm is actually growing. It is growing, isn’t it?

Now what’s left is to automate calling dummy_grow/0 and screenshot/1. I think this is the most interesting part. We will write it using processes.

Here is the idea: we will use two processes. One for keeping the browser session, clicking the buttons, growing our Swarm and taking screenshots. The second process will be responsible for sending two types of messages periodically to the first process:

:grow message - send every couple of seconds. It will force the second process to call dummy_grow/0 function
:screenshot message - send every minute. It will force the second process to call screenshot/1 function

Here are the implementation details:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24 #file: lib/swarmsimulatorbot.ex
  def start do
    spawn_link __MODULE__, :init, []
  end

  def init do
    Hound.start_session
    navigate_to(@swarm_url)
    execute_script("localStorage.clear()");
    loop
  end

  defp loop do
    receive do
      {:screenshot, path} ->
        screenshot(path)
        loop
      {:grow} ->
        dummy_grow
        loop
      {:stop} ->
        stop
    end
  end


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15 #file: lib/swarmsimulatorbot/cli.ex
defmodule Swarmsimulatorbot.Cli do
  @tick 1000
  @screenshot_tick 60000

  def start do
    spawn_link __MODULE__, :main, []
  end

  def main do
    bot_pid = Swarmsimulatorbot.start
    :timer.send_interval(@tick, bot_pid, {:grow})
    :timer.send_interval(@screenshot_tick, bot_pid, {:screenshot, "growing.png"})
  end
end

So - what is going on here. First of all I have moved the start/0 logic to init/0 function and let start/0 to spawn a process for us (apparently it is some kind of a standard in Elixir world). The key change was adding private function loop/0 this is a ‘place’ where a process will listen for incoming messages. Recursive calling of loop/0 is just a way of saying we want to recieve messages all the time and we don’t want to stop after receiving the first message.

The Swarmsimulatorbot.Cli module is responsible for:

starting the Swarmsimulatorbot process
periodically send messages to Swarmsimulatorbot process using :timer.send_interval/3 function

And believe me or not - this is the end of the part I. Let’s try this out in iex


iex -S mix
iex(1)> Swarmsimulatorbot.Cli.start
#PID<0.157.0>

And here is how my swarm stats looks like after about an hour of BOT running.

Swarm is growing

Sweet!

Unfortunatelly my BOT just stopped due to phantomjs timeout error.


07:49:20.798 [error] Process #PID<0.158.0> raised an exception
** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :timeout}}
    (hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4
    (swarmsimulatorbot) lib/swarmsimulatorbot.ex:41: anonymous fn/1 in Swarmsimulatorbot.dummy_grow/0
    (elixir) lib/enum.ex:610: anonymous fn/3 in Enum.each/2
    (elixir) lib/enum.ex:1478: anonymous fn/3 in Enum.reduce/3
    (elixir) lib/range.ex:80: Enumerable.Range.reduce/5
    (elixir) lib/enum.ex:1477: Enum.reduce/3
    (elixir) lib/enum.ex:609: Enum.each/2
    (swarmsimulatorbot) lib/swarmsimulatorbot.ex:28: Swarmsimulatorbot.loop/0

And we definitelly need to fix it. We need something that would look at our processes and respawn in case it crashes - some supervisor maybe. This might be it, but it also requires a bit more knowledge about OTP - another great Elixir feature inherited from Erlang. I’ll try to cover that in next part of this tutorial.