Testing Scripts With Aruba and Cucumber

Let’s face it. I’m really bad at bash-scripting. The fact that I use zsh on top of it, and use zsh scripting in a few of my scripts, have left me in a spot where I create scripts that are rather ragged, and unstable.

That would probably be okay for most of the scripts I write; convenient methods to trigger other scripts. Some others have grown over time, and have become rather important to me. Still, I tend to break them whenever I touch them. Thus I was searching for a solution to test them, and one solution I found was Aruba

Aruba is an extension framework on top of BDD frameworks like cucumber. The reason I looked into it is BDD, or Behavior Driven Development. The idea is to test the journey through an application via scenarios, and usually, I used it as a thin layer in my test pyramid with mixed success.

However, for most of my scripts, validating the behaviour seemed very valuable. I decided to give it a go.

The project

One of the scripts I use, that’s important to me but held together with good luck and wishful thinking, is managing a few of my settings in the zsh. I’m a person that has usually a lot of tabs open in my terminal, and as a consultant, I do private things (like writing this blog entry) on the work computer as well.

To avoid accidentally committing with my private user, and test with the wrong version of the JDK/node version, I needed a very defined session in my tabs, that should be easy to identify. For this, I created zsh-lightweight-modes, a plugin for zsh that allows me to switch between different modes with the command mode. This little script is working well until I have to switch to the next client, and something has to be added, or something special pops up. Clients don’t pay me for getting my terminal set up, so I had to invest a lot of private time to make sure the old things are still working whenever I add something new.

A few regression tests sound just like the thing to have, don’t they?

Getting started

Aruba allows running commands in a specific environment. The sanest way to make to sure I have a well-defined environment sounded like a docker container. I started with an Ubuntu and installed ruby and zsh. Via bundler I installed Aruba, and copied Gemfile, *.gemspec and feature-files together with the plugin.

The full Docker file can be found on the github site.

Once everything is set up, the tests can be run via bundle exec cucumber, so let’s write a few to see how well that goes.

The first test

Let’s create a feature file in the feature folder, and put the following content in:

Feature: An example of how to use Aruba

  Scenario: No mode selected
    When I run the following commands in `zsh`:
    """
    echo "Hello World!"
    """
    Then the stderr should not contain anything
    Then the stdout should contain "Hello World!"

If we execute it, we will see the following output

Feature: An example of how to use Aruba

  Scenario: No mode selected                      # features/example.feature:3
    When I run the following commands in `zsh`:   # features/example.feature:4
      """
      echo "Hello World!"
      """
    Then the stderr should not contain anything   # features/example.feature:8
    Then the stdout should contain "Hello World!" # features/example.feature:9

1 scenario (1 undefined)
3 steps (3 undefined)
0m0.040s

You can implement step definitions for undefined steps with these snippets:

When(/^I run the following commands in `zsh`:$/) do |string|
  pending # Write code here that turns the phrase above into concrete actions
end

Then(/^the stderr should not contain anything$/) do
  pending # Write code here that turns the phrase above into concrete actions
end

Then(/^the stdout should contain "([^"]*)"$/) do |arg1|
  pending # Write code here that turns the phrase above into concrete actions
end

It basically tells us to implement the steps we have defined. This is were Aruba can help us. Add an env.rb file in the features/support folder with the following line:

require 'aruba/cucumber'

If we run the test again, this is our new result:

Feature: An example of how to use Aruba

  Scenario: No mode selected                      # features/example.feature:3
    When I run the following commands in `zsh`:   # aruba-1.0.0/lib/aruba/cucumber/command.rb:15
      """
      echo "Hello World!"
      """
    Then the stderr should not contain anything   # aruba-1.0.0/lib/aruba/cucumber/command.rb:370
    Then the stdout should contain "Hello World!" # aruba-1.0.0/lib/aruba/cucumber/command.rb:122

1 scenario (1 passed)
3 steps (3 passed)
0m0.139s

The tests passed! Aruba implements the steps for us, and we can simply use them. The Aruba Website gives all possible tests as generated documentation from the BDD tests for Aruba - which is neat, because not only does it give you living documentation, but also a source to copy and paste your tests together.

Digging deeper

When you start using something, getting into yak-shaving mode is an easy thing to do. I wanted to test a zsh plugin, and mu rabbit hold was in figuring out on how to set up zsh in the test. Let’s look at one of the tests in a way in which it fails:

  Scenario: Running the help
    When I run the following commands with `zsh`:
    """
    mode help
    """
    Then the stderr should not contain anything
    And the stdout should contain "Usage:"
    And the stdout should contain "- minimal"

The test fails on the stderr, because mode: command not found. If you connect to the docker container and open the zshell it will work. However, Aruba launches the command as zsh -c yourtext, and zshell will not load the .zshrc in that case. To make it pass, we have to source it manually for each call. To make the test pass, the code will have to look like this:

  Scenario: Running the help
    When I run the following commands with `zsh`:
    """
    source ~/.zshrc
    mode help
    """
    Then the stderr should not contain anything
    And the stdout should contain "Usage:"
    And the stdout should contain "- minimal"

The same is also true if you’re running bash, so make sure that you don’t test anything you set up in your .bashrc without sourcing it.

It looks obvious, but without knowing what was going on I wasted an hour trying to understand why the command was not available in the test.

Also, while the statement ~/.zshrc seems okay, to be able to pick it from the user’s dir, and not a test temporary directory, you will have to specify it in the env.rb file via

Aruba.configure do |config|
    config.home_directory = '/home/tester'
end

Conclusion

While there are some rabbit holes on the way, I liked the ease of creating feature tests for my bash script with Aruba. The predefined methods make it really easy to get started, and the Behavior flow works really well with bash scripts. How readable and maintainable they will be in the long run? Time will tell. But for now, I found them to be a valuable addition to my toolset, and I’m actively thinking to propose using them on a project in the future as well.