fedux.org – Handle bugs in your Cucumber Test-Suite

- - Handle bugs in your Cucumber Test-Suite
- 12.08.2015 ~5 min
  - aruba cucumber ruby tdd

Having a suite testing a lot of features is a good thing. But if your project supports a lot of different Rubies – MRI-Ruby, JRuby etc. – you might face platform dependent bugs. Read on, if you like to know how we worked around some platform dependent bugs in aruba to keep the build status “green” on all support Rubies.

Introduction

Cucumber is quite a cool tool to build a test suite using a more or less natural language. I use it in some of my projects together with aruba – an extension for some popular testing frameworks to write tests for commandline applications. It supports different versions of MRI-Ruby and JRuby.

Maybe you also know the situation: You are the developer of an application/a library supporting different platforms like JRuby and/or MRI-Ruby on Linux, Windows and/or Mac OS X. You might have gotten into a pretty pickle adding a new feature whose tests only fails if you run them with JRuby.

If the reason for this failure is a bug in the platform – like jruby/ruby#3126 – the only thing you can do some times, is to skip the test until the bug is fixed. This article describes how you can skip tests in your cucumber test suite both for cucumber 1.x and cucumber 2.x starting with the more easy to use version 2 (for this use case).

Preparations

First, you need to add cucumber and aruba to your Gemfile:

source 'https://rubygems.org'

gem 'cucumber', '~> 2.0.2'
gem 'aruba', '~> 0.9.0.pre'

And run this command afterwards to install the gems.

bundle install

Next, create a file named “features/support/env.rb”. This file should contain the following code.

require 'aruba/cucumber'

After this, please create a feature-test named “features/convert_text_file.feature” with the following content:

Feature: Convert Text file

  Scenario: Existing input file
    Given an empty file named "test.in"
    When I successfully run `mycommand.sh`
    Then the file "test.db" should exist

  Scenario: Un-existing input file
    Given a file named "test.in" does not exist
    When I run `mycommand.sh`
    Then the exit status should be 1

Additionally create the executable bin/mycommand.sh with the following content:

#!/usr/bin/env bash
[[ -f test.in ]] && touch test.db

And make it an executable.

chmod +x bin/mycommand.sh

Skipping tests

Cucumber 2.x

You can count yourself lucky if you can use cucumber 2. This versions has a method to skip a test - aka “scenario”. You can use this method to skip it under certain conditions.

If you need to skip the scenario “Un-existing input file” on Windows, Mac OS X and Unix attach the following tags to it: @ignore-platform-windows, @ignore-platform-mac and @ignore-platform-unix – normally this would be a single tag skipping only one platform.

Feature: Convert Text file

  Scenario: Existing input file
    Given an empty file named "test.in"
    When I successfully run `mycommand.sh`
    Then the file "test.db" should exist

  @ignore-platform-windows
  @ignore-platform-mac
  @ignore-platform-unix
  Scenario: Un-existing input file
    Given a file named "test.in" does not exist
    When I run `mycommand.sh`
    Then the exit status should be 1

Next, you need to create Before-hooks for your supported platforms within features/step_definitions/hooks.rb – the filename is arbitrary chosen – with the following content.

Before '@ignore-platform-windows' do
  skip_this_scenario
end

Before '@ignore-platform-unix' do
  skip_this_scenario
end

Before '@ignore-platform-mac' do
  skip_this_scenario
end

Then run cucumber and you will notice, that the tagged scenario is skipped.

bundle exec cucumber
# => Feature: Convert Text file
# => 
# =>   Scenario: Existing input file           
# =>     Given an empty file named "test.in"   
# =>     When I successfully run `mycommand.sh`
# =>     Then the file "test.db" should exist  
# => 
# =>   @ignore-platform-windows @ignore-platform-mac @ignore-platform-unix
# =>   Scenario: Un-existing input file             
# =>     Given a file named "test.in" does not exist
# =>     When I run `mycommand.sh`                  
# =>     Then the exit status should be 1           
# => 
# => 2 scenarios (1 skipped, 1 passed)
# => 6 steps (3 skipped, 3 passed)
# => 0m0.122s

Cucumber 1.x

Unfortunately this only works for cucumber 2.x. For cucumber 1.x you need to use a hack. Both versions of cucumber support the use of the cucumber.yml – the cucumber-configuration file. You can safely use ERB within this file.

To activate cucumber 1.x, you need to modify your Gemfile. Please replace the version of cucumber with 1.3.20 and run bundle install after that.

source 'https://rubygems.org'

gem 'cucumber', '~> 1.3.20' # <- Change version here
gem 'aruba', '~> 0.9.0.pre'

Additionally there’s a library called “FFI”. You typically use this library for programmatically loading dynamic libraries. But this library also comes with some methods to detect the platform you’re running your code on – e.g. Windows, Linux, Mac OS X. Make sure, that you add this library to your Gemfile to make it available for your test-setup.

gem 'ffi', '~> 1.9.10'

Now run this command.

bundle install

Combining both things – cucumber.yml with ffi – you can build a solution for cucumber 1.x which gives you the same result like the #skip_this_scenario-method known from cucumber 2.x.

<%
require 'ffi'

ignore_opts = []
ignore_opts <<  "--tags ~@ignore-platform-java"    if RUBY_PLATFORM.include? 'java'
ignore_opts <<  "--tags ~@ignore-platform-mri"     if !RUBY_PLATFORM.include? 'java'
ignore_opts <<  "--tags ~@ignore-platform-windows" if FFI::Platform.windows?
ignore_opts <<  "--tags ~@ignore-platform-unix"    if FFI::Platform.unix?
ignore_opts <<  "--tags ~@ignore-platform-mac"     if FFI::Platform.mac?
ignore_opts = ignore_opts.join(' ')
%>
default: --tags ~@wip <%= ignore_opts %>
wip: --wip --tags @wip:3 <%= ignore_opts %>

After that run cucumber. You will notice, that only one scenario is run by cucumber. The other one is simply ignored.

bundle exec cucumber
# => Using the default profile...
# => Feature: Convert Text file
# => 
# =>   Scenario: Existing input file
# =>     Given an empty file named "test.in"
# =>     When I successfully run `mycommand.sh`
# =>     Then the file "test.db" should exist
# => 
# => 1 scenario (1 passed)
# => 3 steps (3 passed)
# => 0m0.111s

Conclusion

If you need to skip scenarios under certain conditions, you can use the #skip_this_scenario-method within a Before-hook for cucumber 2.x. If you need to use cucumber 1.x, you can “build” your own solution by combining ERB within the cucumber.yml together with ffi.

Thanks for reading.

Discussion

If you found a mistake in this article or would like to contribute some content to it, please file an issue in this Git Repository

Disclaimer

The contents of this article are put together to the best of the authors' knowledge, but it cannot be guaranteed that it's always accurate in any environment. It is up to the reader to make sure that all information found in this article, does not do any damage to the reader's working environment or wherever this information is applied to. In no event shall the authors or copyright holders be liable for any claim, damages or other liability, arising from, out of or in connection with this article. Please also note the information given on the Licences' page.