At FreeAgent, we run 45,000 tests on every code change to make sure that our rails monolith continues to work as expected. These include unit, integration, and acceptance tests. Recently, we switched from Capybara-webkit to Headless Chrome with Selenium for running JavaScript and acceptance tests.

Why did we switch?

Capybara-webkit has now been deprecated and uses an old version of webkit engine, so we had to look for alternatives. We preferred Headless Chrome over Chrome because it provides a real browser context without the memory overhead of running Chrome. JavaScript and feature tests now have the same execution context as end users of our site, so we have more accurate feedback from tests.

Chromium vs Chrome Browser

We use the open source Chromium browser on our Continuous Integration (CI) servers. Chromium is lightweight and has a smaller memory footprint than Chrome.

Steps we took to prepare CI

Switching OS

Previously, our Jenkins CI setup ran on CentOS. Chromium fails to install on CentOS 6, so we switched to Ubuntu. Now, we use the Jenkins EC2 plugin to spin up Ubuntu spot instances on AWS. This also took us one step closer towards our goal to move to AWS.

Webdrivers gem

ChromeDriver is an open source tool used by Selenium to control Chrome. It provides capabilities for navigating to web pages, user input, JavaScript execution, and many more. ChromeDriver is a standalone server which implements WebDriver’s wire protocol for Chromium.

To keep Chromium and Chromedriver versions in sync, we introduced the webdrivers gem into our setup. It automatically pulls the appropriate driver version when the test runs on a machine for the first time. Webdrivers also provides a rake task in case you run tests in parallel and don’t want each process to spend time upgrading the driver.

Migrating our Jasmine test suite

We use Karma to run Jasmine tests in headless mode. Karma works with any of the most popular testing frameworks (Jasmine, Mocha, QUnit).

Migrating Acceptance tests

We’ve hooked-up Capybara with the selenium-webdriver gem to drive our tests in Headless Chrome. Previously, we used capybara-webkit but that only drives the QtWebKit browser which is now deprecated. Whereas, selenium-webdriver opens up possibilities for testing on a variety of browsers.

On the downside, a whole host of tests started failing when we made this switch. Here’s a list of gotchas we encountered:

1. Many text assertions changed because we get more real text data now with Chrome. We corrected text to match output from Chrome. For example: Webkit ignores non-breaking spaces but chrome returns them.
2. Capybara-webkit provides the have_http_status and request.headers methods. Selenium does not provide any request/response inspection methods. We added a middleware to intercept requests to allow us to inject headers. 
3. The methods to set or delete cookies are different. Also selenium is strict, you cannot set cookies until you visit a page in the domain you intend to scope your cookies to.

# In capybara-webkit
page.driver.clear_cookies
page.driver.set_cookie(
  "fa_user_session_key=#{sign_cookie(user_session.key)}; path=/; domain=#{user.account.subdomain}.lvh.me"
)
visit(login_path)

# In selenium
visit '/'
page.driver.browser.manage.delete_all_cookies
page.driver.browser.manage.add_cookie(
  name: 'fa_user_session_key',
  value: sign_cookie(user_session.key),
  path: '/',
)
visit(login_path)  

4. Selenium does not have a method to perform downloads. We’ve added a download helper to fetch downloaded files. You can fetch the last downloaded file using last_download! method in feature spec. We run tests in parallel worker processes, so we maintained a separate download directory for each worker to avoid races.
5. element.send_keys only works on focusable elements, e.g. sending an escape keypress to a div (not a focusable element) closes a modal window in WebKit but we had to send keys to a focusable element in Selenium.

# In capybara-webkit
find(".fe-Modal[data-modal-name='practice_dashboard_sample']").send_keys(:escape)

# In selenium
within ".fe-Modal[data-modal-name='practice_dashboard_sample']" do 
  find(".fe-Modal-closeButton").send_keys(:escape) 
end

6. WebKit handles JavaScript confirmation dialog boxes so your test doesn’t have to. In Selenium, a click action needs to be wrapped in a accept_confirm , or dismiss_confirm block.

# In capybara-webkit
click_link("Delete Yodlee")

# In selenium
accept_confirm do
  click_link("Delete Yodlee")
end

7. Selenium cannot find empty elements if Capybara.ignore_hidden_elements is set to true. Selenium can not find check-boxes or empty fields in this case. We’ve fixed tests by using visible:any in find methods or by setting ignore_hidden_elements=false.
8. Selenium does not support the .trigger method. You will need to call or simulate the event.

# In capybara-webkit
within "[data-target$='practice-select.results']" do
  find("option[value='#{practice.id}']").trigger(:mouseover)
end

# In selenium
within "[data-target$='manager-select.results']" do
  find("option[value='#{manager.id}']").hover
end

Noteworthy Selenium driver configs 

Non-headless mode

We’ve also enabled non-headless mode in capybara’s selenium driver config, which allows us to debug tests in a browser window by setting an environment variable

Logging errors from the driver and browser

Selenium webdriver takes in loggingPrefs to capture browser and driver logs.

driver = Capybara::Selenium::Driver.new(
  app,
  browser: :chrome,
  options: options,
  driver_path: chrome_driver_path,
  desired_capabilities: Selenium::WebDriver::Remote::Capabilities.chrome(
    loggingPrefs: {
      browser: ENV['CHROME_LOGGING'] || 'SEVERE', # Capture JavaScript errors
      driver: ENV['CHROMEDRIVER_LOGGING'] || 'SEVERE', # Capture WebDriver errors
      client: 'SEVERE',
      server: 'SEVERE'
    }
  )
)

Which are then flushed-out to a file in a test’s after execution callback.

config.after :each, type: :feature, js: true do
  browser_errors = page.driver.browser.manage.logs.get(:browser)
  driver_errors = page.driver.browser.manage.logs.get(:driver)

  browser_log.puts browser_errors
  browser_log.puts driver_errors

  browser_log.puts
end

Hope this information comes in handy if you’re looking to migrate.

Happy testing!