Accessibility Testing with aXe-core and WebdriverJS

Share on FacebookShare on LinkedInShare on Twitter

aXe LogoIn this tutorial, we’ll set up automated accessibility testing for a JavaScript project using axe-webdriverjs, a Node.js module making axe-core easy to use with Selenium Webdriver. This tutorial uses the Jasmine testing framework–which could be swapped out for Mocha or another testing solution–but is otherwise framework-agnostic and can be integrated into any project.

Why automated accessibility testing?

Automated testing can free up 30% of your manual testing effort, making it a valuable tool for software development. By scripting a browser to programmatically open web pages and perform user tasks, you can validate features without having to open each page yourself. By integrating axe-core into your tests, you can add coverage for accessibility without having to become an expert. In addition, you can benchmark accessibility support and catch regressions in your builds, preventing broken code from going to production.

Requirements and Setup

To get started, check out the demo for this tutorial from Github. You will need Node.js and npm (Node package manager) installed. Selenium Webdriver can be used to test multiple browsers, including Firefox, Chrome, Safari, Edge and Opera. This demo uses Chrome, for which you’ll need ChromeDriver installed somewhere on your $PATH. [How to install Chromedriver on OSX]

Setting up a test project with axe-webdriverjs is very straightforward using npm. There are a few dependencies in our demo’s package.json file, including selenium-webdriver, jasmine and axe-webdriverjs, which we can install seamlessly using npm install. These modules, along with the axe-core module (installed automatically), give us everything we need to write browser integration tests for accessibility.

To recap, you will need:

  • Git
  • Node.js
  • npm
  • Chrome
  • ChromeDriver

Github demo: https://github.com/marcysutton/axe-webdriverjs-demo

Install dependencies:

npm install

That’s it for setup!

Working with Tests

To programmatically test a web page, you can direct Selenium Webdriver to a URL, provide any necessary setup details, and assert that it has no accessibility errors in that state. If your environment already has integration tests, it’s easy to work in aXe since you’ve already got an infrastructure for loading URLs and scripting user scenarios.

Here’s an example of testing a page running on a local web server:

var selenium = require('selenium-webdriver'),
    AxeBuilder = require('axe-webdriverjs');

describe('Accessibility', function() {
  var driver;

  beforeEach(function(done) {
      driver = new selenium.Builder()
          .forBrowser('chrome')
          .build();

      driver.get('http://localhost:4000/')
          .then(function () {
              done();
          });
  });

  // Close website after each test is run (so it is opened fresh each time)
  afterEach(function(done) {
      driver.quit().then(function () {
          done();
      });
  });

  it('should change state with the keyboard', function() {
    var selector = 'span[role="radio"][aria-labelledby="radiogroup-0-label-0"]';

    driver.findElement(selenium.By.css(selector))
      .then(function (element) {
          element.sendKeys(Key.SPACE);
          return element;
      })
      .then(function (element) {
          return element.getAttribute('aria-checked')
      })
      .then(function (attr) {
          expect(attr).toEqual('true');
      });
  });

  it('should analyze the page with aXe', function (done) {
     AxeBuilder(driver)
       .analyze(function(results) {
            console.log('Accessibility Violations: ', results.violations.length);
            if (results.violations.length > 0) {
                console.log(results.violations);
            }
            expect(results.violations.length).toBe(0);
            done();
        })
  });
});

In the test file above, we pull in our dependencies of selenium-webdriver and axe-webdriverjs, allowing us to call functions built into those libraries. An existing integration test checks keyboard and ARIA support of a custom radio button. Using that test’s Webdriver setup along with axe-webdriverjs, a second test calls AxeBuilder to analyze the page with axe-core. Once AxeBuilder has returned its audit results through a JavaScript promise, we can output them to the console.

Harnessing the value of aXe

Integrating aXe into a larger test environment with many developers means you’ll have to do more with the results–simply printing out “expected 2 to equal 0” for accessibility violations won’t provide much insight to a developer who unknowingly introduced regressions. There are a few options for dealing with aXe results data beyond the basics: the WorldSpace Attest Reporting API, which can be easily integrated into your existing test infrastructure; or, building your own functionality to log and then report on those results across the entire test suite once completed.

A useful reporter would digest accessibility test results and display the information necessary to quickly diagnose failures, such as the test name, impact level and relevant markup. A utility for logging the results object during testing may come in handy as well, to provide developers with more information while avoiding repetitive boilerplate code in every test file.

Gotchas

Timeouts

If you are integrating aXe into existing functional tests, you will already be waiting for pages to load and checking for specific UI widgets to appear before testing for assertions. However, if you don’t already have those mechanisms in place, you may run into a few tricky spots. First, there are significant timing differences between local and remote URLs: without any kind of network throttling to simulate real world latency and download speeds, locally running URLs will return much faster than remote ones. If you try testing remote URLs (i.e. http://www.deque.com, versus http://localhost:4000, which would run on only your machine), you’ll see timeout errors left and right. Fortunately, this can be fixed by increasing Jasmine and Webdriver’s timeout intervals.

To increase Jasmine’s default timeout interval, increase it anywhere in your test file:

jasmine.DEFAULT_TIMEOUT_INTERVAL = 60000;

Once you have instantiated Selenium Webdriver, you can also increase its script timeout limit:

browser.manage().timeouts().setScriptTimeout(60000);

Promises

Another big gotcha when working with WebdriverJS is the issue of asynchronous commands in a synchronous environment. To deal with this, you must be comfortable using JavaScript Promises. Every command takes time to execute and you have to wait for confirmation before you can utilize the data returned. You’ll learn to love the return statement so you can keep your continuous stream of data flowing. For more on learning how to use Promises in WebdriverJS, check out my blog post on writing Protractor plugins.

Conclusion

It’s clear that automated testing provides value in software development. By using tools like axe-core, you can get a significant portion of your accessibility testing done without any manual testing. You can start testing for things like color contrast, form labels, or ARIA usage without having to become an expert on implementation algorithms. Further, if accessibility support regresses in your app, you’ll know right away because you’ve already benchmarked the current level of support.

Online course provider edX was able to cut down their manual accessibility testing significantly by integrating aXe into their testing framework. From Mark Sadecki:

“Because axe-core is designed to eliminate false positives, edX can confidently use the ruleset in our acceptance testing framework, keeping some of the most egregious accessibility failures out of our production environment.”

Let automated tests identify the low-hanging fruit with regards to accessible software quality in your project and take a big step towards digital equality!

Marcy Sutton is a Senior Front-End Engineer at Deque, where she works on the aXe-core team focusing on accessibility test integrations. Marcy is passionate about making the web accessible for everyone. She’s an active contributor to the Angular framework, where she regularly brings her accessibility expertise to the table. Her blog, Accessibility Wins, highlights accessible user interfaces and tools, contributing a positive voice to the web development space. When away from the keyboard, Marcy can usually be found riding a bicycle or a snowboard.

About 

Marcy Sutton is a Senior Front-End Engineer at Deque. She loves applying her passion for accessibility on the axe-core project, speaking at conferences around the world and producing accessibility screencasts for Egghead.io. Her blog, Accessibility Wins, highlights accessible user interfaces and tools, contributing a positive voice to the web development space. When away from the keyboard, Marcy can usually be found riding a bicycle or a snowboard.

1 comment

  • Joshua Muheim Permalink

    Thanks for this article. I’m also deeply convinced that a high quality project needs to be tested using an automated test suite, and developments have come a long way in the last few years in allowing testing even quite complex JavaScript widgets. Still, it’s not really possible to write tests on the accessibility API level, but at least the behaviour of JavaScript itself can be tested, which works out for most needs in this respect.

    In my case, I’m writing web apps using Ruby on Rails, and with the integration of RSpec and Capybara, I can write very serious feature specs that also execute JavaScript specs automatically (either by using Capybara’s JavaScript support [together with e.g. PhantomJS], or even by adding a wrapper for “real” browser with e.g. Konacha). It’s a dream come true, and I’m so sorry for any developer who hasn’t realised the incredible power of an automated suite.

    Just one side note: as an accessibility related site, you should at least connect the labels of the reply form to their inputs…! In Rails, using the integrated form generator, this is done all automatically! I LOVE RAILS. :-)

Leave a Reply

You can use these HTML tags:

<a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>