Testing Client-side JavaScript with Karma

This post originally appeared on strongloop.com. StrongLoop provides tools and services for companies developing APIs in Node. Learn more...

I have often quipped that if I were stuck on a desert island with only one npm module, I'd choose karma. The one place where you can't avoid using JavaScript is in the browser. Even Dart needs to be transpiled to JavaScript to work on browsers other than Dartium. And code that can't be tested should not be written. Karma is a widely-adopted command-line tool for testing JavaScript code in real browsers. It has myriad plugins that enable you to write tests using virtually any framework (mocha, jasmine, ngScenario, etc.) and run them against a local browser or in Sauce's Selenium cloud.

In this article, you'll see how to use karma to test standalone JavaScript locally, to test DOM interactions locally, and to run tests on Sauce Labs. The code for all these examples is available on my karma-demo GitHub repo.

Your First Karma Test

Like gulp or metalsmith, karma is organized as a lightweight core surrounded by a loose confederation of plugins. Although you will interact with the karma core executable, you will typically need to install numerous plugins to do anything useful.

Suppose you have a trivial browser-side JavaScript file like below. This code is lib/sample.js in the karma-demo GitHub repo

function theAnswer() {
  return 42;
}

Testing this in NodeJS would be fairly simple. But, what if you wanted to test this in a real browser, like Chrome? Let's say you have the following test file.

describe('sample', function() {
  it('returns 42', function() {
    assert.equal(theAnswer(), 42);
  });
});

To run this test in Chrome, set up your package.json with karma and 3 plugins. As you might expect, karma-mocha is a karma plugin that enables you to use the mocha test framework, and karma-chrome-launcher enables karma to launch Chrome locally. The karma-chai package is a wrapper around the chai assertion framework, since neither mocha nor karma includes an assertion framework by default.

{
  "devDependencies": {
    "karma": "0.12.31",
    "karma-chai": "0.1.0",
    "karma-chrome-launcher": "0.1.4",
    "karma-mocha": "0.1.10"
  }
}

Now that you have karma installed, you need a karma config file. You can run ./node_modules/karma/bin/karma init to have karma initialize one for you, but the karma config file you're going to need is shown below.

module.exports = function(config) {
  config.set({
    // Use cwd as base path
    basePath: '',

    // Include mocha test framework and chai assertions
    frameworks: ['mocha', 'chai'],

    // Include the test and source files
    files: ['../lib/*.js', './sample.test.js'],

    // Use built-in 'progress' reporter
    reporters: ['progress'],

    // Boilerplate
    urlRoot : '/__karma__/',
    port: 8080,
    runnerPort: 9100,
    colors: true,
    logLevel: config.LOG_INFO,

    /* Karma can watch the file system for changes and
     * automatically re-run tests. Making karma do it
     * is more efficient than using gulp because karma
     * can re-use the same browser process. Set this to
     * true and `singleRun` to false to run tests
     * continuously */
    autoWatch: false,

    // Set the browser to run
    browsers: ['Chrome'],

    // See autoWatch
    singleRun: true,

    // Consider browser as dead if no response for 5 sec
    browserNoActivityTimeout: 5000
  });
};

One key feature of the above config file is that karma configs are NodeJS JavaScript, not just JSON or YAML. You can do file I/O, access environment variables, and require() npm modules in your karma configs.

Once you have this config file, you can run karma using the ./node_modules/karma/bin/karma start test/karma.mocha.conf.js command. You should see a Chrome browser pop up and output similar to what you see below in the shell:

$ ./node_modules/karma/bin/karma start ./test/karma.mocha.conf.js 
WARN [karma]: Port 8080 in use
INFO [karma]: Karma v0.12.31 server started at http://localhost:8081/__karma__/
INFO [launcher]: Starting browser Chrome
INFO [Chrome 42.0.2311 (Mac OS X 10.10.3)]: Connected on socket NE7VtfIGgPAVbIB8nBIA with id 416059
Chrome 42.0.2311 (Mac OS X 10.10.3): Executed 1 of 1 SUCCESS (0 secs / 0.001 secChrome 42.0.2311 (Mac OS X 10.10.3): Executed 1 of 1 SUCCESS (0.012 secs / 0.001 secs)

Testing DOM Interactions with ngScenario

Karma isn't just useful for testing isolated JavaScript. With ngScenario and the karma-ng-scenario plugin you can test DOM interactions against your local server in a real browser. In this particular example, you will test interactions with a trivial page. However, ngScenario, as the name implies, was developed by the AngularJS team to test AngularJS applications, so you can use karma to test single page apps too.

Suppose you want to test the following page, which uses the sample.js file from the previous section to display the number "42" on the page.

<html>
  <head>
    <script type="text/javascript" src="/sample.js"></script>
    <script type="text/javascript">
      function onLoad() {
        document.getElementById('content').innerHTML = theAnswer();
      }
    </script>
  </head>
  <body onload="onLoad()">
    <div id="content"></div>
  </body>
</html>

In order to serve this page and the sample.js file, you need an HTTP server. You can put together a simple one using express as show below.

var express = require('express');
var app = express();

app.use(express.static('./lib'));

app.listen(3000);
console.log('Server listening on 3000');

Navigate to http://localhost:3000/sample.html and you should see the number "42". Now, let's set up karma to test this.

In order to use ngScenario with karma, you need to install the karma-ng-scenario plugin.

{
  "dependencies": {
    "express": "4.12.3"
  },
  "devDependencies": {
    "karma": "0.12.31",
    "karma-chrome-launcher": "0.1.4",
    "karma-ng-scenario": "0.1.0"
  }
}

Once you have the karma-ng-scenario plugin, you will need to create a slightly modified karma config file. The git diff between this file and the previous section's config file karma.mocha.conf.js is shown below.

 module.exports = function(config) {
   config.set({
     basePath: '',

-    frameworks: ['mocha', 'chai'],
+    frameworks: ['ng-scenario'],

-    files: ['../lib/*.js', './sample.test.js'],
+    files: ['./scenario.test.js'],

     reporters: ['progress'],

     urlRoot : '/__karma__/',

+    proxies : {
+      '/': 'http://localhost:3000'
+    },
+
     port: 8080,
     runnerPort: 9100,
     colors: true,

     logLevel: config.LOG_INFO,

-    autoWatch: false,
+    autoWatch: true,

     browsers: ['Chrome'],

-    singleRun: true,
+    singleRun: false,

-    // 5 sec
-    browserNoActivityTimeout: 5000
+    // 45 sec
+    browserNoActivityTimeout: 45000
   });
 };

That's all you need! Now, start your web server with node index.js (your web server needs to be running for ngScenario to work) and run karma start to execute your tests.

$ ./node_modules/karma/bin/karma start test/karma.ng-scenario.conf.js 
WARN [proxy]: proxy "http://localhost:3000" normalized to "http://localhost:3000/"
WARN [karma]: Port 8080 in use
INFO [karma]: Karma v0.12.31 server started at http://localhost:8081/__karma__/
INFO [launcher]: Starting browser Chrome
INFO [Chrome 42.0.2311 (Mac OS X 10.10.3)]: Connected on socket pyFsC9jnnhcbkTXT5y3t with id 7382597
Chrome 42.0.2311 (Mac OS X 10.10.3): Executed 1 of 1 SUCCESS (0 secs / 0.137 secChrome 42.0.2311 (Mac OS X 10.10.3): Executed 1 of 1 SUCCESS (0.152 secs / 0.137 secs)

The karma.ng-scenario.conf.js file is configured for karma to automatically re-run tests if any of the loaded files change. Suppose you changed the test file scenario.test.js to break:

describe('sample.html', function() {
  it('displays 42', function() {
    browser().navigateTo('/sample.html');
    // This assertion will fail
    expect(element('#content').text()).toBe('43');
  });
});

Karma will automatically re-run tests and show you the following output.

INFO [watcher]: Changed file "/Users/vkarpov/Workspace/karma-demo/test/scenario.test.js".
Chrome 42.0.2311 (Mac OS X 10.10.3) sample.html displays 42 FAILED
  expect element '#content' text toBe "43"
  /Users/vkarpov/Workspace/karma-demo/test/scenario.test.js:4:32: expected "43" but was "42"
Chrome 42.0.2311 (Mac OS X 10.10.3): Executed 1 of 1 (1 FAILED) (0 secs / 0.121 Chrome 42.0.2311 (Mac OS X 10.10.3): Executed 1 of 1 (1 FAILED) ERROR (0.136 secs / 0.121 secs)

Note the AngularJS team considers ngScenario to be deprecated in favor of protractor. However, I still think ngScenario and karma have a key niche to fill in terms of testing JavaScript DOM interactions against a local server as part of TDD workflow. If you're interested in working with me to re-imagine how ngScenario should work, shoot me an email at val [at] karpov [dot] io.

To the Cloud!

So far, you've only tested your code in Chrome. While testing in Chrome is better than not testing in the browser at all, it's far from a complete testing paradigm. Thankfully, there's Sauce Labs, a cloud Selenium service that you can think of as Amazon EC2 for browsers. You can tell Sauce to start Internet Explorer 6 running on Windows XP and give you full control over this browser for testing purposes. In order to get the tests in this section to work, you will have to sign up for an account with Sauce Labs to get an API key.

To integrate karma with Sauce, you're going to need the karma-sauce-launcher plugin.

{
  "dependencies": {
    "express": "4.12.3"
  },
  "devDependencies": {
    "karma": "0.12.31",
    "karma-chai": "0.1.0",
    "karma-chrome-launcher": "0.1.4",
    "karma-ng-scenario": "0.1.0",
    "karma-mocha": "0.1.10",
    "karma-sauce-launcher": "0.2.8"
  }
}

To run the ngScenario test from the previous section on IE9 and Safari 5 using Sauce, create another karma config file. The git diff between this new file and the previous section's config file is show below.

module.exports = function(config) {
+  var customLaunchers = {
+    sl_safari: {
+      base: 'SauceLabs',
+      browserName: 'safari',
+      platform: 'OS X 10.6',
+      version: '5'
+    },
+    sl_ie_9: {
+      base: 'SauceLabs',
+      browserName: 'internet explorer',
+      platform: 'Windows 7',
+      version: '9'
+    }
+  };
+
   config.set({
     basePath: '',

     frameworks: ['ng-scenario'],

     files: ['./scenario.test.js'],

-    reporters: ['progress'],
+    reporters: ['saucelabs'],

     urlRoot : '/__karma__/',

     proxies : {
       '/': 'http://localhost:3000'
     },

     port: 8080,
     runnerPort: 9100,
     colors: true,

     logLevel: config.LOG_INFO,

-    autoWatch: true,
+    autoWatch: false,
+
+    // Use these custom launchers for starting browsers on Sauce
+    customLaunchers: customLaunchers,

-    browsers: ['Chrome'],
+    // start these browsers
+    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
+    browsers: Object.keys(customLaunchers),

-    singleRun: false,
+    singleRun: true,

     // 45 sec
-    browserNoActivityTimeout: 45000
+    browserNoActivityTimeout: 45000,
+
+    sauceLabs: {
+      testName: 'Web App E2E Tests'
+    },
   });
 };

This is all you need to do. Karma handles setting up an SSH tunnel for you, so the Sauce browser can make real requests to your local server.

$ env SAUCE_USERNAME=<username here> env SAUCE_ACCESS_KEY=<API key here> ./node_modules/karma/bin/karma start test/karma.sauce.conf.js 
WARN [proxy]: proxy "http://localhost:3000" normalized to "http://localhost:3000/"
WARN [karma]: Port 8080 in use
INFO [karma]: Karma v0.12.31 server started at http://localhost:8081/__karma__/
INFO [launcher]: Starting browser safari 5 (OS X 10.6) on SauceLabs
INFO [launcher]: Starting browser internet explorer 9 (Windows 7) on SauceLabs
INFO [launcher.sauce]: internet explorer 9 (Windows 7) session at https://saucelabs.com/tests/d6d4c7a5eebe493d9e98bc3559106e54
INFO [launcher.sauce]: safari 5 (OS X 10.6) session at https://saucelabs.com/tests/49dc351d56664ecb92a854b37158c1a8
INFO [IE 9.0.0 (Windows 7)]: Connected on socket vGbX2IuqZHZEP6rj9Ltx with id 56376293
INFO [Safari 5.1.9 (Mac OS X 10.6.8)]: Connected on socket K7Nfu1zw5yhVOozy9Lty with id 49486594
TOTAL: 2 SUCCESS
INFO [launcher.sauce]: Shutting down Sauce Connect

Note the above example does not use autowatch. This is because, in my experience, end-to-end tests run on Sauce labs are too slow for the type of fast feedback I'd like from the 'test on save' paradigm. These heavy end-to-end tests are best run through a CI framework like Travis or Shippable.

Conclusion

Karma is a powerful and extensible tool for testing client-side JS, and very much deserves its spot on the npm home page. If there's one npm module I can't write JS without, it's karma. Karma enables you to test standalone JavaScript or DOM integrations, locally or in the Sauce Labs cloud. With karma, you have no excuse for half-answers like "I think so" when your boss asks you if the site works in IE8.

Like this article? Chapter 9 of my upcoming book, Professional AngularJS, is a detailed guide to testing AngularJS applications. It includes more detail about using karma and ngScenario to test AngularJS applications, as well as protractor.