plotly.js image testing

Test plotly.js with Plotly's image testing docker container.

Requirements:

• docker | installation guidelines
• docker-machine (for Mac and Windows users only) | installation guidelines
• docker-compose | installation guidelines

Step 0: Start the docker machine (Mac and Windows users only)

Boot up docker machine (named default):

docker-machine start default

If this is your first time, you'll need to create the machine instead:

docker-machine create --driver virtualbox default

Set up the docker environment for docker-compose:

eval $(docker-machine env default)

the above evaluates the output of docker-machine env default.

Step 1: Run the testing container

We use docker-compose to ease the creation/stopping/deletion of the testing docker container.

Inside your plotly.js directory, run

docker-compose up -d

In the docker-compose.yml file, latest is the latest Plotly Image-Server docker container version as listed on hub.docker.com and imagetest is the name of the docker container. The -d flag tells docker to start the containers in the background and leave them running.

Step 2: Run the image tests

The image testing docker container allows plotly.js developers to (A) run image comparison tests, (B) run image export tests and (C) generate baseline images.

IMPORTANT: the image tests scripts do not bundle the source files before running the image tests. We recommend running npm run watch or npm start in a separate tab to ensure that the most up-to-date code is used.

A: Run image comparison tests

Image comparison tests take in plotly.js mock json files (found in test/image/mocks), generate test png images (saved in build/test_images/ - which is git-ignored) and compare them pixel-by-pixel to their corresponding baseline images (found in test/image/baselines) using GraphicsMagick.

To run the image comparison tests, in your plotly.js directory:

npm run test-image

which runs all image comparison tests in batch. If some tests fail, compare their outputs by booting up the test image viewer using npm run start-image_viewer.

As an alternative to running all image comparison tests at once, you can provide a glob as argument to target one or multiple test mocks found in test/image/mocks. For example,

# Run one test (e.g. the 'contour_nolines' test):
$ npm run test-image -- contour_nolines

# Run all gl3d image test in batch:
$ npm run test-image -- gl3d_*

Developers on weak hardware might encounter batch timeout issue. These are most common when generated WebGL-based graphs. In this case, running the image comparison tests in queue (i.e. with no concurrency) is recommended:

# Run all gl3d image test in queue:
$ npm run test-image -- gl3d_* --queue

B: Run image export tests

Image export tests check that image export works for formats other than png.

To run the image export tests, in your plotly.js directory:

npm run test-export

# or
npm run test-export -- <glob>

C: Generate or update existing baseline image

To generate a new baseline image, add a new mock file in test/image/mocks. Note that mock file needs to be a valid JSON and have both a "data" and a "layout" field. Then, in your plotly.js directory, run:

npm run baseline -- <name-of-mock>

which generates a baseline png image in test/image/baselines.

To update existing baseline image(s), run

npm run baseline -- <glob-of-mocks-to-update>

Step 3: Stop your testing container

Once done testing, inside your plotly.js directory, run

docker-compose stop

Mac and Windows user should also kill their docker-machine (named default) once done testing:

docker-machine kill default

Docker tricks

Get into docker container

docker exec -ti imagetest /bin/bash

List docker machines

docker-machine ls

List all images

docker images

List all containers

docker ps -a

whereas docker ps lists only the started containers.

Remove your testing container

Inside your plotly.js directory, run

docker-compose rm -f

Remove your docker machine

If named default:

docker-machine kill default

For more comprehensive information about docker, please refer to the docker docs.