One of the really nice features of VTK’s testing framework is the use of image-based regression tests. These allow developers to write tests that result in a final image, which can be recorded and compared to known baseline images in order to verify that the OpenGL rendering code is rendering the same (or similar) image on all platforms. If this fails then CDash will display the image the test produced, the baseline image it was compared to and an image difference. Any project that performs rendering or visualization needs tests like these in addition to unit tests if they want to assure visualization code continues to function as expected across a range of platforms.
We recently extracted the relevant code from the VTK testing framework to perform image based regressions in Avogadro 2, with the bulk of that code living in utilities/vtktesting/imageregressiontest.h. This is currently used in one of the tests, with plans to extend it to cover all major types of rendering, this can in seen in action intests/qtopengl/glwidgettest.cpp with the important lines that take the snapshot/do the image comparison being:
// Grab the frame buffer of the GLWidget and save it to a QImage.
QImage image = widget.grabFrameBuffer(false);
// Set up the image regression test.
Avogadro::VtkTesting::ImageRegressionTest test(argc, argv);
// Do the image threshold test, printing output to the std::cout.
return test.imageThresholdTest(image, std::cout);
The CMake code that feeds in the command line arguments, and ensures the test runs correctly is in tests/qtopengl/CMakeLists.txt, and largely involves passing in paths to the baseline directory, a temporary directory and the test name (using the standard CMake generated test driver).
add_test(NAME
"QtOpenGL-${test}" COMMAND AvogadroQtOpenGLTests
"${testname}test" "--baseline" "${AVOGADRO_DATA_ROOT}/baselines/avogadro/qtopengl" "--temporary" "${PROJECT_BINARY_DIR}/Testing/Temporary")
The above is the baseline image, that is stored in a known location and compared with the image produced by the test (shown below).
If the images don't match a difference image is produced and uploaded (shown below). In this case you can see that an extra sphere was rendered, and this can clearly be seen in the difference image. There is also a numerical difference returned by the test, which is a measure of how much the images differ. The tolerance can be tweaked depending on the test to allow some minor pixel differences, although care must be taken not to raise the number too high.
We have not implemented it in Avogadro 2 yet, but VTK can use multiple baselines and returns the smallest image difference. This allows for OS/GPU specific baselines to be uploaded where necessary as an alternative to increasing the tolerance. Using special tags returned by the tests in the standard output will prompt the ctest command to upload the image files when necessary (in the case the baseline image cannot be found, or the image comparison fails).