Is it possible to use Catch2 for testing an MPI code?

I'm working with a fairly big MPI code. I started to include unit tests into the existing code base. However, as soon as the unit under test uses an MPI routine the test executable crashes with the error message "calling an MPI routine before MPI_Init"

  • What the best way around this?
  • Can I run tests with multiple MPI ranks?

Yes, it is possible.

As explained in https://github.com/catchorg/Catch2/issues/566, you will have to supply a custom main function.

#define CATCH_CONFIG_RUNNER
#include "catch.hpp"
#include <mpi.h>

int main( int argc, char* argv[] ) {
    MPI_Init(&argc, &argv);
    int result = Catch::Session().run( argc, argv );
    MPI_Finalize();
    return result;
}

To amplify your experience using Catch2 in combination with MPI, you might want to avoid redundant console output. That requires injecting some code into ConsoleReporter::testRunEnded of catch.hpp.

#include <mpi.h>
void ConsoleReporter::testRunEnded(TestRunStats const& _testRunStats) {
    int rank id = -1;
    MPI Comm rank(MPI COMM WORLD,&rank id);
    if(rank id != 0 && testRunStats.totals.testCases.allPassed())
        return;
    printTotalsDivider(_testRunStats.totals);
    printTotals(_testRunStats.totals);
    stream << std::endl;
    StreamingReporterBase::testRunEnded(_testRunStats);
}

Finally you might also want to execute your test cases with a different number of MPI ranks. I found the following to be an easy and well working solution:

SCENARIO("Sequential Testing", "[1rank]") {
    // Perform sequential tests here
}
SCENARIO("Parallel Testing", "[2ranks]") {
    // Perform parallel tests here
}

Then you can call the tagges scenarios individually with

mpiexec -1 ./application [1rank]
mpiexec -2 ./application [2rank]

use MPI in tests (aka MPI_Init in main) · Issue #566 · catchorg/Catch2, Describe the bug Catch2 test of libMesh based program intermittently to host and review code, manage projects, and build software together. In this course, C++ Unit Testing Fundamentals Using Catch2, you will learn how to write robust unit tests using Catch2, a simple-to-use yet flexible and powerful unit testing framework for C++. You will learn what makes Catch2 different from other xUnit frameworks, and how it can be used to write unit tests for your C++ code.


For anyone looking to remove all duplicated console output when running Catch2 distributed, here's a solution.

Find the definition of ConsoleReporter in catch.hpp (in v2.10.0, it's at line 15896). It will look something like:

ConsoleReporter::ConsoleReporter(ReporterConfig const& config)
    : StreamingReporterBase(config),
    m_tablePrinter(new TablePrinter(config.stream(),
        [&config]() -> std::vector<ColumnInfo> {
        if (config.fullConfig()->benchmarkNoAnalysis())
        {
            return{
                { "benchmark name", CATCH_CONFIG_CONSOLE_WIDTH - 43, ColumnInfo::Left },
                { "     samples", 14, ColumnInfo::Right },
                { "  iterations", 14, ColumnInfo::Right },
                { "        mean", 14, ColumnInfo::Right }
            };
        }
        else
        {
            return{
                { "benchmark name", CATCH_CONFIG_CONSOLE_WIDTH - 32, ColumnInfo::Left },
                { "samples      mean       std dev", 14, ColumnInfo::Right },
                { "iterations   low mean   low std dev", 14, ColumnInfo::Right },
                { "estimated    high mean  high std dev", 14, ColumnInfo::Right }
            };
        }
    }())) {}
ConsoleReporter::~ConsoleReporter() = default;

Though not shown here, the base-class StreamingReporterBase provides a stream attribute which we'll disable, by the failbit trick shown here.

Inside the final {} above (an empty constructor definition), insert:

// I solemnly swear that I am up to no good
int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);

// silence non-root nodes
if (rank != 0)
    stream.setstate(std::ios_base::failbit);  

You can see an example on this repo.

Catch2 test never completes when using MPI in libMesh-based , Compiling this code into an executable yields a unit test. Being able to use simple C++ comparison operators in the assertion macros However, calling MPI_Finalize at the end of a test will disable MPI for all the other tests. Catch2, Google Test and doctest Adapter for the VSCode - matepek/vscode-catch2-test-adapter


Apart from supplying a custom main function, the oft-encountered issue of duplicate output from multiple MPI processes can be circumvented by forcing each MPI process to dump their test reports into separate files, e.g. process p dumps it's test report to report_p.xml.

A quick and dirty approach would be to extend the vector of command line arguments argv with extra entries for the process dependent filename paired with a --out.

Source:

// mytests.cpp
#define CATCH_CONFIG_RUNNER

#include "catch.hpp"
#include <mpi.h>
#include <string>
#include <vector>

int 
main(int argc, char* argv[]) {
  MPI_Init(&argc, &argv);

  int mpi_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank);

  // make space for two extra arguments
  std::vector<const char*> new_argv(argc + 2);
  for (int i = 0; i < argc; i++) {
    new_argv[i] = argv[i];
  }

  // set "--out report_p.xml" as last two arguments
  auto filename = "report_" + std::to_string(mpi_rank) + ".xml";
  new_argv[argc] = "--out";
  new_argv[argc+1] = filename.data();

  int result = Catch::Session().run(new_argv.size(), new_argv.data());
  MPI_Finalize();
  return result;
}

Compile and Run:

mpic++ mytests.cpp -o mytests
mpirun -np 4 ./mytests --reporter junit

Expected Output Files

report_0.xml
report_1.xml
report_2.xml
report_3.xml

MPI processes writing to the same file using std::ostream run the risk of breaking XML formatting, crashing JUnit XML parsers and failing CI pipelines even if all the tests pass. Dumping the test report from each process into their own separate file avoids this issue. The separate files can be concatenated later on, if needed.

Even though the extra arguments in the above example have been placed at the end of the argument list, they do not override any previous --out key-value from the command line. Instead, the built-in CLI parser treats it as a duplication of argument keyword and throws an error. So a more inclusive implementation of the above approach would not extend the argument list but replace the argv pointer corresponding to the value of --out with the appended filename.

PS: pwaul and Anti Earth must have done some serious digging into the Catch source code to figure out the smart code injections, salutations to them!

Unit Testing C++ with Catch, This page is a list of tables of code-driven unit testing frameworks for various programming Name, xUnit, Fixtures, Group fixtures, Generators, MPI, OpenMP, Source, Remarks The goal of the JS Test Runner is to be able to test JavaScript code using an approach that test driven programmers "catchorg/​Catch2 · GitHub". By using our site, 1 Is it possible to use Catch2 for testing an MPI code? Oct 8 '19. 0 Cheating constness? Apr 14. Code Review; Magento;


List of unit testing frameworks, Usually this means a small and self-contained piece of code that uses Catch In order to be able to use this reporter an additional header must also be included. For +instance to run some tests using `MPI` and other sequentially, one can  Magnetic Particle Inspection (MPI) also sometimes called as Magnetic Test (MT) is a non-destructive test method for the detection of surface and sub-surface discontinuities in ferrous materials. The test method involves application of magnetic field externally or applying electric current through the material which in turn produces magnetic


Diff - platform/external/catch2 - Git at Google, Install and build Boost Testing Framework following these instructions. You can also use a live template for boost_with_libs, then adjust the template code to mimic the snippet above. In CLion, there are several ways to start a run/debug session for tests, one of which is using Unit Testing TutorialCatch2 support. Magnetic particle inspection (often abbreviated MT or MPI) is a nondestructive inspection method that provides detection of linear flaws located at or near the surface of ferromagnetic materials. It is viewed primarily as a surface examination method.


Boost.Test support - Help, This path will take you from the very basics of programming using C++ and will vault you After completing this course, you will be able to use high-quality efficient and well-tested C++ Refine your C++ skills even further by learning how to write unit tests for your C++ code. C++ Unit Testing Fundamentals Using Catch2. Animal research strictly controlled. Animal use in research, testing and teaching is strictly controlled under the Animal Welfare Act 1999. Any person or organisation using animals must follow an approved code of ethical conduct, which sets out the policies and procedures that must be followed by the organisation and its animal ethics committee.