PHPUnit Strict Mode - setUp() - Coverage

phpunit configuration file example
phpunit code coverage
phpunit exclude file from code coverage
phpunit generate xml
run phpunit
phpunit no tests executed
phpunit no code coverage driver is available
laravel phpunit

I currently started using the strict-Mode in PHPUnit, when I came across a Problem with the code-coverage:

If I use the setUp-method to create a new instance of a my class the __constructor-method is listed in the code-coverage as covered when i run the tests.

This is my test-setup:

phpunit.config.xml

<?xml version="1.0" encoding="UTF-8"?>
<phpunit
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/4.5/phpunit.xsd"
    bootstrap="../vendor/autoload.php"
    backupGlobals="false"
    backupStaticAttributes="false"
    colors="true"
    verbose="true"    
    beStrictAboutOutputDuringTests="true"
    beStrictAboutTestSize="true"
    beStrictAboutTestsThatDoNotTestAnything="true"
    beStrictAboutTodoAnnotatedTests="true"

    checkForUnintentionallyCoveredCode="true"
    processIsolation="false"
>
<testsuites>
    <testsuite name="FooTests">
        <directory suffix="Test.php">../tests</directory>
    </testsuite>
</testsuites>
<filter>
    <whitelist>
        <directory suffix=".php">../src</directory>
    </whitelist>
</filter>
<logging>
    <log type="coverage-html" target="coverage/" higlight="true" showUncoveredFiles="true"></log>    
</logging>

Foo.php

class Foo
{

    protected $_bar;

    public function __construct($bar)
    {
        $this->_bar=$bar;             //Line 10
    }                                 //Line 11

    public function getBar()
    {
        return $this->_bar;
    }

    public function getBar2()
    {
        return $this->_bar;
    }

}

and the Test: FooTest.php

class FooTest extends \PHPUnit_Framework_TestCase
{

    protected $_foo;

    protected function setUp()
    {
        $this->_foo=new Foo(10);
    }

    public function testGetBar()
    {
        $this->assertSame(10, $this->_foo->getBar());
    }

    /**
     * @covers Foo::getBar2
     */
    public function testGetBar2()
    {
        $this->assertSame(10, $this->_foo->getBar2());
    }

}

If I run the tests I get this result:

PHPUnit 4.5.0 by Sebastian Bergmann and contributors.

Configuration read from C:\xampp\htdocs\unittest\build\phpunit.config.xml

.R

Time: 88 ms, Memory: 3.50Mb

There was 1 risky test:
1) FooTest::testGetBar2
This test executed code that is not listed as code to be covered or used:
- C:\xampp\htdocs\unittest\src\Foo.php:10
- C:\xampp\htdocs\unittest\src\Foo.php:11

OK, but incomplete, skipped, or risky tests!
Tests: 2, Assertions: 2, Risky: 1.

Generating code coverage report in HTML format ... done

As soon as I specify the @covers on the test the problem appears.

Is this expected behavior?

Some things I tried:

  • Changing checkForUnintentionallyCoveredCode to false obviously works, but I want to use this feature...
  • Using processIsolation="true" works too. I don't know why?
  • Adding @covers or @uses to setUp() doesn't work
  • Adding @covers that the setUp() uses to the test does work, but the test doesn't actually cover the code. (If the tests get more complex, this seems to be a lot of writing...)
  • Different phpunit-Version: I tried this with 4.3 and 4.5 with same results
  • Different PHP-Setup: I tried this on a Win8 with XAMPP and a LinuxMint - same results

Is there a way to remove the setUp() code from the code-coverage and using @covers on tests with methods they actually test?

Edit: This also affects inheritance. So if Bar extends Foo, and passes arguments on to Foo::__construct that will be on the code-coverage too - which makes writing the @covers for the __construct a pain in the a**...

Additional info:

PHP 5.6.3 (cli) (built: Nov 12 2014 17:18:08)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2014 Zend Technologies
with Xdebug v2.2.5, Copyright (c) 2002-2014, by Derick Rethans

Since this question started to gain some momentum: Here is my kind-of-solution for the problem.

My unit-test (FooTest) of Foo will always use Foo, therefore I add @uses Foo to the class.

This is also important if protected/private functions are used by public functions, because else you have to add each and every protected/private function to a test, if the class uses the function internally. I even think it is wrong if you are doing unit-tests, because a unit-test must not care about how the class does "stuff", it should only assert that a specific input results in a specific output.

(Additionally: The constructor should only do assignments, nothing else.)

After adding @uses the error will disapear.

(You can add @covers Foo::_construct to the class to have code-coverage of your constructor.)

/**
 * @uses Foo 
 * (optional)@covers Foo::__construct
 */
class FooTest extends \PHPUnit_Framework_TestCase
{
    protected $_foo;

    protected function setUp()
    {
        $this->_foo=new Foo(10);
    }

    public function testGetBar()
    {
        $this->assertSame(10, $this->_foo->getBar());
    }

    /**
     * @covers Foo::getBar2
     */
    public function testGetBar2()
    {
        $this->assertSame(10, $this->_foo->getBar2());
    }
}

More friendly strict coverage configuration · Issue #2413 , As of now if you are using the @covers with --strict-coverage, you get warnings for So my suggestions is when running the tests in strict mode, \PHPUnit\​Framework\TestCase { public function test_foo() { $dummy = new  In addition to being helpful for persons reading the code, this annotation is useful in strict coverage mode where unintentionally covered code will cause a test to fail. See Unintentionally Covered Code for more information regarding strict coverage mode. Please note that this annotation requires a fully-qualified class name (FQCN).

You have specified strict coverage with checkForUnintentionallyCoveredCode="true". And since PHPUnit 4.0 PHPUnit has the following behaviour:

Dealing with unintentionally covered code

PHPUnit 4.0 can optionally be strict about unintentionally covered code (strict > coverage mode). When enabled, PHPUnit will fail a test that uses the @covers annotation and executes code that is not specified using a @covers annotation.

PHPUnit XML configuration file, A failure is a violated PHPUnit assertion such as a failing assertSame() call. for ignoring code coverage --no-coverage Ignore code coverage configuration tests that do not test anything --strict-coverage Be strict about @covers annotation​  --strict-coverage Be strict about unintentionally covered code. --disallow-test-output Be strict about output during tests. --enforce-time-limit Enforce time limit based on test size. --disallow-todo-tests Disallow @todo-annotated tests. --strict Run tests in strict mode (enables all of the above).

On PHPUnit >= 6.0 set beStrictAboutCoversAnnotation to false on phpunit.xml:

<phpunit
     // ....     
     beStrictAboutCoversAnnotation="false"
>
// ....

Also, you can run phpunit without --strict-coverage

More info on Risky Tests: Unintentionally Covered Code

3. The Command-Line Test Runner, Applications such as PHPUnit use this information at runtime to configure @​beforeClass */ public static function setUpSomeSharedFixtures() { // . is useful in strict coverage mode where unintentionally covered code will cause a test to fail. 2. Writing Tests for PHPUnit. Example 2.1 shows how we can write tests using PHPUnit that exercise PHP’s array operations. The example introduces the basic conventions and steps for writing tests with PHPUnit: The tests for a class Class go into a class ClassTest. ClassTest inherits (most of the time) from PHPUnit\Framework\TestCase.

I don't think the accepted answer is correct. It basically says that whatever is in this class is OK to be used. Which may or may not be what you want. Specifically if you want to make sure your testGetBar2() method does not use some other code you won't be able to.

What you can do is to ignore what your set up method does.

    /**
     * @codeCoverageIgnore
     */
    protected function setUp()
    {
        $this->_foo=new Foo(10);
    }

This way whatever you put in your setup will not count as tested code, but anything you specifically cover will be shown.

https://phpunit.de/manual/3.7/en/code-coverage-analysis.html#code-coverage-analysis.ignoring-code-blocks

2. Annotations, PHPUnit. The attributes of the <phpunit> element can be used to configure PHPUnit's core Code Coverage will only be recorded for tests that use the @​covers  PHPUnit is a programmer-oriented testing framework for PHP. It is an instance of the xUnit architecture for unit testing frameworks. Take the first steps Let me help you Get the eBook Say "Thank you!"

PHPUnit Manual – Appendix C. The XML Configuration File, Applications such as PHPUnit use this information at runtime to configure their @beforeClass */ public static function setUpSomeSharedFixtures() { // . especially useful in strict coverage mode where unintentionally covered code will cause  PHPUnit can optionally backup all global and super-global variables before each test and restore this backup after each test. This attribute configures this operation for all tests. This configuration can be overridden using the @backupGlobals annotation on the test case class and test method level.

PHPUnit Manual – Appendix B. Annotations, The code coverage report feature requires the Xdebug (2.5.0 or later) and tokenizer Please note that it is not recommended to install PHPUnit globally, to the setUp() method. A test that emits output will fail in strict mode. Set up fixture 3–5. Run Test, Verify outcome and Teardown gpg phpunit.phar.asc gpg: This package is required to enforce test timeouts in strict mode.

[PDF] PHPUnit Manual, PHPUnit 4's new strict mode applies different code coverage rules than but have you tried to create a coverage report with these settings? Related issues: +#2597644: ComposerIntegrationTest::testAllModulesReplaced()  Combining --strict and --coverage-[anything] apparently turns on a super-strict mode for coverage, where any method call not explicitly @covered or @used gets flagged as risky. In truth, it *is* risky, because it's not in isolation, so we can't really unit test it.

Comments
  • It's usually advisable that constructors do as little work as possible, in fact if at all possible they shouldn't do anything other than assign their arguments to instance variables. A simple constructor doesn't really need to be tested so its effect on the code coverage report shouldn't really matter. As for your case, you could try simply having an empty setup() method and doing $this object = new ObjectUnderTest() in every test case.
  • @GordonM The only thing i want to test is, if the typehinting for the dependencies (database, logger, ...) is set up correct. The __construct-method does assignments only.
  • I am fully aware of that. The problem is: The test does not execute __construct, only the setUp does, nevertheless I have to specify @covers for __construct on every test, else the test is classified as risky.
  • setUp() and teardown() are executed for every test, so aren't they part of the test in that sense? I haven't tested it, but I expect that if I use the @expectedException annotation and throw that exception in my setup() or teardown(), the test will pass because of that....
  • In my opinion for unit-tests it is okay, yes. The @codeCoverageIgnore in your example won't work, because it marks the setUp function to be ignored, not the constructor. One could add it to the __construct function, but then it would be ignored by all tests.