Hot questions for Using Mockito in apache httpclient 4.x

Top 10 Java Open Source / Mockito / apache httpclient 4.x

Question:

The code I'm testing works correctly, logs are right.


Tests in error: ConnectorTest: Unable to initialize @Spy annotated field 'entity'.


  • Why can't I use verify on the http entity?
  • Is there a better way to test it? Should I spy on the logs otherwise?

CLASS TO BE TESTED:

public class Connector {

    private static final String hostname = "localhost";
    private int varnishPort = 8000;

    private int connectionTimeout = 5000; //millis
    private int requestTimeout = 5000;
    private int socketTimeout = 5000;

    private final HttpHost host;
    private HttpClient httpClient;
    private HttpEntity entity;
    private HttpResponse response;

    public Connector(){

        host = new HttpHost(this.hostname, this.varnishPort);

        RequestConfig.Builder requestBuilder = RequestConfig.custom();
        requestBuilder = requestBuilder.setConnectTimeout(connectionTimeout);
        requestBuilder = requestBuilder.setConnectionRequestTimeout(requestTimeout);
        requestBuilder = requestBuilder.setSocketTimeout(socketTimeout);

        HttpClientBuilder builder = HttpClientBuilder.create();     
        builder.setDefaultRequestConfig(requestBuilder.build());
        httpClient = builder.build();
    }


    public void invalidateVarnishCache( String level, String idDb ) {

        try{
            String xPurgeRegex = level+"/"+idDb+"$";
            Header header = new BasicHeader( "X-Purge-Regex", xPurgeRegex );
            BasicHttpRequest purgeRequest = new BasicHttpRequest("PURGE", "/" );
            purgeRequest.setHeader(header);

            response = httpClient.execute(host, purgeRequest);

            entity = response.getEntity();

            int statusCode = response.getStatusLine().getStatusCode();

            if( statusCode >= 300 ){

                int respLength = entity.getContent().available();
                byte[] errorResp = new byte[ respLength ];
                entity.getContent().read(errorResp, 0, respLength);
                // log error
            }else{
                // log success
            }

            EntityUtils.consume(entity);

        }catch(Exception e){
            // log exception
        }
    }

}

MY TEST:

@RunWith(MockitoJUnitRunner.class)
public class ConnectorTest {

    @Mock
    private HttpClient httpClient;

    // @Spy
    // private HttpEntity entity;

    @InjectMocks
    private Connector varnishPurger = new Connector();


    @Test
    public void purgeFail() throws Exception{

        HttpResponse response500 = new BasicHttpResponse( new ProtocolVersion( "HTTP/1.1", 0, 0), 500, "NO!_TEST" );
        HttpEntity entity500 = new StringEntity("BAD entity. [test]");
        response500.setEntity(entity500);

        doReturn( response500 )
        .when( httpClient ).execute( isA(HttpHost.class), isA(BasicHttpRequest.class) );

        varnishPurger.invalidateVarnishCache("level_test_500", "id_test_500");

        // verify( entity, times( 1 ) ).getContent().available();  // HOW DO I MAKE THIS WORK?
    }

    ...

}

Answer:

HttpEntity is an interface, it cannot be spied, only mocked. So just change the annotation with @Mock, or another option is to declare an initialised instance :

@Spy HttpEntity entity = new BasicHttpEntity();

Anyway the exception message is rather clear on why this happens :

org.mockito.exceptions.base.MockitoException: Unable to initialize @Spy annotated field 'entity'.
Type 'HttpEntity' is an interface and it cannot be spied on.

    at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl$1.withBefores(JUnit45AndHigherRunnerImpl.java:27)
    at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:276)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37)
    at org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: org.mockito.exceptions.base.MockitoException: Type 'HttpEntity' is an interface and it cannot be spied on.
    ... 21 more

This behaviour is aligned with Mockito.spy(Object), this method cannot be invoked if there's no instance. However the recent Mockito.spy(Class) does not complain about that. This may be a functionality that wasn't ported to the annotation subsystem.

Nonetheless it is semantically wrong to spy on an interface as it doesn't have behaviour.

Question:

I want to mock a behavior by returning different CloseableHttpResponse objects based on urls. For URL1 I want to give 302 response and for url2 I want to give 200 ok response. Method under this test takes url as input and create a HttpGet request object internally and do something with httpresponse object. But I am unable to match the HttpGet argument. Is there any way I can test this method. P.S. httpClient is also a mock object. The following code is not working as the expectation are not able to mock new HttpGet(Url).

   CloseableHttpResponse httpResponse = mock(CloseableHttpResponse.class);
    when(httpClient.execute(new HttpGet(URL1))).thenReturn(httpResponse);
    when(httpResponse.getStatusLine()).thenReturn(new BasicStatusLine(new ProtocolVersion("1.1",0,0),HttpStatus.SC_MOVED_PERMANENTLY,""));
    when(httpResponse.getHeaders(HttpHeaders.LOCATION)).thenReturn( new Header[]{new BasicHeader(HttpHeaders.LOCATION, URL2)});

    CloseableHttpResponse httpResponse1 = mock(CloseableHttpResponse.class);
    when(httpClient.execute(new HttpGet(URL2))).thenReturn(httpResponse1);
    when(httpResponse.getStatusLine()).thenReturn(new BasicStatusLine(new ProtocolVersion("1.1",0,0),HttpStatus.SC_OK,""));
    when(httpResponse.getHeaders(HttpHeaders.CONTENT_LENGTH)).thenReturn( new Header[]{new BasicHeader(HttpHeaders.CONTENT_LENGTH, "0")});

Thanks in advance.


Answer:

You need a custom argument matcher.

So something like this in your test class:

static class HttpGetMatcher extends ArgumentMatcher<HttpGet> {

    private final URL expected;

    //Match by URL
    public HttpGetMatcher(URL expected) {
        this.expected = expected;
    }

    @Override
    public boolean matches(Object actual) {
        // could improve with null checks
        return ((HttpGet) actual).getURI().equals(expected);
    }

    @Override
    public void describeTo(Description description) {
        description.appendText(expected == null ? null : expected.toString());
    }
}

private static HttpGet aHttpGetWithUriMatching(URI expected){
    return argThat(new HttpGetMatcher(expected));
}

The above could also reside in some test utils class if you need in multiple test classes. In that case, the method aHttpGetWithUriMatching would then need to be public.

And then in your test method:

CloseableHttpResponse httpResponse = mock(CloseableHttpResponse.class);
when(httpClient.execute(aHttpGetWithUriMatching(URL1))).thenReturn(httpResponse);
when(httpResponse.getStatusLine()).thenReturn(new BasicStatusLine(new ProtocolVersion("1.1",0,0),HttpStatus.SC_MOVED_PERMANENTLY,""));
when(httpResponse.getHeaders(HttpHeaders.LOCATION)).thenReturn( new Header[]{new BasicHeader(HttpHeaders.LOCATION, URL2)});

Hope this helps.

Question:

I am working on apache http client to consume services and I have a requirement to retry the requests based on timeout and based on response code. For this I have implemented the code as below. How can I write the junit tests for retry logic for both timeout and response code scenarios. I would like to write a unit test in such a way that when I send any post/get request if it returns 429 error code response or any TimeOutException I should make sure retry logic is executed properly. I did not get an idea on how to write unit test for retry logic. By googling I found the below link but it doesn't helpful to me.

Unit testing DefaultHttpRequestRetryHandler

I am using junit,Mockito to write unit tests and PowerMock to mock static methods.

public class GetClient {

private static CloseableHttpClient httpclient;

public static CloseableHttpClient getInstance() {

        try {
            HttpClientBuilder builder = HttpClients.custom().setMaxConnTotal(3)
                    .setMaxConnPerRoute(3);
            builder.setRetryHandler(retryHandler());
            builder.setServiceUnavailableRetryStrategy(new ServiceUnavailableRetryStrategy() {
            int waitPeriod = 200;

            @Override
            public boolean retryRequest(final HttpResponse response, final int executionCount,
                final HttpContext context) {

                int statusCode = response.getStatusLine().getStatusCode();
                return (((statusCode == 429) || (statusCode >= 300 && statusCode <= 399))
                            && (executionCount < 3));
            }

            @Override
            public long getRetryInterval() {
                return waitPeriod;
            }
            });            

            httpclient = builder.build();

        } catch (Exception e) {
            //handle exception
        }
        return httpclient;
    }

     private static HttpRequestRetryHandler retryHandler() {
        return (exception, executionCount, context) -> {
            if (executionCount > maxRetries) {
                // Do not retry if over max retry count
                return false;
            }
            if (exception instanceof InterruptedIOException) {                
                // Timeout
                return true;
            }
            if (exception instanceof UnknownHostException) {
                // Unknown host
                return false;
            }
            if (exception instanceof ConnectTimeoutException) {
                // Connection refused
                return false;
            }
            if (exception instanceof SSLException) {
                // SSL handshake exception
                return false;
            }
            HttpClientContext clientContext = HttpClientContext.adapt(context);
            HttpRequest request = clientContext.getRequest();
            boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
            if (idempotent) {
                // Retry if the request is considered idempotent
                return true;
            }
            return false;
        };
    }
}

public CloseableHttpResponse uploadFile(){    
        CloseableHttpClient httpClient = GetClient.getInstance();
        CloseableHttpResponse response = null;
        try {
            response = httpClient.execute(post);
        } catch (Exception ex) {
            //handle exception
        }
        return response;    
   }

Can anyone help me on this.


Answer:

Your httpClient has a "target" url, lets say localhost:1234. What you want to test is your retry code so you shouldn't touch the httpClient itself ( as it is not your component you shouldn't need to test it also. )

So the problem in hand is when your localhost:1234 response is problematic you want to see that the retry logic that WILL run ( not your implementation.. if it doesn't run with correct conf its their problem ) works.. The only thing you have to do is mock the "localhost:1234" !

This tool http://wiremock.org/ is the perfect choice to do this. You can create stub for your target url and give series of responses depending on almost anything you like.

Your code should look like before calling uploadFile

    stubFor(post(urlEqualTo("/hash"))
        .willReturn(aResponse()
            .withStatus(200)
            .withBody(externalResponse)));

and after calling uploadFile

and verify steps to verify your mock requests that reached your mock endpoint

    Assert.assert* //... whatever you want to assert in your handlers / code / resposnes
    verify(postRequestedFor(urlEqualTo("/hash")));

Question:

I have a mock object:

HttpUriRequest request = Mockito.mock(HttpUriRequest.class);

on which my code calls

request.setHeader(key, value);

How can the mock return the value when the getFirstHeader() method is called

request.getFirstHeader(key).getValue();

where the key and the expected return value are the parameters from the setHeader invocation ?

request.getFirstHeader(key) returns an object of type Header which has a method getValue()


Answer:

Alternatively, you might want to 'spy' on a real object instead of mocking it.

HttpGet request = Mockito.spy(new HttpGet("/"));
request.setHeader(key, value);

Mockito.verify(request).getFirstHeader(key);

Question:

I know how to mock a default HttpClient, but how do I mock the latest (v4.4) HttpClient that is created using a PoolingHttpClientConnectionManager with Mockito?

My code looks like this:

PoolingHttpClientConnectionManager mgr = new PoolingHttpClientConnectionManager();
...
CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(mgr).build();
HttpResponse response = httpClient.execute(request);            
... // here I want to substitute a mocked HttpResponse

Answer:

It is likely to be easier to mock out HttpRequestExecutor than HttpClientConnection. You would still need to provide a no-op implementation of HttpClientConnectionManager in order to prevent HttpClient from creating and connecting sockets

HttpRequestExecutor requestExecutor = Mockito.mock(HttpRequestExecutor.class);
Mockito.when(requestExecutor.execute(
        Mockito.<HttpRequest>any(),
        Mockito.<HttpClientConnection>any(),
        Mockito.<HttpContext>any())).thenReturn(new BasicHttpResponse(HttpVersion.HTTP_1_1, 200, "Hah"));

HttpClientConnectionManager cm = Mockito.mock(HttpClientConnectionManager.class);
HttpClientConnection conn = Mockito.mock(HttpClientConnection.class);
ConnectionRequest connRequest = Mockito.mock(ConnectionRequest.class);
Mockito.when(cm.requestConnection(
        Mockito.<HttpRoute>any(),
        Mockito.any())).thenReturn(connRequest);
Mockito.when(connRequest.get(
        Mockito.anyLong(),
        Mockito.<TimeUnit>any())).thenReturn(conn);

CloseableHttpClient client = HttpClients.custom()
        .setRequestExecutor(requestExecutor)
        .setConnectionManager(cm)
        .build();

CloseableHttpResponse response = client.execute(new HttpGet("http://pampa/"));
try {
    System.out.println(response.getStatusLine());
} finally {
    response.close();
}