Hot questions for Using Mockito in hbase

Question:

This is the method that I am testing. This method gets some Bytes from a Hbase Database based on an specific id, in this case called dtmid. The reason I why I want to return some specific values is because I realized that there is no way to know if an id will always be in Hbase. Also, the column Family and column name could change.

@Override
   public void execute(Tuple tuple, BasicOutputCollector collector) {
      try {
        if (tuple.size() > 0) {
            Long dtmid = tuple.getLong(0);

            byte[] rowKey = HBaseRowKeyDistributor.getDistributedKey(dtmid);
            Get get = new Get(rowKey);
            get.addFamily("a".getBytes());
            Result result = table.get(get);
            byte[] bidUser = result.getValue("a".getBytes(),
                    "co_created_5076".getBytes());
            collector.emit(new Values(dtmid, bidUser));
        }

    } catch (IOException e) {
        e.printStackTrace();

    }

}

On my main class when this method is called I want to return a specific value. The method should return some bytes.

    byte[] bidUser = result.getValue("a".getBytes(),
                    "co_created_5076".getBytes());

This is what I have on my Unit Test.

   @Test
   public void testExecute() throws IOException {
      long dtmId = 350000000770902930L;
      final byte[] COL_FAMILY = "a".getBytes();
      final byte[] COL_QUALIFIER = "co_created_5076".getBytes();

      //setting a key value pair to put in result
      List<KeyValue> kvs = new ArrayList<KeyValue>();
      kvs.add(new KeyValue("--350000000770902930".getBytes(), COL_FAMILY, COL_QUALIFIER, Bytes.toBytes("ExpedtedBytes")));
      // I create an Instance of result
      Result result = new Result(kvs);

      // A mock tuple with a single dtmid
      Tuple tuple = mock(Tuple.class);
      bolt.table = mock(HTable.class);
      Result mcResult = mock(Result.class);
      when(tuple.size()).thenReturn(1);
      when(tuple.getLong(0)).thenReturn(dtmId);
      when(bolt.table.get(any(Get.class))).thenReturn(result);
      when(mcResult.getValue(any(byte[].class), any(byte[].class))).thenReturn(Bytes.toBytes("Bytes"));

      BasicOutputCollector collector = mock(BasicOutputCollector.class);

      // Execute the bolt.
      bolt.execute(tuple, collector);


      ArgumentCaptor<Values> valuesArg = ArgumentCaptor
            .forClass(Values.class);
      verify(collector).emit(valuesArg.capture());

      Values d = valuesArg.getValue();
      //casting this object in to a byteArray.
      byte[] i = (byte[]) d.get(1);

      assertEquals(dtmId, d.get(0));
   }

I am using this down here to return my bytes.For some reason is not working.

  when(mcResult.getValue(any(byte[].class), any(byte[].class))).thenReturn(Bytes
        .toBytes("myBytes"));

For some reason when I capture the values, I still get the bytes that I specified here:

  List<KeyValue> kvs = new ArrayList<KeyValue>();
    kvs.add(new KeyValue("--350000000770902930".getBytes(),COL_FAMILY,       COL_QUALIFIER, Bytes
            .toBytes("ExpedtedBytes")));
    Result result = new Result(kvs);

Answer:

How about replacing

when(bolt.table.get(any(Get.class))).thenReturn(result);

with...

when(bolt.table.get(any(Get.class))).thenReturn(mcResult);

Question:

I am running my code with mockito framework. Framework is creating mocked object for One Implementation and not creating any mock object for other object due to that it is throwing null pointer exceptions. Here is my code and output:

package com.sohi;

import java.io.IOException;

import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;

public class HbaseExample {

    private HTablePool pool;

    private static final String HTABLE_NAME = "table1";

    public String getValue(String rowKey, String columnFamily, String columnName) throws IOException {

        HTableInterface table = pool.getTable(HTABLE_NAME);

        Get get = new Get(Bytes.toBytes(rowKey)).addColumn(Bytes.toBytes(columnFamily), Bytes.toBytes(columnName));

        System.out.println("Is table Null  ? " + (table == null));

        Result result = table.get(get);

        System.out.println("is result null ? " + (result == null));

        byte [] val = result.value();



        return Bytes.toString(val);
    }

}

My Mockito Test class is :


import static org.junit.Assert.*;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.runners.MockitoJUnitRunner;

import com.sohi.HbaseExample;


@RunWith(MockitoJUnitRunner.class)
public class HbaseExampleTest {

    @Mock
    HTablePool pool;

    @Mock
    HTable hTable;

    @Mock
    Result result;


    @InjectMocks
    HbaseExample hbase = new HbaseExample();

    private static final String HTABLE_NAME = "table1";

    private static final String ROW_KEY = "k1";
    private static final String COLUMN_FAMILY = "col1";
    private static final String COLUMN_NAME = "c1";

    private static final String CELL_VALUE = "v1";

    @Test
    public void test1() throws Exception {
        Get get1 = new Get(Bytes.toBytes(ROW_KEY)).addColumn(Bytes.toBytes(COLUMN_FAMILY), Bytes.toBytes(COLUMN_NAME));

        Mockito.when(pool.getTable(HTABLE_NAME)).thenReturn(hTable);
        Mockito.when(hTable.get(get1)).thenReturn(result);
        Mockito.when(result.value()).thenReturn(Bytes.toBytes(CELL_VALUE));

        String str = hbase.getValue(ROW_KEY, COLUMN_FAMILY, COLUMN_NAME);

        assertEquals(str, CELL_VALUE);

    }

}

Output is :

Is table Null ? false is result null ? true

And Also throwing null pointer exception near result.value().

only table object is getting mocked.


Answer:

The problem is here:

 Mockito.when(hTable.get(get1)).thenReturn(result);

This does not match your actual call, because your get1 is not equal to the Get object that is actually passed. (It looks the same, but Get does not override equals() and so uses the default behaviour of treating any two different objects as being unequal.)

I suggest that you use a Captor to capture the Get object and add asserts to verify that the correct information is present. (I think this is a better way to write this sort of test anyway - it keeps all the assertions together, and leads to better error messages if you pass the wrong thing.)

Question:

I'd like to create an RDD (an actual one, not mocked) that contains mocked elements (with Mockito) in a unit test.

My attempt is:

lazy val sc = SparkContext.getOrCreate()
val myRDD = sc.parallelize(Seq( (Mockito.mock(classOf[ImmutableBytesWritable]), Mockito.mock(classOf[Result])) ))

where ImmutableBytesWritable and Result come from HBase API. I got org.apache.spark.SparkException: Task not serializable

Is there any way possible to achieve my goal? Thank you!


Answer:

By default, Mockito mocks are not serializable, that's why you get the error.

To create serializable mocks, you have to define it explicitly :

mock = Mockito.mock(
    classOf[ImmutableBytesWritable],
    Mockito.withSettings().serializable()
)

The same thing should be applied to your Result mock.

In case you got a java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable$MockitoMock$... exception, you might need to use :

import org.mockito.mock.SerializableMode

mock = Mockito.mock(
    classOf[ImmutableBytesWritable],
    Mockito.withSettings().serializable(SerializableMode.ACROSS_CLASSLOADERS)
)

Finally, you should have something like :

import org.apache.spark.SparkContext 
import org.apache.spark.SparkConf    

import org.apache.hadoop.hbase.io.ImmutableBytesWritable
import org.apache.hadoop.hbase.client.Result

import org.mockito.Mockito
import org.mockito.mock.SerializableMode

object Test extends App {

    val conf = new SparkConf()
        .setMaster("local[2]")
        .setAppName("test")
    lazy val sc = new SparkContext(conf)

    val mockImmutableBytesWritable = Mockito.mock(
        classOf[ImmutableBytesWritable],
        Mockito.withSettings().serializable(
            SerializableMode.ACROSS_CLASSLOADERS
        )
    )

    val mockResult = Mockito.mock(
        classOf[Result],
        Mockito.withSettings().serializable(
            SerializableMode.ACROSS_CLASSLOADERS
        )
    )

    val myRDD = sc.parallelize(Seq((mockImmutableBytesWritable, mockResult)))

    println(myRDD.count)

    sc.stop()

}