HBase Error IllegalStateException when starting Master: hsync

how to start hmaster in hbase
how to start hbase daemons
hbase quick start
hbase master failed to complete initialization
hbase master port
error org apache hadoop hbase regionserver hregionserver failed construction regionserver
hbase hadoop3
hbase error main regionserver hregionserver failed construction regionserver

I'm trying to install HBase on a hadoop cluster and can't figure out why the HMaster fails to initiate when called from start-hbase.sh. The log files indicate an issue with hsync. I have confirmed that zookeeper is running correctly in distributed mode,and I have not had any issues working with the hadoop cluster through spark. When attempting to start HBase the region servers start on all data nodes.

I have hadoop version 3.0.0, zookeeper 3.4.11 and hbase 2.0.0 beta 1.

I have cleared out the zookeeper hbase node and hdfs hbase directory and restarted with no luck. I have the myid zookeeper files stored in the same directory as the .pid files for zookeeper and hbase region servers /usr/local/zookeeper/data/

Below I have an excerpt of the hbase master log file pertaining to the hsync exception.

2018-02-08 17:26:54,786 ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Failed to become active master java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it. at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532) at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547) at java.lang.Thread.run(Thread.java:748) 2018-02-08 17:26:54,787 ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Master server abort: loaded coprocessors are: [] 2018-02-08 17:26:54,787 ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Unhandled exception. Starting shutdown. java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it. at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532) at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547) at java.lang.Thread.run(Thread.java:748)

Below is the hbase-site.xml file

<property>
    <name>hbase.master</name>
    <value>localhost:60000</value>
    <description>The host and port that the HBase master runs at.A value of 'lo$
</property>
<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
    <description>Property from ZooKeeper's config zoo.cfg.The port at which the$
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>MasterNode,Slave1,Slave2</value>
    <description>Comma separated list of servers in the ZooKeeper Quorum.For ex$
</property>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://MasterNode:9000/hbase</value>
</property>
<property>
    <name>hbase.tmp.dir</name>
    <value>/usr/local/zookeeper/data</value>
</property>
<property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/usr/local/zookeeper/data</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
</property>

Below are the alterations made to the hbase-env.sh file settings

export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HBASE_HOME=/usr/local/hadoop/HBase
export HADOOP_HOME=/usr/local/hadoop
export HBASE_MANAGES_ZK=false
export HBASE_PID_DIR=/usr/local/zookeeper/data

Below is the core-site.xml

<property>
    <name>hadoop.tmp.dir</name>
    <value>/app/hadoop/tmp</value>
</property>
<property>
    <name>fs.default.name</name>
    <value>hdfs://MasterNode:9000</value>
</property>

Below is the zoo.cfg

tickTime=2000
dataDir=/usr/local/zookeeper/data
dataLogDir=/usr/local/zookeeper/log
clientPort=2181
initLimit=5
syncLimit=2
server.1=MasterNode:2888:3888
server.2=Slave1:2888:3888
server.3=Slave2:2888:3888

Below is the .bashrc settings related to the hadoop setup

export HADOOP_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_PREFIX=/usr/local/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONFDIR=$HADOOP_HOME/etc/hadoop
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/
export PATH=$PATH:$HADOOP_HOME/bin
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HBASE_HOME=/usr/local/hadoop/HBase
export PATH=$PATH:$HBASE_HOME/bin

I encountered the same problem and my solution is as follows: Edit hbase-site.xml and add the following:

<property>
  <name>hbase.unsafe.stream.capability.enforce</name>
  <value>false</value>
</property>

The property controls whether HBase will check for stream capabilities (hflush/hsync).More details refer to the official website.

[jira] [Commented] (HBASE-20145) HMaster start fails with , https://issues.apache.org/jira/browse/HBASE-20145?page=com. hflush/hsync, which is not acceptable since calling hflush/hsync on the HMaster start fails with IllegalStateException when HADOOP_HOME is set {noformat} > 2018-03-​07 16:59:52,654 ERROR [master//10.200.4.200:16000] master. I have also encountered the same problem, while using HBase standalone mode. The jps is able to list HMaster service, but when I start 'hbase shell' and issue any command it throws ERROR: KeeperErrorCode = NoNode for /hbase/master and HMaster service gets terminated abruptly. So using Hbase in pseudo-distributed mode.

I encountered exactly the same problem as you did. I was running hbase-2.0.1 and hadoop-2.9.1 when I encountered the problem. My solution was I downloaded a later version of hbase(hbase-0.98.13-hadoop2). Probably there is version incompatibility between the hadoop and hbase version you are using.

'hbase.procedure.store.wal.use.hsync' to set the desired , IllegalStateExceptionThe procedure WAL relies on the ability to hsync for proper operation HBase Error IllegalStateException when starting Master: hsync. For some reason a client sometimes fails to locate a row with a IllegalStateException when the region was split and deleted. > [2008-02-25 16:12:39,171] ERROR [http

It appears you downloaded HBase-2.0-beta1 distribution from HBase mirror sites. The default compilation of HBase is done with default hadoop profile i.e 2.0. HBase lib folder contains hadoop dependencies with hadoop-2.7 versions which doesn't support hsync capabilities. So try building HBase from source by passing -Dhadoop.profile=3.0. This builds Hbase distribution with hadoop-3.0 dependencies. Use this distribution to run HBase.

master.HMaster, Always start with the master log (TODO: Which lines?) An error rarely comes alone in Apache HBase, usually when something gets screwed up what will IllegalStateException: The procedure WAL relies on the ability to hsync for proper​  ERROR [master/h136:16000:becomeActiveMaster] master.HMaster: Failed to become active master java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so.

I had similar issue while using HBase 2.2.1 with Hadoop 3.2.1 on a 3-N Hbase cluster while using HDFS as the base.rootdir.

Solution

  1. Set the configuration to hbase.unsafe.stream.capability.enforce to false in hbase-site.xml
  2. Not sure of this was necessary , also changed the hbase.wal.dir to an HDFS directory outside the hbase.rootdir

hbase/troubleshooting.adoc at master · apache/hbase · GitHub, This is a method for Hbase Installation, known as Pseudo Distributed mode of The HMaster refuse to start due to the error below: Java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master. IllegalStateException: The procedure WAL relies on the ability to hsync  HMaster fails to restart 'HMaster: Failed to become active master' apache.hadoop.hbase.master master.HMaster: Unhandled exception. Starting shutdown

Install Hbase 2.x with Hadoop 3.x – Malaysia Technical Expert, If HBase is complaining of an inability to use hsync or hflush it's most likely not a false positive. If you get an error indicating that Java is not installed, but it is on your Start and stop a backup HBase Master (HMaster) server. IllegalStateException: The procedure WAL relies on the ability to hsync for proper​  I've setup a Hbase cluster with two nodes and I've noticed the warning "AssignmentManager: STUCK Region-In-Transition" which is not allowing the master to startup. Node 1: observepreserve.corp.com (Master / Zookeeper) Node 2: knewshoe.corp.com (Region Server) Why it is happeneing and how to fix it? Under Hbase UI I can see the below message.

Apache HBase ™ Reference Guide, HBase has a built-in instance of Zookeeper for just development environments and by default when you start HBase by the command start-hbase.sh it start the  HDFS takes a while to mark a node as dead. You can configure HDFS to avoid using stale DataNodes. By default, HDFS does not mark a node as dead until it is unreachable for 630 seconds. In Hadoop 1.1 and Hadoop 2.x, this can be alleviated by enabling checks for stale DataNodes, though this check is disabled by default.

e5 out of hbase to some tmp place, start all region servers, then start the master, then use hbck to fix meta and assignments (hbck -fixMeta -fixAssignments), then load the data in 721da29b6fad5d99ed337664da35f8e5