How do I increase Tez's container physical memory?

I've been running some hive scripts on an aws emr 4.8 cluster with hive 1.0 and tez 0.8.

My configurations look like this:

SET hive.exec.compress.output=true;
SET mapred.output.compression.type=BLOCK;
SET hive.exec.dynamic.partition = true;
SET hive.exec.dynamic.partition.mode = nonstrict;
set hive.execution.engine=tez;
set hive.merge.mapfiles=false;
SET hive.default.fileformat=Orc;
set tez.task.resource.memory.mb=5000;
SET hive.tez.container.size=6656;
SET hive.tez.java.opts=-Xmx5120m;
set hive.optimize.ppd=true;

And my global configs are:

hadoop-env.export   HADOOP_HEAPSIZE 4750
hadoop-env.export   HADOOP_DATANODE_HEAPSIZE    4750
hive-env.export HADOOP_HEAPSIZE 4750

While running my script, I get the following error:

Container [pid=19027,containerID=container_1477393351192_0007_02_000001] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.9 GB of 5 GB virtual memory used. Killing container.

On googling this error, I read that set tez.task.resource.memory.mb will change the physical memory limit, but clearly I was mistaken. What am I missing?

I have had this problem a lot. The changing

Set hive.tez.container.size=6656;
Set hive.tez.java.opts=-Xmx4g;

does not fix the problem for me but this does:

set tez.am.resource.memory.mb=4096;

How do I increase Tez's container physical memory?, I have had this problem a lot. The changing. Set hive.tez.container.size=6656; Set hive.tez.java.opts=-Xmx4g;. does not fix the problem for me� Tez will sample source vertices, output sizes and adjust the estimates at run time this is the 1st property that determines initial number of reducers once Tez starts the query hive.tex.min.partition.factor =0.25;-- when auto parallelism enable, this property will be used to put a lower limit to number of reducers that Tez specified

Set the Tez container size to be a larger multiple of the YARN container size (4GB):

SET hive.tez.container.size=4096MB

"hive.tez.container.size" and "hive.tez.java.opts" are the parameters that alter Tez memory settings in Hive. If "hive.tez.container.size" is set to "-1" (default value), it picks the value of "mapreduce.map.memory.mb". If "hive.tez.java.opts" is not specified, it relies on the "mapreduce.map.java.opts" setting. Thus, if Tez specific memory settings are left as default values, memory sizes are picked from mapreduce mapper memory settings "mapreduce.map.memory.mb".

https://documentation.altiscale.com/memory-settings-for-tez

For more info Tez configuration and Tez memory tuning

Note: Set in MB with Ambari

How to change Tez container heapsize, How to change Tez container heapsize. When you run some queries on Hive on Tez, some queries may fail with an OutOfMemory exception. While troubleshooting HIVE performance issues when a TEZ engine is being used, there may be a need to increase the number of mappers used during a query. For example, in the example query, the TEZ engine decided only one mapper was needed.

Incase anyone else stumbles upon this thread trying to solve this above, here is a link to a real solution that worked for me where all the other solutions did not.

http://moi.vonos.net/bigdata/hive-cli-memory/

TL;DR add these to your hive call --hiveconf tez.am.resource.memory.mb=<size as int> --hiveconf tez.am.launch.cmd-opts=""

Tez Memory Tuning – Container is Running Beyond Physical , Can reducing the Tez memory settings help solving memory limit problems? so you need to increase the container size to solve the problem. These settings do: Increase the number of vCores available on each node. This allows to run more tasks than there are actual physical cores on the node. Decrease the amount of RAM requested by the Hive/Tez application. This allows run more containers on each node. The effect of these: More than one tasks can be launched per physical CPU core;

Set hive.tez.container.size=6656
Set hive.tez.java.opts=-Xmx4g

Demystify Apache Tez Memory Tuning - Step by Step, mb and hive.tez.container.sizerespectfully. NOTE: tez.am.launch.cmd-opts is automatically set, so no need to change this. from 2003 found evidence that T levels may increase by nearly 150 percent seven days after you’ve stopped masturbating. A 2007 study on rats found that frequent masturbation lowered androgen

How do I increase Tez's container physical memory?, On googling this error, I read that set tez.task.resource.memory.mb will change the physical memory limit, but clearly I was mistaken. What am I missing? To avail Tez Advance, the customer must fill in an application within the Tez App, select an Advance amount, choose an Advance tenure and agree to the terms and conditions of the Advance. How do I register for Tez Advance?

TezConfiguration, Increasing this can help improve app master scalability for a large number of concurrent tasks. Expert level setting. integer, false, false, false. tez.task.launch. cluster� You keep 85% of the reward we receive by your delegation. With the other 15%, TEZ Rocket will support and further expand the Tezos ecosystem. The purpose of our delegation service is to use most of the profits for Tezos marketing. To delegate to us is to contribute to expanding the large ecosystem.

Apache Hive and Apache Tez – Memory management and Tuning, Tez improves the MapReduce paradigm by dramatically improving its speed, while maintaining MapReduce's ability to scale to petabytes of data. #NetSpeed #Internet #HowToDoAnythingInHindi How To Increase Internet Speed With Proof इन्टरनेट स्पीड को कैसे बढ़ाये? https://theindianstreet.in

Comments
  • tez.am.resource.memory.mb=4096; is the solution for me. in my case it's the app master (am) has too little memory so the job failed when scheduling.
  • Is there a reason SET hive.tez.container.size=6656; won't work?
  • Also, Query returned non-zero code: 1, cause: 'SET hive.tez.container.size=6656MB' FAILED because hive.tez.container.size expects INT type value.
  • So I had set the container size, as well as the tez.java.opts in my configurations already. Despite that, my physical memory is 1G. The altiscale page is exactly where I found these, but didn't work for me. :/
  • does this commands work on hadoop conf? or just for hive? should this works for example (set mapreduce.reduce.memory.mb = 4096)?
  • SO how is it different from SET hive.tez.container.size=6656; SET hive.tez.java.opts=-Xmx5120m;
  • There are no semicolons at the end of the statements. I had a similar failure "... expects INT type value" with container.size. I removed the semicolon at the end of the statement and the failure went away. But I have no idea why that fix works.
  • does this commands work on hadoop conf? or just for hive? should this works for example (set mapreduce.reduce.memory.mb = 4096)?