Caused by: java.lang.OutOfMemoryError: Java heap space

This particular case is related to "Reducer tasks of hive job fails with Out Of Memory error during shuffle fetcher stage"

Fix:

  • Increase hive.tez.container.size if it is set too low.
  • tez.runtime.shuffle.memory.limit.percent from default value 0.7

Changed to 0.4

  • Decrease tez.runtime.shuffle.fetch.buffer.percent from default 0.25 to .15 if needed. (Different values were tested between the range of 0.25 to 0.10)
 
----------------------
Also, If we are using MR then we might need to revisit the mapper/reducer container's heap size accordingly.
 
 

Other Popular Courses