Tutorial Link

SandboxVersion: HDP 2.3.2
AmbariVersion2.1.2
Hadoop stack version:Hadoop2.7.1.2.3.2.0-2950

Issue 1: Error initializing SparkContext when executing spark-shell command

When you issue the command as root

spark-shell --master yarn-client --driver-memory 512m --executor-memory 512m

You would would receive the error:

ERROR SparkContext: Error initializing SparkContext.org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user/root/.sparkStaging/application_1451921066894_0001":hdfs:hdfs:drwxr-xr-x

Resolution

sudo su - hdfs
hdfs dfs -mkdir /user/root
hdfs dfs -chown root:hdfs /user/root
exit

 

Other Popular Courses