Skip to main content

Posts

Showing posts with the label Apache Spark

spark-submit java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcII$sp

Exception  Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcII$sp         at SparkPi$.main(SparkPi.scala:14)         at SparkPi.main(SparkPi.scala)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Problem: It seems that you have compiled and generated your jar file with an upper version of the Scala compiler than the one spark is using. Solution Step 1 Run the following command from spark-installed-directory\bin spark-shell.cmd (or.sh) and note the Scala version, My Spark version was 2.4.3 and Scala version 2.11.12 Step 2 Change the scala version into your build.sbt to 2.11.12 (per your configuration). My build.sbt is name := "SparkPi Project" version := "1.0" scalaVersion := "2.11.12" val sparkVersion = "2.4.5&qu

Set default JAVA_HOME path for Apache Spark

Introduction  Currently, Apache Spark does not support JAVA 11 and later versions. When you try to run the Spark application, you may get the following exception Exception pyspark.sql.utils.IllegalArgumentException: 'Unsupported class file major version 55' Solution There are different ways to fix this exception like Set environmental variable for JAVA_HOME Modify Apache spark environment configuration file i.e. spark-env.sh or spark-env.cmd In this post, I will help you to set JAVA_HOME using Spark's configuration file Windows Environment  Go to the spark-directory\ conf Create a file by the name of spark-env.cmd  Paste the following line spark-env.cmd                   set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_201 Linux and Mac Go to the spark-directory\ conf Open spark-env.sh Paste the following line spark-env.cmd        export JAVA_HOME=$(user/Java/jdk1.8.0_201 -v 1.8) Note : Change the installed Java directory accordingly.

Apache Spark - Could not locate executable null\bin\winutils.exe in the Hadoop binaries

I am passionate about BigData technologies. I am exploring the Apache Spark these days. I found IBM's BigDataUniversity the best place to learn the trending BigData technologies. I wrote my first Apache Spark application but when I ran it, I encountered the following error Could not locate executable null\bin\winutils.exe  in the Hadoop binaries I searched for it over the internet and found variously detailed solutions. In this post,  I am listing down the steps in an easy way to explain how to fix this issue. Step 1  Download the winutils.exe from the following link winutils 32-bit      (depends on your installed Windows version) winutils 64-bit      (depends on your installed Windows version) Step 2  Create the following two directories in C or any other drive  c:\hadoop\bin  C:\tmp\hive Step 3 Place the downloaded winutils.exe inside the c:\hadoop\ bin folder Step 4  Set the following environmental variable Name : HADOOP_HOME