Techie Shah

Exception  Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcII$sp         at SparkPi$.main(SparkPi.scala:14)         at SparkPi.main(SparkPi.scala)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Problem: It seems that you have compiled and generated your jar file with an upper version of the Scala compiler than the one spark is using. Solution Step 1 Run the following command from spark-installed-directory\bin spark-shell.cmd (or.sh) and note the Scala version, My Spark version was 2.4.3 and Scala version 2.11.12 Step 2 Change the scala version into your build.sbt to 2.11.12 (per your configuration). My build.sbt is name := "SparkPi Project" version := "1.0" scalaVersion := "...

Introduction  Currently, Apache Spark does not support JAVA 11 and later versions. When you try to run the Spark application, you may get the following exception Exception pyspark.sql.utils.IllegalArgumentException: 'Unsupported class file major version 55' Solution There are different ways to fix this exception like Set environmental variable for JAVA_HOME Modify Apache spark environment configuration file i.e. spark-env.sh or spark-env.cmd In this post, I will help you to set JAVA_HOME using Spark's configuration file Windows Environment  Go to the spark-directory\ conf Create a file by the name of spark-env.cmd  Paste the following line spark-env.cmd                   set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_201 Linux and Mac Go to the spark-directory\ conf Open spark-env.sh Paste the following line spark-env.cmd        export JAVA_HOME=$(user/J...

I am passionate about BigData technologies. I am exploring the Apache Spark these days. I found IBM's BigDataUniversity the best place to learn the trending BigData technologies. I wrote my first Apache Spark application but when I ran it, I encountered the following error Could not locate executable null\bin\winutils.exe  in the Hadoop binaries I searched for it over the internet and found variously detailed solutions. In this post,  I am listing down the steps in an easy way to explain how to fix this issue. Step 1  Download the winutils.exe from the following link winutils 32-bit      (depends on your installed Windows version) winutils 64-bit      (depends on your installed Windows version) Step 2  Create the following two directories in C or any other drive  c:\hadoop\bin  C:\tmp\hive Step 3 Place the downloaded winutils.exe inside the c:\hadoop\ bin folder Step 4...