Exception
Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcII$sp
at SparkPi$.main(SparkPi.scala:14)
at SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Problem:
It seems that you have compiled and generated your jar file with an upper version of the Scala compiler than the one spark is using.
Solution
Step 1
Run the following command from spark-installed-directory\binspark-shell.cmd (or.sh)
and note the Scala version, My Spark version was 2.4.3 and Scala version 2.11.12
Step 2
Change the scala version into your build.sbt to 2.11.12 (per your configuration).My build.sbt is
name := "SparkPi Project"
version := "1.0"
scalaVersion := "2.11.12"
val sparkVersion = "2.4.5"
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion % "provided"
Now when you submit you jar to Spark, it will run without errors.
Hope you find this post helpful.
Comments
Post a Comment