Skip to main content

spark-submit java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcII$sp

Exception 


Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcII$sp
        at SparkPi$.main(SparkPi.scala:14)
        at SparkPi.main(SparkPi.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)


Problem:


It seems that you have compiled and generated your jar file with an upper version of the Scala compiler than the one spark is using.

Solution

Step 1

Run the following command from spark-installed-directory\bin

spark-shell.cmd (or.sh)

and note the Scala version, My Spark version was 2.4.3 and Scala version 2.11.12


Step 2

Change the scala version into your build.sbt to 2.11.12 (per your configuration).

My build.sbt is

name := "SparkPi Project"
version := "1.0"
scalaVersion := "2.11.12"
val sparkVersion = "2.4.5"

libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion % "provided"

Now when you submit you jar to Spark, it will run without errors.

Hope you find this post helpful.

Comments

Popular posts from this blog

Eclipse - Server Tomcat v8.5 Server at localhost failed to start.

When I try to launch the tomcat from Eclipse, I encountered the following error Server Tomcat v8.5 Server at localhost failed to start. Solution Step 1  Delete the .snap file located at the following location     eclipse workspace Path\ .metadata\.plugins\org.eclipse.core.resources Step 2 Delete the  tmp0  folder from the following path      eclipse workspace Path \.metadata\.plugins\org.eclipse.wst.server.core Step 3  Delete the server from servers list Step 4  Remove already added Tomcat Server      i)  Click on Define a new Server     ii)  Select Server Runtime Environments     iii) Select the Tomcat Server and remove it as follows Remove Selected Server Step 5 Make sure that correct version of Server is configured in Project Properties Step 6 Restart the Eclipse IDE.

Intellij : How to add @author comment to every new class

 Introduction In this tutorial, we will learn how to add @author comments to every new class that we create. We can achieve it using either of the following two solutions Solution 1:  Automatically add @author comments to every new class using Files and Code Templates Open File -> Settings -> Editor -> File and Code Templates -> Includes Click on Includes . Under File Header , enter the following comments text /**  * @author ${USER}  * @Date ${DATE}   */ Intellij - add @author comments Solution 2: Autocompletion of @author Open File  ->  Settings  ->  Editor  -> Live Templates Select Java and then click on + button In Abbreviation, enter @a In template text , enter the following comments           /**             * @author ${USER}             * @Date ${DATE}            */ In option , Expands with select SPACE Intellij - Autocompletion @author You can simply add the @author comments by typing @a and then click SPACE

hibernate-release-5.4.4.Final - Required Jars

Introduction Hibernate (Object Relational Mapping framework) is an implementation of Java Persistence API (JPA) specification.   Required Jars for Hibernate 5.4.4 Following Jars resided inside the required folder are the mandatory jars required for Hibernate 5.4.4 antlr-2.7.7.jar byte-buddy-1.9.11.jar classmate-1.3.4.jar dom4j-2.1.1.jar FastInfoset-1.2.15.jar hibernate-commons-annotations-5.1.0.Final.jar hibernate-core-5.4.4.Final.jar istack-commons-runtime-3.0.7.jar jandex-2.0.5.Final.jar javassist-3.24.0-GA.jar javax.activation-api-1.2.0.jar javax.persistence-api-2.2.jar jaxb-api-2.3.1.jar jaxb-runtime-2.3.1.jar jboss-logging-3.3.2.Final.jar jboss-transaction-api_1.2_spec-1.1.1.Final.jar stax-ex-1.8.jar txw2-2.3.1.jar Hibernate 5.4.4 release is compatible with  Java 8 or 11  JPA 2.2 References https://hibernate.org/orm/releases/5.4/