How to build Spark 1.2 with Maven (gives java.io.IOException: Cannot run program "javac")?

spark java
spark java tutorial
spark 2.3 download
apache spark tools
spark sql tool
run spark in local mode
spark jar download
spark application

I am trying to build Spark 1.2 with Maven. My goal is to use PySpark with YARN on Hadoop 2.2.

I saw that this was only possible by building Spark with Maven. First, is this true?

If it is true, what is the problem in the log below? How do I correct this?

C:\Spark\spark-1.2.0>mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -DskipTests
clean package
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Parent POM
[INFO] Spark Project Networking
[INFO] Spark Project Shuffle Streaming Service
[INFO] Spark Project Core
[INFO] Spark Project Bagel
[INFO] Spark Project GraphX
[INFO] Spark Project Streaming
[INFO] Spark Project Catalyst
[INFO] Spark Project SQL
[INFO] Spark Project ML Library
[INFO] Spark Project Tools
[INFO] Spark Project Hive
[INFO] Spark Project REPL
[INFO] Spark Project YARN Parent POM
[INFO] Spark Project YARN Stable API
[INFO] Spark Project Assembly
[INFO] Spark Project External Twitter
[INFO] Spark Project External Flume Sink
[INFO] Spark Project External Flume
[INFO] Spark Project External MQTT
[INFO] Spark Project External ZeroMQ
[INFO] Spark Project External Kafka
[INFO] Spark Project Examples
[INFO] Spark Project YARN Shuffle Service
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Parent POM 1.2.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ spark-parent ---
[INFO] Deleting C:\Spark\spark-1.2.0\target
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-versions) @ spark-parent
 ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-scala-sources) @ spark-
parent ---
[INFO] Source directory: C:\Spark\spark-1.2.0\src\main\scala added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-parent --
-
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ spark-parent
 ---
[INFO] No sources to compile
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-test-source (add-scala-test-sources
) @ spark-parent ---
[INFO] Test Source directory: C:\Spark\spark-1.2.0\src\test\scala added.
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile-first) @ spa
rk-parent ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-dependency-plugin:2.9:build-classpath (default) @ spark-parent
---
[INFO] Wrote classpath file 'C:\Spark\spark-1.2.0\target\spark-test-classpath.tx
t'.
[INFO]
[INFO] --- gmavenplus-plugin:1.2:execute (default) @ spark-parent ---
[INFO] Using Groovy 2.3.7 to perform execute.
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-p
arent ---
[INFO]
[INFO] --- maven-shade-plugin:2.2:shade (default) @ spark-parent ---
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO]
[INFO] --- maven-source-plugin:2.2.1:jar-no-fork (create-source-jar) @ spark-par
ent ---
[INFO]
[INFO] --- scalastyle-maven-plugin:0.4.0:check (default) @ spark-parent ---
[WARNING] sourceDirectory is not specified or does not exist value=C:\Spark\spar
k-1.2.0\src\main\scala
Saving to outputFile=C:\Spark\spark-1.2.0\scalastyle-output.xml
Processed 0 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 32 ms
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Networking 1.2.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ spark-network-common_2
.10 ---
[INFO] Deleting C:\Spark\spark-1.2.0\network\common\target
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-versions) @ spark-networ
k-common_2.10 ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-scala-sources) @ spark-
network-common_2.10 ---
[INFO] Source directory: C:\Spark\spark-1.2.0\network\common\src\main\scala adde
d.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-network-c
ommon_2.10 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-netw
ork-common_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory C:\Spark\spark-1.2.0\network\common\s
rc\main\resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ spark-networ
k-common_2.10 ---
[WARNING] Zinc server is not available at port 3030 - reverting to normal increm
ental compile
[INFO] Using incremental compilation
[INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null
)
[INFO] Compiling 42 Java sources to C:\Spark\spark-1.2.0\network\common\target\s
cala-2.10\classes...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  5.267 s]
[INFO] Spark Project Networking ........................... FAILURE [  1.922 s]
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project Bagel ................................ SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Parent POM ...................... SKIPPED
[INFO] Spark Project YARN Stable API ...................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.048 s
[INFO] Finished at: 2015-02-09T10:17:47+08:00
[INFO] Final Memory: 49M/331M
[INFO] ------------------------------------------------------------------------
[**ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compi
le (scala-compile-first) on project spark-network-common_2.10: wrap: java.io.IOE
xception: Cannot run program "javac": CreateProcess error=2, The system cannot f
ind the file specified -> [Help 1]**
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please rea
d the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
xception
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :spark-network-common_2.10

I had first installed JRE instead of JDK. My environment variables still referenced the JRE folder, and so it couldn't find the javac.exe binary.

Building Spark with Maven: error finding javac (but path is correct , This questioner How to build Spark 1.2 with Maven (gives java.io.IOException: Cannot run program "javac")? had a similar problem - it was fixed by fixing the  I posted the complete pom.xml above for the sample standalone project and, in the screenshot, the entire main method code, too. In my last posting, I also included a screenshot for what my maven repo looks like relative to the pom.xml file.

Building Spark - Spark 2.4.5 Documentation, Building a Runnable Distribution; Specifying the Hadoop Version and PySpark Tests with Maven or SBT; Running R Tests; Running Docker-based Integration Test Suites; Change Scala Version Building Spark using Maven requires Maven 3.5.4 and Java 8. By default Spark will build with Hive 1.2.1 bindings. Failed to run (com.github.eirslett:frontend-maven-plugin:1.2:webpack:webpack build:generate-resources) #689 Open DragonWarriorPoo opened this issue Feb 5, 2018 · 2 comments

For this problem you need to set your java environment path correctly in .bashrc file. Then you need to build maven correct on set maven path for that check mvn -version.

Then it will build automaticaly without error.

[#SPARK-15851] Spark 2.0 does not compile in Windows 7, Give Feedback · Switch to desktop version "mvn compile" fails on spark-core due to trying to execute a bash script <arg value="${project.build.directory}/​extra-resources"/> <arg An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "C:\dev\spark\core\. Caused: java.io.IOException: Cannot run program "sh" (in directory "D:\myjenkins\workspace\cal"): CreateProcess error=2, The system cannot find the file specified Home Community

Building Spark - Spark 1.2.2 Documentation, Building Spark using Maven requires Maven 3.0.4 or newer and Java 6+. If you don't run this, you may see errors like the following: and the version on each node, included with yarn.application.classpath. The project site gives instructions for building and running zinc ; OS X users can install it using brew install zinc . Spark 1.6.1 programming guide in Java, Scala and Python. Spark 1.6.1 works with Java 7 and higher. If you are using Java 8, Spark supports lambda expressions for concisely writing functions, otherwise you can use the classes in the org.apache.spark.api.java.function package.

Maven clean package failure with Create Process id =2, I'm trying to build Apache Spark to do this I execute mvn clean package (​obviously IOException: Cannot run program "bash" (in directory CreateProcess error=2, The system cannot find the file specified at java.lang. that I succeed in build giraph 1.1 with jre but giraph 1.2 gives me above error if I use jre instead of jdk. [WARNING] [WARNING] Some problems were encountered while building the effective model for com.journaldev:java-word-count:jar:1.0-SNAPSHOT [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-jar-plugin is missing. @ line 41, column 21 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten

Apache Spark Example: Word Count Program in Java, Apache Spark Example, Apache Spark Word Count Program in Java, Apache To create the project, execute the following command in a directory that you will If you are running maven for the first time, it will take a few seconds to [INFO] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:compile [INFO]  Spark 2.3.1 is built and distributed to work with Scala 2.11 by default. (Spark can be built to work with other versions of Scala, too.) To write applications in Scala, you will need to use a compatible Scala version (e.g. 2.11.X).