How to link PyCharm with PySpark?

pyspark tutorial
pycharm connect to spark cluster
pyspark-stubs pycharm
pyspark download
pyspark_driver_python pycharm
pyspark ide
install pyspark
could not find valid spark_home while searching pycharm

I'm new with apache spark and apparently I installed apache-spark with homebrew in my macbook:

Last login: Fri Jan  8 12:52:04 on console
user@MacBook-Pro-de-User-2:~$ pyspark
Python 2.7.10 (default, Jul 13 2015, 12:05:58)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/01/08 14:46:44 INFO SparkContext: Running Spark version 1.5.1
16/01/08 14:46:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/08 14:46:47 INFO SecurityManager: Changing view acls to: user
16/01/08 14:46:47 INFO SecurityManager: Changing modify acls to: user
16/01/08 14:46:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); users with modify permissions: Set(user)
16/01/08 14:46:50 INFO Slf4jLogger: Slf4jLogger started
16/01/08 14:46:50 INFO Remoting: Starting remoting
16/01/08 14:46:51 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.64:50199]
16/01/08 14:46:51 INFO Utils: Successfully started service 'sparkDriver' on port 50199.
16/01/08 14:46:51 INFO SparkEnv: Registering MapOutputTracker
16/01/08 14:46:51 INFO SparkEnv: Registering BlockManagerMaster
16/01/08 14:46:51 INFO DiskBlockManager: Created local directory at /private/var/folders/5x/k7n54drn1csc7w0j7vchjnmc0000gn/T/blockmgr-769e6f91-f0e7-49f9-b45d-1b6382637c95
16/01/08 14:46:51 INFO MemoryStore: MemoryStore started with capacity 530.0 MB
16/01/08 14:46:52 INFO HttpFileServer: HTTP File server directory is /private/var/folders/5x/k7n54drn1csc7w0j7vchjnmc0000gn/T/spark-8e4749ea-9ae7-4137-a0e1-52e410a8e4c5/httpd-1adcd424-c8e9-4e54-a45a-a735ade00393
16/01/08 14:46:52 INFO HttpServer: Starting HTTP Server
16/01/08 14:46:52 INFO Utils: Successfully started service 'HTTP file server' on port 50200.
16/01/08 14:46:52 INFO SparkEnv: Registering OutputCommitCoordinator
16/01/08 14:46:52 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/01/08 14:46:52 INFO SparkUI: Started SparkUI at http://192.168.1.64:4040
16/01/08 14:46:53 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/01/08 14:46:53 INFO Executor: Starting executor ID driver on host localhost
16/01/08 14:46:53 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50201.
16/01/08 14:46:53 INFO NettyBlockTransferService: Server created on 50201
16/01/08 14:46:53 INFO BlockManagerMaster: Trying to register BlockManager
16/01/08 14:46:53 INFO BlockManagerMasterEndpoint: Registering block manager localhost:50201 with 530.0 MB RAM, BlockManagerId(driver, localhost, 50201)
16/01/08 14:46:53 INFO BlockManagerMaster: Registered BlockManager
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.5.1
      /_/

Using Python version 2.7.10 (default, Jul 13 2015 12:05:58)
SparkContext available as sc, HiveContext available as sqlContext.
>>>

I would like start playing in order to learn more about MLlib. However, I use Pycharm to write scripts in python. The problem is: when I go to Pycharm and try to call pyspark, Pycharm can not found the module. I tried adding the path to Pycharm as follows:

Then from a blog I tried this:

import os
import sys

# Path for spark source folder
os.environ['SPARK_HOME']="/Users/user/Apps/spark-1.5.2-bin-hadoop2.4"

# Append pyspark  to Python Path
sys.path.append("/Users/user/Apps/spark-1.5.2-bin-hadoop2.4/python/pyspark")

try:
    from pyspark import SparkContext
    from pyspark import SparkConf
    print ("Successfully imported Spark Modules")

except ImportError as e:
    print ("Can not import Spark Modules", e)
    sys.exit(1)

And still can not start using PySpark with Pycharm, any idea of how to "link" PyCharm with apache-pyspark?.

Update:

Then I search for apache-spark and python path in order to set the environment variables of Pycharm:

apache-spark path:

user@MacBook-Pro-User-2:~$ brew info apache-spark
apache-spark: stable 1.6.0, HEAD
Engine for large-scale data processing
https://spark.apache.org/
/usr/local/Cellar/apache-spark/1.5.1 (649 files, 302.9M) *
  Poured from bottle
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/apache-spark.rb

python path:

user@MacBook-Pro-User-2:~$ brew info python
python: stable 2.7.11 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org
/usr/local/Cellar/python/2.7.10_2 (4,965 files, 66.9M) *

Then with the above information I tried to set the environment variables as follows:

Any idea of how to correctly link Pycharm with pyspark?

Then when I run a python script with the above configuration I have this exception:

/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Users/user/PycharmProjects/spark_examples/test_1.py
Traceback (most recent call last):
  File "/Users/user/PycharmProjects/spark_examples/test_1.py", line 1, in <module>
    from pyspark import SparkContext
ImportError: No module named pyspark

UPDATE: Then I tried this configurations proposed by @zero323

Configuration 1:

/usr/local/Cellar/apache-spark/1.5.1/ 

out:

 user@MacBook-Pro-de-User-2:/usr/local/Cellar/apache-spark/1.5.1$ ls
CHANGES.txt           NOTICE                libexec/
INSTALL_RECEIPT.json  README.md
LICENSE               bin/

Configuration 2:

/usr/local/Cellar/apache-spark/1.5.1/libexec 

out:

user@MacBook-Pro-de-User-2:/usr/local/Cellar/apache-spark/1.5.1/libexec$ ls
R/        bin/      data/     examples/ python/
RELEASE   conf/     ec2/      lib/      sbin/

How to link PyCharm with PySpark?, 1 Answer. Pycharm. Spark. Firstly in your Pycharm interface, install Pyspark by following these steps: Go to File -> Settings -> Project Interpreter. Now, create Run configuration: Add PySpark library to the interpreter path (required for code completion): Go to File -> Settings -> Project Interpreter. Firstly in your Pycharm interface, install Pyspark by following these steps: Go to File -> Settings -> Project Interpreter. Click on install button and search for PySpark. Click on install package button. Manually with user provided Spark installation.

Here's how I solved this on mac osx.

  1. brew install apache-spark
  2. Add this to ~/.bash_profile

    export SPARK_VERSION=`ls /usr/local/Cellar/apache-spark/ | sort | tail -1`
    export SPARK_HOME="/usr/local/Cellar/apache-spark/$SPARK_VERSION/libexec"
    export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
    export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH
    
  3. Add pyspark and py4j to content root (use the correct Spark version):

    /usr/local/Cellar/apache-spark/1.6.1/libexec/python/lib/py4j-0.9-src.zip
    /usr/local/Cellar/apache-spark/1.6.1/libexec/python/lib/pyspark.zip
    

How to use PySpark in PyCharm IDE - Steven Gong, Table Of Contents. Pre-Requisites; Getting started with Spark on Windows; PyCharm Configuration. Pre-Requisites. Both Java and Python are  Here is the setup that works for me. Set up Intellisense: Click File -> Settings -> Project: -> Project Interpreter. Click the gear icon to the right of the Project Interpreter dropdown. Click More… from the context menu. Choose the interpreter, then click the “Show Paths” icon (bottom right)

Here is the setup that works for me (Win7 64bit, PyCharm2017.3CE)

Set up Intellisense:

  1. Click File -> Settings -> Project: -> Project Interpreter

  2. Click the gear icon to the right of the Project Interpreter dropdown

  3. Click More... from the context menu

  4. Choose the interpreter, then click the "Show Paths" icon (bottom right)

  5. Click the + icon two add the following paths:

    \python\lib\py4j-0.9-src.zip

    \bin\python\lib\pyspark.zip

  6. Click OK, OK, OK

Go ahead and test your new intellisense capabilities.

Getting started with PySpark on Windows and PyCharm – Harshad , Introduction - Setup Python, PyCharm and Spark on Windows As part of this blog post we will Click on the link to launch the download page. Develop Python program using PyCharm. you will find ‘gettingstarted’ folder under project. Right click on the ‘gettingstarted’ folder. choose new Python file and name it HelloWorld. type command “print(“Hello World”) in the file. Right click and run the program. You should see Hello World in the

Configure pyspark in pycharm (windows)

File menu - settings - project interpreter - (gearshape) - more - (treebelowfunnel) - (+) - [add python folder form spark installation and then py4j-*.zip] - click ok

Ensure SPARK_HOME set in windows environment, pycharm will take from there. To confirm :

Run menu - edit configurations - environment variables - [...] - show

Optionally set SPARK_CONF_DIR in environment variables.

Setup Spark Development Environment – PyCharm and Python , Connect with me or follow me at https://www.linkedin.com/in/durga0gadiraju https​://www Duration: 7:07 Posted: Dec 28, 2017 Install pyspark and pypandoc as shown below. PyCharm –> Preferences –> Project Interpreter. Go to PyCharm –> Preferences –> Project Interpreter. Click on Add Content Root. Here you need to add paths

I used the following page as a reference and was able to get pyspark/Spark 1.6.1 (installed via homebrew) imported in PyCharm 5.

http://renien.com/blog/accessing-pyspark-pycharm/

import os
import sys

# Path for spark source folder
os.environ['SPARK_HOME']="/usr/local/Cellar/apache-spark/1.6.1"

# Append pyspark  to Python Path
sys.path.append("/usr/local/Cellar/apache-spark/1.6.1/libexec/python")

try:
    from pyspark import SparkContext
    from pyspark import SparkConf
    print ("Successfully imported Spark Modules")
except ImportError as e:
    print ("Can not import Spark Modules", e)
sys.exit(1)

With the above, pyspark loads, but I get a gateway error when I try to create a SparkContext. There's some issue with Spark from homebrew, so I just grabbed Spark from the Spark website (download the Pre-built for Hadoop 2.6 and later) and point to the spark and py4j directories under that. Here's the code in pycharm that works!

import os
import sys

# Path for spark source folder
os.environ['SPARK_HOME']="/Users/myUser/Downloads/spark-1.6.1-bin-hadoop2.6"

# Need to Explicitly point to python3 if you are using Python 3.x
os.environ['PYSPARK_PYTHON']="/usr/local/Cellar/python3/3.5.1/bin/python3"

#You might need to enter your local IP
#os.environ['SPARK_LOCAL_IP']="192.168.2.138"

#Path for pyspark and py4j
sys.path.append("/Users/myUser/Downloads/spark-1.6.1-bin-hadoop2.6/python")
sys.path.append("/Users/myUser/Downloads/spark-1.6.1-bin-hadoop2.6/python/lib/py4j-0.9-src.zip")

try:
    from pyspark import SparkContext
    from pyspark import SparkConf
    print ("Successfully imported Spark Modules")
except ImportError as e:
    print ("Can not import Spark Modules", e)
    sys.exit(1)

sc = SparkContext('local')
words = sc.parallelize(["scala","java","hadoop","spark","akka"])
print(words.count())

I had a lot of help from these instructions, which helped me troubleshoot in PyDev and then get it working PyCharm - https://enahwe.wordpress.com/2015/11/25/how-to-configure-eclipse-for-developing-with-python-and-spark-on-hadoop/

I'm sure somebody has spent a few hours bashing their head against their monitor trying to get this working, so hopefully this helps save their sanity!

Develop pyspark program using Pycharm on Windows 10, Download Apache Spark distribution pre-built for Hadoop (link). 2. Unpack the archive. This directory will be later referred as $SPARK_HOME . 3. Start PyCharm  PyCharm Configuration. Configure the python interpreter to support pyspark by following the below steps. Create a new virtual environment (File -> Settings -> Project Interpreter -> select Create Virtual Environment in the settings option) In the Project Interpreter dialog, select More in the settings option and then select the new virtual

Integrating Apache Spark 2.0 with PyCharm CE, I found running Spark on Python in an IDE was kinda tricky, hence writing this post to get started with development on IDE using pyspark I have  A wonderful IDE to work with. This is where PyCharm from the great people at JetBrains come into play. But PySpark is not a native Python program, it merely is an excellent wrapper around Spark which in turn runs on the JVM. Therefore it’s not completely trivial to get PySpark working in PyCharm

Get started with Pyspark on Mac using an IDE-PyCharm, Install pyspark and pypandoc as shown below. PyCharm –> Preferences –> Project Interpreter. Set_Project_Interpreter. Go to PyCharm  How to programe in pyspark on Pycharm locally, and execute the spark job remotely. I have a Hadoop cluster of 4 worker nodes and 1 master node. Spark distribution (spark-1.4.0-bin-hadoop2.6) in installed on all nodes. I use ssh to connect to the master node from my laptop to execute Hadoop or Spark jobs on the cluster.

How to Setup PyCharm to Run PySpark Jobs, 2 - Articles Related. 3 - Prerequisites. 4 - Steps. 4.1 - Install Python. 4.2 - Install Spark. 4.3 - Install third package. 4.4 - Default Run Configuration.

Comments
  • BTW, this is how you're editing the interpreter paths, at least in PyCharm 2016: jetbrains.com/help/pycharm/2016.1/… Select the "Show paths for the selected interpreter" button
  • On Mac version of PyCharm (v-2017.2), the Project Interpreter is under Preferences... instead of File/Settings
  • With option 1, how do you add Spark JARs/packages? e.g., I need com.databricks:spark-redshift_2.10:3.0.0-preview1
  • @lfk Either through configuration files (spark-defaults.conf) or through submit args - same as with Jupyter notebook. Submit args can defined in PyCharm's Environment variables, instead of code, if you prefer this option.
  • which version of pycharm is this? I am on 2016.1 community edition and I don't see this window.
  • 2016.1 Im on osx but it should be similar. Go under 'Preferences'. Click on your project on the left.
  • Thanks. This helped me on IntelliJ IDEA, which doesn't have the Project Interpreter setting.
  • Could you explain what adding to the content root does? I didn't need to do that... I just put the $SPARK_HOME/python in the interpreter classpath and added the Environment variables and it works as expected.
  • @cricket_007 The 3rd point: Add pyspark and py4j to content root (use the correct Spark version) helped me in code completion. How did you get it done by changing Project Interpreter?
  • @ml_student I'll also mention that if you follow the video method (which would be my recommendation for its speed and ease) you'll need to instantiate a SparkContext object at the beginning of your script as well. I note this because using the interactive pyspark console via the command line automatically creates the context for you, whereas in PyCharm, you need to take care of that yourself; syntax would be: sc = SparkContext()
  • you should be able to get it working in PyCharm by setting the project's interpreter to spark-submit -- Tried it. "The selected file is not a valid home for Python SDK". Same for bin/pyspark
  • This way pycharm will think its another lib.