reading a password from local file and passing in pyspark script

how to protect passwords in python scripts
spark submit --files
python store password config file
how to store username and password in python
encrypt password in python script
python import password from file
spark-submit --py-files
python read password from file

i am trying to read the password from file and passing into SAP connection. Since i have 50+ scripts using same password, where if password gets chnaged i no need to go in each script and change instead i can simply change in one file.

from pyspark.context import SparkContext
from pyspark.sql import HiveContext
from pyspark.sql.functions import *
from pyspark.sql.types import *

sc = SparkContext()
sqlContext = HiveContext(sc)
sqlContext.setConf("spark.sql.tungsten.enabled", "false")

ZTGLINT011_query = """ (select * from ZTGLINT011) a """

#passwd = sc.textFile("file:///root/Documents/pwd.txt")

passwd=cat /root/Documents/pwd.txt

ZTGLINT011 = sqlContext.read.format("jdbc").options(url="jdbc:sap://myIP:30015",currentschema="SAPABAP1",user="loginName",password=passwd,dbtable=ZTGLINT011_query).load()

ZTGLINT011.write.format("parquet").save("/user/admin/sqoop/base/sap/ZTGLINT011/")

in txt file enter content:

{ "Password":"your password" }

Copy paste you code this will work:

from pyspark.context import SparkContext
from pyspark.sql import HiveContext
from pyspark.sql.functions import *
from pyspark.sql.types import *
import json

sc = SparkContext()
sqlContext = HiveContext(sc)
sqlContext.setConf("spark.sql.tungsten.enabled", "false")

ZTGLINT011_query = """ (select * from ZTGLINT011) a """

passwfile = open("/root/Documents/pwd.txt", "r")
contents = passwfile.read()
info = json.dumps(contents)

passwd = info['Password']

ZTGLINT011 = sqlContext.read.format("jdbc").options(url="jdbc:sap://myIP:30015",currentschema="SAPABAP1",user="loginName",password=passwd,dbtable=ZTGLINT011_query).load()    
    ZTGLINT011.write.format("parquet").save("/user/admin/sqoop/base/sap/ZTGLINT011/")

Submitting Applications - Spark 2.1.2 Documentation, The spark-submit script in Spark's bin directory is used to launch applications on Whether to deploy your driver on the worker nodes ( cluster ) or locally as an For Python applications, simply pass a .py file in the place of <application-jar> By default it will read options from conf/spark-defaults.conf in the Spark directory. $ ./bin/pyspark --master local [4] Or, to also add code.py to the search path (in order to later be able to import code), use: $ ./bin/pyspark --master local [4]--py-files code.py. For a complete list of options, run pyspark --help. Behind the scenes, pyspark invokes the more general spark-submit script.


Best thing is to store the password, Username, IP, PORT etc you file /root/Documents/pwd.txt as

============pwd.txt : contents ============

{ "IP": "myIP", "PORT": 8080, "Usernme": "user", "Password": "pass" }

===============================================

In your code :

import json
f= open('/root/Documents/pwd.txt','r')
contents = f.read()
server_details = json.loads(contents)
password = server_details['Password']
username=server_details['Password']
Port =server_details["PORT"]
IP=server_details["IP"]

ZTGLINT011 = sqlContext.read.format("jdbc").options(url="jdbc:sap://"+IP+":"+Port,currentschema="SAPABAP1",user=username,password=password,dbtable=ZTGLINT011_query).load()

you can add currentschema and dbtable also in the txt file

Spark Programming Guide, However, for local testing and unit tests, you can pass “local” to run Spark in- process. Behind the scenes, spark-shell invokes the more general spark- submit script. wholeTextFiles lets you read a directory containing multiple small text files,� Missing value for AzureWebJobsStorage in local.settings.json. This is required for all triggers other than httptrigger, kafkatrigger. You can run 'func azure functionapp fetch-app-settings ' or specify a connection string in local.settings.json. The fix, just add this in to your local.settings,json


from pyspark.context import SparkContext
from pyspark.sql import HiveContext
from pyspark.sql.functions import *
from pyspark.sql.types import *

sc = SparkContext()
sqlContext = HiveContext(sc)
sqlContext.setConf("spark.sql.tungsten.enabled", "false")

ZTGLINT011_query = """ (select * from ZTGLINT011) a """

passwfile = open("/root/Documents/pwd.txt", "r")

for passwd in passwfile:
    ZTGLINT011 = sqlContext.read.format("jdbc").options(url="jdbc:sap://myIP:30015",currentschema="SAPABAP1",user="loginName",password=passwd,dbtable=ZTGLINT011_query).load()    
    ZTGLINT011.write.format("parquet").save("/user/admin/sqoop/base/sap/ZTGLINT011/")

Encrypting passwords for use with Python and SQL Server, We don't want a clear text password stored in the Python scripts and we're not sure how we can secure the passwords. Solution. Securing� Executing the script in an EMR cluster as a step via CLI. Let me explain each one of the above by providing the appropriate snippets. 1.0 Reading csv files from AWS S3 :


Load Data from Teradata in Spark (PySpark), Create a PySpark script file named teradata-jdbc.py with the following code: " PySpark Teradata Example" master = "local" # Create Spark session spark jdbc_url, sql, user, password): return spark.read \ .format('jdbc') \ .option('driver', driver) of your Spark, you may be able to directly use query parameter to pass in your� Create pyspark application and bundle that within script preferably with .py extension. Once a user application is bundled, it can be launched using the bin/spark-submit script. This script takes care of setting up the CLASSPATH with Spark and its dependencies, and can support different cluster managers and deploy modes that Spark supports:


ConfigParser – Work with configuration files, Use the read() method of SafeConfigParser to read the configuration file. [ bug_tracker] url = http://localhost:8080/bugs/ username = dhellmann password = ����†. The codecs file handle can be passed to readfp(), which uses the readline()� I am a newbie to Spark. I'm trying to read a local csv file within an EMR cluster. The file is located in: /home/hadoop/. The script that I'm using is this one: spark = SparkSession \ .builder


Using the Python Connector — Snowflake Documentation, Using execute_stream to Execute SQL Scripts After reading the connection information, connect using either the default authenticator or conn = snowflake. connector.connect( user=USER, password=PASSWORD, The code decrypts the private key file and passes it to the Snowflake driver to create a connection:. Step1: Create a password file and type in your password as a clear text ( Not a Secure Method ) # Write the password into a file and Save it [vagrant@mwivmapp01 ~]$ cat > .passwrdfile vagrant # Display the content of the file [vagrant@mwivmapp01 ~]$ cat .passwrdfile vagrant . Step2: Refer the password file in SSHPASS and pass it to SSH.