How to Setup the PYSPARK_PYTHON Environment Variable to Make Use of a Specific (non-default) Version of Python on the BDA

Asset ID:	1-71-2384776.1
Update Date:	2018-04-29
Keywords:

Solution Type Technical Instruction Sure

Solution 2384776.1 : How to Setup the PYSPARK_PYTHON Environment Variable to Make Use of a Specific (non-default) Version of Python on the BDA

Applies to:

Big Data Appliance X6-2 Hardware - Version All Versions to All Versions [Release All Releases]
x86_64

Goal

The BDA ships with an OS version which comes with a specific version of Python.

This note details how to use the PYSPARK_PYTHON environment variable to use a specific (non-default) version of Python.

Solution

Two ways of setting the PYSPARK_PYTHON environment variable are:

1. To use PYSPARK_PYTHON on a per-session basis, set the environment variable with:

# export PYSPARK_PYTHON=/path/to/python

2. To retain the PYSPARK_PYTHON definition between sessions for consistent use add the following to: /etc/spark/conf/spark-env.sh and/or /opt/cloudera/parcels/CDH/lib/spark/conf/spark-env.sh (if using parcels):

export PYSPARK_PYTHON=/path/to/python

Attachments

This solution has no attachment