| Table of Contents | ||
|---|---|---|
|
Overview
Remote Python Script Snap has the ability to execute a Python script natively on local/remote Python executors. Remote Python Executor (RPE) can be installed either on the same node as the Snaplex or a remote node which is accessible by the Snaplex.
Steps
Download the SnapLogic Python Server Package
This package is available for existing customers. Contact us to download the package.
- Establish SSH connection to the node.
- Install Docker CE.
- Create and start RPE container.
SSH to the Node
Copy the package and extract to the node. Change the directory to snaplogic_python_server_imageEstablish SSH connection to the node where you want to install the RPE. Root access may be required for the latter steps.
Install Docker
Follow the instructions on the the official Docker site to install Docker CE on your machine.
Configure the Python Server
You can change the port and token in run.sh. The default port for Python 2 is 5201, while the default port for Python 3 is 5202. The default security token is "" that can be changed. If the port 5201 and 5202 are exposed to the public, ensure to have a strong security token.
| Paste code macro |
|---|
#!/bin/bash
nohup python /opt/python_server_data/python2_server.py 5201 "" > /opt/python_server_data/python2_server.log &
python3 /opt/python_server_data/python3_server.py 5202 "" > /opt/python_server_data/python3_server.log |
Click here for Ubuntu, here for CentOS, and here for Windows.
Start RPE
Docker Images
Docker container is an instance created from Docker image. We provide RPE as Docker images which can be found here. Currently, there are 4 tags (types) available.
- min: the minimal version for CPU instances.
- ds: the data science version for CPU instances containing recommended libraries for data science.
- min-cuda9-cudnn7: the minimal version for GPU instances.
- ds-cuda9-cudnn7: the data science version for GPU instances containing recommended libraries for data science.
Below is the list of Python libraries in the ds and ds-cuda9-cudnn7 tags.
Library | Version |
|---|---|
| simplejson | 3.16.0 |
| requests | 2.20.0 |
| jsonpickle | 1.0 |
| python-dateutil | 2.7.4 |
| more-itertools | 4.3.0 |
| pydub | 0.23.0 |
| numpy | 1.15.3 |
| scipy | 1.1.0 |
| scikit-learn | 0.20.0 |
| xgboost | 0.80 |
| lightgbm | 2.2.1 |
| pillow | 5.3.0 |
| bokeh | 0.13.0 |
| pandas | 0.23.4 |
| tensorflow | 1.5.0 |
| keras | 2.2.4 |
| nltk | 3.3 |
| textblob | 0.15.1 |
| Note |
|---|
ds-cuda9-cudnn7 has tensorflow-gpu==1.11.0 instead of tensorflow==1.5.0. |
CPU Instance
Execute the following command to start the RPE container minimal version:
| Paste code macro |
|---|
sudo docker run --memory-swap="-1" -dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic/rpe:min |
Execute the following command to start the RPE container data science version:
| Paste code macro |
|---|
sudo docker run --memory-swap="-1" -dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic/rpe:ds |
GPU Instance
Execute the following command to start the RPE container minimal version for GPU instance:
| Paste code macro |
|---|
|
|
Build the Docker Image
Execute the following command to build the Docker image
sudo docker run --memory-swap="-1" -dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic/rpe:min-cuda9-cudnn7 |
Execute the following command to start the RPE container data science version for GPU instance:
| Paste code macro |
|---|
sudo docker run --memory-swap="-1" -dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic/rpe:ds-cuda9-cudnn7 |
Options
See below for explanation of each option:
Option | Format | Description |
|---|---|---|
| --memory-swap="-1" | --memory-swap="<memory_swap_limit>" | The container can use unlimited space for memory swapping. See Memory Swap for more information. |
| -p 5301:5301 | -p <host_port>:5301 | The RPE is accessible from host port which is 5301 by default. You may change the host port to support multiple containers on the same node. |
| -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" | -e "REMOTE_PYTHON_EXECUTOR_TOKEN=<token>" | The default token is empty. Strong token is recommended. |
| -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ | -v <log_dir>/opt/remote_python_executor_log/ | The log is mounted to /opt/remote_python_executor_log/by default. This location can be changed. |
| --name=rpe | --name=<container_name> | The container's name which can be changed in case of multiple containers on the same node. |
Manage RPE
Since the RPE starts with the container, use the following commands to start, stop, restart and remove the RPE.
| Paste code macro |
|---|
sudo docker start rpe
sudo docker stop rpe
sudo docker restart rpe
sudo docker rm rpe |
Custom Image
The custom RPE package can be download here. This package contains Dockerfile and others. You can modify the Dockerfile and add required libraries in requirements.txt. Then, executing the following command to build the Docker image (make sure to cd into the directory containing Dockerfile).
| Paste code macro |
|---|
sudo docker build --no-cache -t snaplogic_ |
custom_ |
rpe . |
Create Container
ExecuteOnce the image is built, execute the following command to create and
start the custom RPE container.
| Paste code macro |
|---|
sudo docker run --memory-swap="-1" -dti -p 5201-5202:5201-5202 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=python_serverrpe snaplogic_pythoncustom_server |
Restarting Python Server
Since the Python server starts with the container, use the following commands to restart the Python server:
rpe |
Helpful Links
- Docker installation:
- RPE Docker image
- RPE package download