...
On this Page
| Table of Contents | ||||
|---|---|---|---|---|
|
Overview
...
has the ability to execute a Python script natively on local/remote Python executors. Remote Python Executor (RPE) can be installed either on the same node as the Snaplex or a remote node
...
that is accessible by the Snaplex.
Steps
Establish an SSH connection to the node.
Install Docker CE.
Create and start the RPE container.
SSH to the Node
...
Establish an SSH
...
connection to the node where you want to install the RPE. Root access may be required for the latter steps.
Install Docker
Follow the instructions on the
...
official Docker site
...
to install Docker CE on your machine.
...
Click here
...
for Ubuntu, here
...
for CentOS,
...
and here for Windows.
...
If using Amazon Linux AMI
...
, click here.
Limitation
The Remote Python Execute can be installed only on Groundplexes and not on Cloudplexes.
Start RPE
Docker Images
A Docker container is an instance created from a Docker image. We provide RPE as Docker images which you can
...
download from here. Currently, there are
...
four tags (types) available.
min: the minimal version for CPU instances.
ds: the data science version for CPU instances containing recommended libraries for data science.
min-cuda9-cudnn7: the minimal version for GPU instances.
ds-cuda9-cudnn7: the data science version for GPU instances containing recommended libraries for data science.
Below is the list of Python libraries in
...
the ds
...
and ds-cuda9-cudnn7
...
tags.
Library | Version |
|---|---|
simplejson | 3.16.0 |
requests | 2.20.0 |
jsonpickle | 1.0 |
python-dateutil | 2.7.4 |
more-itertools | 4.3.0 |
pydub | 0.23.0 |
numpy | 1.15.3 |
scipy | 1.1.0 |
scikit-learn | 0.20.0 |
xgboost | 0.80 |
lightgbm | 2.2.1 |
pillow | 5.3.0 |
bokeh | 0.13.0 |
pandas | 0.23.4 |
tensorflow | 1.5.0 |
keras | 2.2.4 |
nltk | 3.3 |
textblob | 0.15.1 |
| Note |
|---|
ds-cuda9-cudnn7 has tensorflow-gpu==1.11.0 instead of tensorflow==1.5.0. |
CPU Instance
Execute the following command to start the RPE container data science version:
...
| Code Block |
|---|
...
sudo docker pull snaplogic/rpe:ds sudo docker run --memory-swap="-1" --restart=always |
...
-dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic/rpe:ds |
Execute the following command to start the RPE container minimal version:
...
| Code Block |
|---|
...
sudo docker pull snaplogic/rpe:min sudo docker run --memory-swap="-1" --restart=always |
...
-dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic/rpe:min |
GPU Instance
Execute the following command to start the RPE container data science version for GPU instance:
...
| Code Block |
|---|
...
sudo docker pull snaplogic/rpe:ds-cuda9-cudnn7 sudo docker run --runtime=nvidia --memory-swap="-1" --restart=always |
...
-dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic/rpe:ds-cuda9-cudnn7 |
Execute the following command to start the RPE container minimal version for GPU instance:
...
| Code Block |
|---|
...
sudo docker pull snaplogic/rpe:min-cuda9-cudnn7 sudo docker run --runtime=nvidia --memory-swap="-1" --restart=always |
...
-dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic/rpe:min-cuda9-cudnn7 |
Options
...
The following table describes each option:
Option | Format | Description |
|---|---|---|
--memory-swap="-1" --restart=always | --memory-swap="<memory_swap_limit>" --restart=always | The container can use unlimited space for memory swapping. See Memory Swap |
...
for more information. "–restart=always" automatically restarts the docker container if the machine restarts. | ||
-p 5301:5301 | -p <host_port>:5301 | The RPE is accessible from the host port which is 5301 by default. You may change the host port to support multiple containers on the same node. |
-e "REMOTE_PYTHON_EXECUTOR_TOKEN=" | -e "REMOTE_PYTHON_EXECUTOR_TOKEN=<token>" | The default token is empty. |
...
A strong token is recommended. | ||
-v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ | -v <log_dir>/opt/remote_python_executor_log/ | The log is mounted to /opt/remote_python_executor_log/by default. This location can be changed. |
--name=rpe | --name=<container_name> | The |
...
container’s name |
...
can be changed in case of multiple containers on the same node. |
Manage RPE
Since the RPE starts with the container, use the following commands to start, stop, restart and remove the RPE.
...
| Code Block |
|---|
...
sudo docker start rpe
sudo docker stop rpe
sudo docker restart rpe
sudo docker rm rpe |
Custom Image
...
Download the custom RPE package
...
. This package contains Dockerfile and others. You can modify the Dockerfile and add required libraries in requirements.txt. Then,
...
run the following command to build the Docker image
...
.
Tip: Make sure to cd into the directory containing the Dockerfile
...
.
...
| Code Block |
|---|
...
|
...
sudo docker build --no-cache -t snaplogic_custom_rpe . |
Once the image is built, execute the following command to create and start the custom RPE container.
...
| Code Block |
|---|
...
...
sudo docker run --memory-swap="-1" --restart=always |
...
-dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic_custom_rpe |
...
...
| Attachments | ||||||
|---|---|---|---|---|---|---|
|
Related content
Docker installation: