Spark development environment for k8s

Sep 29, 2021 1 min read

Local Spark Dev Env with Docker

Development environment for k8s.

Using the spark-operator image to ensure it will be the same environment.

Start container

docker-compose up -d

Copy dependencies jars

docker cp jars/. spark:/opt/spark/jars

Create a local alias for spark-submit

alias spark-s="docker exec -it spark /opt/spark/bin/spark-submit"

Run exemple

spark-s app_3.py

Clean after work

docker-compose down

GitHub

Docker K8s

John was the first writer to have joined pythonawesome.com. He has since then inculcated very effective writing and reviewing culture at pythonawesome which rivals have found impossible to imitate.

Spark development environment for k8s

Local Spark Dev Env with Docker

Start container

Copy dependencies jars

Create a local alias for spark-submit

Run exemple

Clean after work

GitHub

John

A ML training platform to help engineers to build ML models at scale

A python library and command line tool for creating invisible watermark over image

Local Spark Dev Env with Docker

Start container

Copy dependencies jars

Create a local alias for spark-submit

Run exemple

Clean after work

GitHub

A ML training platform to help engineers to build ML models at scale

A python library and command line tool for creating invisible watermark over image

You might also like...