MLOps
A project based example of Data pipelines, ML workflow management, API endpoints
and Monitoring.
Tools used:
- Data Pipeline: Dagster
- ML workflow: MLflow
- API Deployment: FastAPI
- Monitoring: ElasticAPM
Requirements
Poetry (dependency management)
$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
$ poetry --version
# Poetry version 1.1.10
pre-commit (static code analysis)
$ pip install pre-commit
$ pre-commit --version
# pre-commit 2.15.0
Minio (s3 compatible object storage)
Follow the instructions here – https://min.io/download
Setup
Environment setup
$ poetry install
MLflow
$ poetry shell
$ export MLFLOW_S3_ENDPOINT_URL=http://127.0.0.1:9000
$ export AWS_ACCESS_KEY_ID=minioadmin
$ export AWS_SECRET_ACCESS_KEY=minioadmin
# make sure that the backend store and artifact locations are same in the .env file as well
$ mlflow server \
--backend-store-uri sqlite:///mlflow.db \
--default-artifact-root s3://mlflow \
--host 0.0.0.0
Minio
$ export MINIO_ROOT_USER=minioadmin
$ export MINIO_ROOT_PASSWORD=minioadmin
$ mkdir minio_data
$ minio server minio_data --console-address ":9001"
# API: http://192.168.29.103:9000 http://10.119.80.13:9000 http://127.0.0.1:9000
# RootUser: minioadmin
# RootPass: minioadmin
# Console: http://192.168.29.103:9001 http://10.119.80.13:9001 http://127.0.0.1:9001
# RootUser: minioadmin
# RootPass: minioadmin
# Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
# $ mc alias set myminio http://192.168.29.103:9000 minioadmin minioadmin
# Documentation: https://docs.min.io
Go to http://127.0.0.1:9001/buckets/ and create a bucket called mlflow.
Dagster
$ poetry shell
$ dagit -f mlops/pipeline.py
ElasticAPM
$ docker-compose -f docker-compose-monitoring.yaml up
FastAPI
$ poetry shell
$ export PYTHONPATH=.
$ python mlops/app/application.py
TODO
- Setup with
docker-compose. - Load testing.
- Test cases.
- CI/CD pipeline.
- Drift detection.
