MLops review

MLops tools review focused on pipeline execution using multiple clusters: slurm, kubernetes, dask…

Working examples so far:

  1. Ploomber pipelines – runs on Slurm, K8s, Airflow, Kubeflow(in progress), no complex DAGs with loops/recursions yet.
    1. Kubernetes – example runs pipeline on local k8s using argo
    2. Slurm – example transforms pipeline using a script into batch jobs

    For example, since we want the tasks to run in the conda environment we created, edit the so it looks like this:

    #SBATCH --job-name={{name}}
    #SBATCH --output=result.out
    # activate myenv
    conda activate myenv
    srun {{command}}
  1. Nextflow pipelines – runs very well on K8s, Slurm, Dask and many others, not focused on ML and hence not ML friendly. No complex DAGs with loops.

  2. Kubeflow pipelines – runs on K8s, supports complex DAGs with loops and recursions, running on slurm would require additional development

    working-examples include:

    1. – how to define recursive pipeline using kubeflow
    2. – how to pass pandas dataframe between pipeline components.
    3. others…


View Github