Overview
This repo is a gentle introduction to using the Lightning.ai ecosystem, placing an emphasis on an industry-like system architecture and tooling that culminates with a simple UI deployed to the Lightning.ai platform.
Using the template
The intent is that users create a new repo from the template in GitHub’s web interface rather than cloning or forking.
Lightning App
The Lightning App is served from the app.py file; and the Plotly Dash UI is written in the components/ui.py file. Additional styling for the UI is found in assets/styles.css.
If using this template, the app can be viewed locally after creating the Python environment using the conda environment.yml or pip requirements.txt by executing the following in terminal:
cd {{ the path to the cloned repo }}
{{ activate the Python environment }}
pip install -e .
lightning run app app.py
Note that pip install -e . is required to install an editable version of the lightning_agents module to the environment, and only needs to be ran once.
The Deep Learning System
The LightningModule and Trainer can be found in lightning_agents/agents/linear. Two example trainers exist – one which is static in the sense that the trainer flags are hardcoded, and another – hydra that supports passing arguments from the command line; the hydra trainer is an example of what is going on behind the scenes of Lightning CLI and serves as a basic example of Meta/Facebook’s open source tool, Hydra.
The data is handled in lightning_agents/pipeline. The focus of the pipeline is to create a LightningDataModule in datamodule.py to pass to either of the trainers.
Users should note that no pre-processing transforms aside from converting the image data to torch.Tensors have been implemented; and the dimensions of the Encoder and Decoder layers are hardcoded. Meaning – there is work to be done to properly pre-process the dataset and to enable hyperparameter optimization with Optuna or a Sweeps App with Lightning.
Lastly, no metrics aside from the objective function (loss) have been calculated. Users are encouraged to research on their own which metrics provided in torchmetrics might be appropriate for image processing problems.
If using this template to train the system locally, run the following in terminal (assumes use of static_trainer.py):
cd {{ the path to the cloned repo }}
{{ activate the Python environment }}
pip install -e .
python lightning_agents/agent/linear/static_trainer.py
Note that pip install -e . is required to install an editable version of the lightning_agents module to the environment, and only needs to be ran once. If executed while running the app locally, this line can be omitted from the above.
Engineering and Deep Learning
The Lightning team has created a Engineering for Researchers video series that will explain software engineering concepts. The Lightning YouTube channel is also extremely helpful.
Individuals interested in learning about Deep Learning can use the NYU lectures taught by Dr. Alfredo Canziani, which are hosted on his YouTube channel; and make use of his lecture notes hosted on GitHub.
Additional Research and Debugging Tools
The provided conda environment.yml and pip requirements.txt enable use of PyTorch Profiler and TensorBoard for examining code execution and training logs in VS Code’s respective integrations.
Additional Engineering Tools
The repo includes basic examples of tools used by the Lightning team, such as: CI/CD with Deepsource and CircleCI; code formatting, linting, and type checking with Black, flake8, and mypy; unit testing with coverage for CodeCov; and source code documentation with Sphinx.
Though not used by the Lightning team, this repo also enables reproducible work with either Gitpod or GitHub Codespaces; and uses a GitHub CodeQL Action for security analysis.
Tooling not included in this repo that is used by Lighting: Azure Pipelines for testing on OS container images, pre-commit for additional code review, and mergify for handling automated merging of PRs once all other QA checks have passed.
More information can be found on these tools at the lightning-pod wiki.
Getting Help
Please join the Lightning Community Slack if you have questions about this template or general questions about the Lightning.ai ecosystem (PyTorch Lightning, Lightning Framework, Lightning Flash, Lightning Transformers, Lightning Bolts, torchmetrics, and Grid.ai Runs/Sessions/Datastores).
Contributing
There is no need to submit PRs or issues; this template is exactly that, a basic template meant to used by others to improve on or extend.
