PyTorch Lightning

import lightning.pytorch as pl

CLI

• https://lightning.ai/docs/pytorch/stable/cli/lightning_cli.html
• Highlight
  • Separate configuration from source code
  • Guarantee reproducibility of experiments
  • Mix any model with any dataset
• yaml
  • python main.py fit --print_config > config.yaml
  • python main.py --config config_1.yaml --config config_2.yaml fit
    • config_2 will take higher precedence than config_1
    • To customize arguments by subcommand, pass the config before the subcommand.
  • Variable interpolation: jsonargparse and the omegaconf documentations.
• code

Optuna(hyperparameter optimization)

Shorts: fewer updates recently

Optuna - A hyperparameter optimization framework

# <https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html>
print(f"Sampler is {study.sampler.__class__.__name__}")
# <https://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.TorchDistributedTrial.html#optuna.integration.TorchDistributedTrial>

• Use optuna with LightningCLI

Hydra

https://github.com/facebookresearch/hydra

W&B

https://wandb.ai/

MLflow

https://github.com/mlflow/mlflow/blob/master/examples/pytorch/AxHyperOptimizationPTL/ax_hpo_iris.py