Checkpoint Deep Learning Models or Machine Learning Models Machine learning training is typically a long-time intensive process. It’s not uncommon to see training jobs running over multiple hours or even multiple days. If these long-running training jobs stop for any reason such as a power failure, or oils fault, or any other unforeseen error, then you’ll have to start the […]
Machine learning training is typically a long-time intensive process. It’s not uncommon to see training jobs running over multiple hours or even multiple days. If these long-running training jobs stop for any reason such as a power failure, or oils fault, or any other unforeseen error, then you’ll have to start the training job from the very beginning. This leads to lost productivity. Even if you don’t encounter any unforeseen errors, there might be situations where you want to start a training job from a known state, to try out new experiments. In these situations, you will use machine learning checkpointing.
Checkpointing is a way to save the current state of a running training job so the training job, if it is stopped, can be resumed from a known state. Checkpoints are basically snapshots of model in training and include details like model architecture, which allows you to recreate the model training once it stopped, also includes model weights that have been learned in the training process so far.
