Fault Tolerant Experiment Logic

Experiments should be stoppable and resumable leveraging the current model tracking feature.

Completed:

Checkpoint model, optimizer, and RNG state to disk after every completed round
Handle stop signals gracefully so runs shut down cleanly instead of crashing mid-round
Resume a stopped run from its last checkpoint, reusing the original data and validating the config matches
Keep decentralized nodes in sync on resume by restarting from the earliest common completed round
Implement automated testing for experiment fault tolerance cases
Scope out distributed version of logging for latest round

Navigation Menu