Prevent negative loss values when sigma is very small, due to numerical instability.

Setup the config for a fast light model to train quickly - can do some signs of spelling and character distribution, good signal that it's learned something, but not practically useful as a model

Resume from mid-epoch step properly

Update README instructions around resuming training

Add infill masking to the model training and inference pipeline

Add gradio app

Only maintain most recent checkpoint - do not fill up disk with models

Make bitsandbytes install by default

Set up the dataset correctly when loading the full dataset

Save checkpoints more frequently than every epoch

Update config to train a larger model

Update gradient checkpointing to match new huggingface api

Add 8-bit optimizers

Add gradient accumulation to simulate larger batches with less memory

Support gradient checkpointing