A PyTorch implementation of discrete diffusion for character-level text generation, based on Score Entropy Discrete Diffusion (SEDD).
Stars
0
Forks
0
Watchers
0
Open Issues
0
Overall repository health assessment
No package.json found
This might not be a Node.js project
17
commits
Prevent negative loss values when sigma is very small, due to numerical instability.
5f85fa0View on GitHubSetup the config for a fast light model to train quickly - can do some signs of spelling and character distribution, good signal that it's learned something, but not practically useful as a model
837ff00View on GitHubAdd infill masking to the model training and inference pipeline
1f14117View on GitHubOnly maintain most recent checkpoint - do not fill up disk with models
70682e4View on GitHubSet up the dataset correctly when loading the full dataset
506c2f6View on GitHubUpdate gradient checkpointing to match new huggingface api
f5f358fView on GitHubAdd gradient accumulation to simulate larger batches with less memory
0c01b78View on GitHub