An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

Ongoing research training transformer language models at scale, including: BERT & GPT-2

No description available

Built upon Megatron-Deepspeed and HuggingFace Trainer, EasyLLM has reorganized the code logic with a focus on usability. While enhancing usability, it also ensures training efficiency.

Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];

LLM checkpointing for DeepSpeed/Megatron

No description available

GLM-Pretrain in Megatron-Deepspeed for DCU

Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP

No description available

Execute Megatron-DeepSpeed using Slurm for multi-nodes distributed training

Turing Tech Blog Repository

microsoft/Megatron-DeepSpeed のフォークです。

Fork of Megatron-DeepSpeed with VIT bug fixes and model parallelisms (TP, TP-SP, Ulysses, etc) enabled for VIT. Pipeline Parallelism is not yet enabled.

GPU Memory Calculator for LLM Training - Calculate GPU memory requirements for training Large Language Models with support for multiple training engines including PyTorch DDP, DeepSpeed ZeRO, Megatron-LM, and FSDP.

Build a llama fine-tuning script from scratch using PyTorch and transformers API. It needs to support 4 optional features: gradient checkpointing, mixed precision, data parallelism, tensor parallelism. Do not use ColossalAI/Megatron/DeepSpeed frameworks, you can refer to the code.

Megatron-Deepspeed benchmark on LUMI

環境構築方法の詳細は以下のLinkから

Checkpoint structure with Deepspeed and Megatron-LM

No description available

No description available

支持中文版的大模型

Ongoing research training transformer language models at scale, including: BERT & GPT-2

No description available

No description available

No description available

No description available

This repository is based on megatron-deepspeed, incorporating the block coordinate descent method for training large-scale models.

A (WIP) lightweight implementation of DeepSpeed/Megatron-ML style 3D parallelism, along with some models and helpful utilities.

LLM Pretraining Framework (100B+ Params): Megatron-LM + DeepSpeed + FSDP. Open-source, HPC-ready system with tiny GPT simulation, distributed training, tokenizer tools, dataset pipelines, and deployment scripts for Slurm, AWS, Azure, and Docker.