> For the complete documentation index, see [llms.txt](https://boinc-ai.gitbook.io/accelerate/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://boinc-ai.gitbook.io/accelerate/reference/megatron-lm-utilities.md).

# Megatron-LM Utilities

## Utilities for Megatron-LM

#### class accelerate.utils.MegatronLMPlugin

[\<source>](https://github.com/huggingface/accelerate/blob/v0.24.0/src/accelerate/utils/dataclasses.py#L1018)

( tp\_degree: int = Nonepp\_degree: int = Nonenum\_micro\_batches: int = Nonegradient\_clipping: float = Nonesequence\_parallelism: bool = Nonerecompute\_activation: bool = Noneuse\_distributed\_optimizer: bool = Nonepipeline\_model\_parallel\_split\_rank: int = Nonenum\_layers\_per\_virtual\_pipeline\_stage: int = Noneis\_train\_batch\_min: str = Truetrain\_iters: int = Nonetrain\_samples: int = Noneweight\_decay\_incr\_style: str = 'constant'start\_weight\_decay: float = Noneend\_weight\_decay: float = Nonelr\_decay\_style: str = 'linear'lr\_decay\_iters: int = Nonelr\_decay\_samples: int = Nonelr\_warmup\_iters: int = Nonelr\_warmup\_samples: int = Nonelr\_warmup\_fraction: float = Nonemin\_lr: float = 0consumed\_samples: typing.List\[int] = Noneno\_wd\_decay\_cond: typing.Optional\[typing.Callable] = Nonescale\_lr\_cond: typing.Optional\[typing.Callable] = Nonelr\_mult: float = 1.0megatron\_dataset\_flag: bool = Falseseq\_length: int = Noneencoder\_seq\_length: int = Nonedecoder\_seq\_length: int = Nonetensorboard\_dir: str = Noneset\_all\_logging\_options: bool = Falseeval\_iters: int = 100eval\_interval: int = 1000return\_logits: bool = Falsecustom\_train\_step\_class: typing.Optional\[typing.Any] = Nonecustom\_train\_step\_kwargs: typing.Union\[typing.Dict\[str, typing.Any], NoneType] = Nonecustom\_model\_provider\_function: typing.Optional\[typing.Callable] = Nonecustom\_prepare\_model\_function: typing.Optional\[typing.Callable] = Noneother\_megatron\_args: typing.Union\[typing.Dict\[str, typing.Any], NoneType] = None )

Plugin for Megatron-LM to enable tensor, pipeline, sequence and data parallelism. Also to enable selective activation recomputation and optimized fused kernels.

#### class accelerate.utils.MegatronLMDummyScheduler

[\<source>](https://github.com/huggingface/accelerate/blob/v0.24.0/src/accelerate/utils/megatron_lm.py#L414)

( optimizertotal\_num\_steps = Nonewarmup\_num\_steps = 0\*\*kwargs )

Parameters

* **optimizer** (`torch.optim.optimizer.Optimizer`) — The optimizer to wrap.
* **total\_num\_steps** (int) — Total number of steps.
* **warmup\_num\_steps** (int) — Number of steps for warmup. \*\*kwargs — Other arguments.

Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file.

#### class accelerate.utils.MegatronLMDummyDataLoader

[\<source>](https://github.com/huggingface/accelerate/blob/v0.24.0/src/accelerate/utils/megatron_lm.py#L144)

( \*\*dataset\_kwargs )

Dummy dataloader presents model parameters or param groups, this is primarily used to follow conventional training

#### class accelerate.utils.AbstractTrainStep

[\<source>](https://github.com/huggingface/accelerate/blob/v0.24.0/src/accelerate/utils/megatron_lm.py#L451)

( name )

Abstract class for batching, forward pass and loss handler.

#### class accelerate.utils.GPTTrainStep

[\<source>](https://github.com/huggingface/accelerate/blob/v0.24.0/src/accelerate/utils/megatron_lm.py#L597)

( args )

Parameters

* **args** (`argparse.Namespace`) — Megatron-LM arguments.

GPT train step class.

#### class accelerate.utils.BertTrainStep

[\<source>](https://github.com/huggingface/accelerate/blob/v0.24.0/src/accelerate/utils/megatron_lm.py#L468)

( args )

Parameters

* **args** (`argparse.Namespace`) — Megatron-LM arguments.

Bert train step class.

#### class accelerate.utils.T5TrainStep

[\<source>](https://github.com/huggingface/accelerate/blob/v0.24.0/src/accelerate/utils/megatron_lm.py#L703)

( args )

Parameters

* **args** (`argparse.Namespace`) — Megatron-LM arguments.

T5 train step class.

**accelerate.utils.avg\_losses\_across\_data\_parallel\_group**

[\<source>](https://github.com/huggingface/accelerate/blob/v0.24.0/src/accelerate/utils/megatron_lm.py#L1408)

( losses )

Parameters

* **losses** (List\[Tensor]) — List of losses to average across data parallel group.

Average losses across data parallel group.