DeepSpeed utilities
Utilities for DeepSpeed
class accelerate.DeepSpeedPlugin
( hf_ds_config: typing.Any = Nonegradient_accumulation_steps: int = Nonegradient_clipping: float = Nonezero_stage: int = Noneis_train_batch_min: str = Trueoffload_optimizer_device: bool = Noneoffload_param_device: bool = Noneoffload_optimizer_nvme_path: str = Noneoffload_param_nvme_path: str = Nonezero3_init_flag: bool = Nonezero3_save_16bit_model: bool = None )
This plugin is used to integrate DeepSpeed.
deepspeed_config_process
( prefix = ''mismatches = Noneconfig = Nonemust_match = True**kwargs )
Process the DeepSpeed config with the values from the kwargs.
class accelerate.utils.DummyOptim
( paramslr = 0.001weight_decay = 0**kwargs )
Parameters
lr (float) — Learning rate.
params (iterable) — iterable of parameters to optimize or dicts defining parameter groups
weight_decay (float) — Weight decay. **kwargs — Other arguments.
Dummy optimizer presents model parameters or param groups, this is primarily used to follow conventional training loop when optimizer config is specified in the deepspeed config file.
class accelerate.utils.DummyScheduler
( optimizertotal_num_steps = Nonewarmup_num_steps = 0lr_scheduler_callable = None**kwargs )
Parameters
optimizer (
torch.optim.optimizer.Optimizer) — The optimizer to wrap.total_num_steps (int, optional) — Total number of steps.
warmup_num_steps (int, optional) — Number of steps for warmup.
lr_scheduler_callable (callable, optional) — A callable function that creates an LR Scheduler. It accepts only one argument
optimizer. **kwargs — Other arguments.
Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file.
class accelerate.utils.DeepSpeedEngineWrapper
( engine )
Parameters
engine (deepspeed.runtime.engine.DeepSpeedEngine) — deepspeed engine to wrap
Internal wrapper for deepspeed.runtime.engine.DeepSpeedEngine. This is used to follow conventional training loop.
class accelerate.utils.DeepSpeedOptimizerWrapper
( optimizer )
Parameters
optimizer (
torch.optim.optimizer.Optimizer) — The optimizer to wrap.
Internal wrapper around a deepspeed optimizer.
class accelerate.utils.DeepSpeedSchedulerWrapper
( scheduleroptimizers )
Parameters
scheduler (
torch.optim.lr_scheduler.LambdaLR) — The scheduler to wrap.optimizers (one or a list of
torch.optim.Optimizer) —
Internal wrapper around a deepspeed scheduler.
Last updated