# DeepSpeed utilities

## DeepSpeedPlugin

## get_active_deepspeed_plugin[[accelerate.utils.get_active_deepspeed_plugin]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>accelerate.utils.get_active_deepspeed_plugin</name><anchor>accelerate.utils.get_active_deepspeed_plugin</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/deepspeed.py#L100</source><parameters>[{"name": "state", "val": ""}]</parameters><raises>- ``ValueError`` -- If DeepSpeed was not enabled and this function is called.</raises><raisederrors>``ValueError``</raisederrors></docstring>

Returns the currently active DeepSpeedPlugin.






</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.DeepSpeedPlugin</name><anchor>accelerate.DeepSpeedPlugin</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/dataclasses.py#L1086</source><parameters>[{"name": "hf_ds_config", "val": ": typing.Any = None"}, {"name": "gradient_accumulation_steps", "val": ": int = None"}, {"name": "gradient_clipping", "val": ": float = None"}, {"name": "zero_stage", "val": ": int = None"}, {"name": "is_train_batch_min", "val": ": bool = True"}, {"name": "offload_optimizer_device", "val": ": str = None"}, {"name": "offload_param_device", "val": ": str = None"}, {"name": "offload_optimizer_nvme_path", "val": ": str = None"}, {"name": "offload_param_nvme_path", "val": ": str = None"}, {"name": "zero3_init_flag", "val": ": bool = None"}, {"name": "zero3_save_16bit_model", "val": ": bool = None"}, {"name": "transformer_moe_cls_names", "val": ": str = None"}, {"name": "enable_msamp", "val": ": bool = None"}, {"name": "msamp_opt_level", "val": ": typing.Optional[typing.Literal['O1', 'O2']] = None"}]</parameters><paramsdesc>- **hf_ds_config** (`Any`, defaults to `None`) --
  Path to DeepSpeed config file or dict or an object of class `accelerate.utils.deepspeed.HfDeepSpeedConfig`.
- **gradient_accumulation_steps** (`int`, defaults to `None`) --
  Number of steps to accumulate gradients before updating optimizer states. If not set, will use the value
  from the `Accelerator` directly.
- **gradient_clipping** (`float`, defaults to `None`) --
  Enable gradient clipping with value.
- **zero_stage** (`int`, defaults to `None`) --
  Possible options are 0, 1, 2, 3. Default will be taken from environment variable.
- **is_train_batch_min** (`bool`, defaults to `True`) --
  If both train & eval dataloaders are specified, this will decide the `train_batch_size`.
- **offload_optimizer_device** (`str`, defaults to `None`) --
  Possible options are none|cpu|nvme. Only applicable with ZeRO Stages 2 and 3.
- **offload_param_device** (`str`, defaults to `None`) --
  Possible options are none|cpu|nvme. Only applicable with ZeRO Stage 3.
- **offload_optimizer_nvme_path** (`str`, defaults to `None`) --
  Possible options are /nvme|/local_nvme. Only applicable with ZeRO Stage 3.
- **offload_param_nvme_path** (`str`, defaults to `None`) --
  Possible options are /nvme|/local_nvme. Only applicable with ZeRO Stage 3.
- **zero3_init_flag** (`bool`, defaults to `None`) --
  Flag to indicate whether to save 16-bit model. Only applicable with ZeRO Stage-3.
- **zero3_save_16bit_model** (`bool`, defaults to `None`) --
  Flag to indicate whether to save 16-bit model. Only applicable with ZeRO Stage-3.
- **transformer_moe_cls_names** (`str`, defaults to `None`) --
  Comma-separated list of Transformers MoE layer class names (case-sensitive). For example,
  `MixtralSparseMoeBlock`, `Qwen2MoeSparseMoeBlock`, `JetMoEAttention`, `JetMoEBlock`, etc.
- **enable_msamp** (`bool`, defaults to `None`) --
  Flag to indicate whether to enable MS-AMP backend for FP8 training.
- **msasmp_opt_level** (`Optional[Literal["O1", "O2"]]`, defaults to `None`) --
  Optimization level for MS-AMP (defaults to 'O1'). Only applicable if `enable_msamp` is True. Should be one
  of ['O1' or 'O2'].</paramsdesc><paramgroups>0</paramgroups></docstring>

This plugin is used to integrate DeepSpeed.





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>deepspeed_config_process</name><anchor>accelerate.DeepSpeedPlugin.deepspeed_config_process</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/dataclasses.py#L1355</source><parameters>[{"name": "prefix", "val": " = ''"}, {"name": "mismatches", "val": " = None"}, {"name": "config", "val": " = None"}, {"name": "must_match", "val": " = True"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>
Process the DeepSpeed config with the values from the kwargs.

</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>select</name><anchor>accelerate.DeepSpeedPlugin.select</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/dataclasses.py#L1506</source><parameters>[{"name": "_from_accelerator_state", "val": ": bool = False"}]</parameters></docstring>

Sets the HfDeepSpeedWeakref to use the current deepspeed plugin configuration


</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.utils.DummyScheduler</name><anchor>accelerate.utils.DummyScheduler</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/deepspeed.py#L362</source><parameters>[{"name": "optimizer", "val": ""}, {"name": "total_num_steps", "val": " = None"}, {"name": "warmup_num_steps", "val": " = 0"}, {"name": "lr_scheduler_callable", "val": " = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **optimizer** (`torch.optim.optimizer.Optimizer`) --
  The optimizer to wrap.
- **total_num_steps** (int, *optional*) --
  Total number of steps.
- **warmup_num_steps** (int, *optional*) --
  Number of steps for warmup.
- **lr_scheduler_callable** (callable, *optional*) --
  A callable function that creates an LR Scheduler. It accepts only one argument `optimizer`.
- ****kwargs** (additional keyword arguments, *optional*) --
  Other arguments.</paramsdesc><paramgroups>0</paramgroups></docstring>

Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training
loop when scheduler config is specified in the deepspeed config file.




</div>

## DeepSpeedEnginerWrapper[[accelerate.utils.DeepSpeedEngineWrapper]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.utils.DeepSpeedEngineWrapper</name><anchor>accelerate.utils.DeepSpeedEngineWrapper</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/deepspeed.py#L253</source><parameters>[{"name": "engine", "val": ""}]</parameters><paramsdesc>- **engine** (deepspeed.runtime.engine.DeepSpeedEngine) -- deepspeed engine to wrap</paramsdesc><paramgroups>0</paramgroups></docstring>

Internal wrapper for deepspeed.runtime.engine.DeepSpeedEngine. This is used to follow conventional training loop.





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_global_grad_norm</name><anchor>accelerate.utils.DeepSpeedEngineWrapper.get_global_grad_norm</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/deepspeed.py#L286</source><parameters>[]</parameters></docstring>
Get the global gradient norm from DeepSpeed engine.

</div></div>

## DeepSpeedOptimizerWrapper[[accelerate.utils.DeepSpeedOptimizerWrapper]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.utils.DeepSpeedOptimizerWrapper</name><anchor>accelerate.utils.DeepSpeedOptimizerWrapper</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/deepspeed.py#L295</source><parameters>[{"name": "optimizer", "val": ""}]</parameters><paramsdesc>- **optimizer** (`torch.optim.optimizer.Optimizer`) --
  The optimizer to wrap.</paramsdesc><paramgroups>0</paramgroups></docstring>

Internal wrapper around a deepspeed optimizer.




</div>

## DeepSpeedSchedulerWrapper[[accelerate.utils.DeepSpeedSchedulerWrapper]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.utils.DeepSpeedSchedulerWrapper</name><anchor>accelerate.utils.DeepSpeedSchedulerWrapper</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/deepspeed.py#L322</source><parameters>[{"name": "scheduler", "val": ""}, {"name": "optimizers", "val": ""}]</parameters><paramsdesc>- **scheduler** (`torch.optim.lr_scheduler.LambdaLR`) --
  The scheduler to wrap.
- **optimizers** (one or a list of `torch.optim.Optimizer`) --</paramsdesc><paramgroups>0</paramgroups></docstring>

Internal wrapper around a deepspeed scheduler.




</div>

## DummyOptim[[accelerate.utils.DummyOptim]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.utils.DummyOptim</name><anchor>accelerate.utils.DummyOptim</anchor><source>https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/deepspeed.py#L339</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "weight_decay", "val": " = 0"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **lr** (float) --
  Learning rate.
- **params** (iterable) -- iterable of parameters to optimize or dicts defining
  parameter groups
- **weight_decay** (float) --
  Weight decay.
- ****kwargs** (additional keyword arguments, *optional*) --
  Other arguments.</paramsdesc><paramgroups>0</paramgroups></docstring>

Dummy optimizer presents model parameters or param groups, this is primarily used to follow conventional training
loop when optimizer config is specified in the deepspeed config file.




</div>

## DummyScheduler

<EditOnGithub source="https://github.com/huggingface/accelerate/blob/main/docs/source/package_reference/deepspeed.md" />