# TODO: `TrainingArguments` users rely on it being fully mutable. In the future see if we can narrow this to a few keys: https://github.com/huggingface/transformers/pull/25903
@dataclass
class TrainingArguments:
"""
TrainingArguments is the subset of the arguments we use in our example scripts **which relate to the training loop
itself**.
Using [`HfArgumentParser`] we can turn this class into
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
command line.
Parameters:
output_dir (`str`):
The output directory where the model predictions and checkpoints will be written.
overwrite_output_dir (`bool`, *optional*, defaults to `False`):
If `True`, overwrite the content of the output directory. Use this to continue training if `output_dir`
points to a checkpoint directory.
do_train (`bool`, *optional*, defaults to `False`):
Whether to run training or not. This argument is not directly used by [`Trainer`], it's intended to be used
by your training/evaluation scripts instead. See the [example
scripts](https://github.com/huggingface/transformers/tree/main/examples) for more details.
do_eval (`bool`, *optional*):
Whether to run evaluation on the validation set or not. Will be set to `True` if `evaluation_strategy` is
different from `"no"`. This argument is not directly used by [`Trainer`], it's intended to be used by your
training/evaluation scripts instead. See the [example
scripts](https://github.com/huggingface/transformers/tree/main/examples) for more details.
do_predict (`bool`, *optional*, defaults to `False`):
Whether to run predictions on the test set or not. This argument is not directly used by [`Trainer`], it's
intended to be used by your training/evaluation scripts instead. See the [example
scripts](https://github.com/huggingface/transformers/tree/main/examples) for more details.
evaluation_strategy (`str` or [`~trainer_utils.IntervalStrategy`], *optional*, defaults to `"no"`):
The evaluation strategy to adopt during training. Possible values are:
- `"no"`: No evaluation is done during training.
- `"steps"`: Evaluation is done (and logged) every `eval_steps`.
- `"epoch"`: Evaluation is done at the end of each epoch.
prediction_loss_only (`bool`, *optional*, defaults to `False`):
When performing evaluation and generating predictions, only returns the loss.
per_device_train_batch_size (`int`, *optional*, defaults to 8):
The batch size per GPU/XPU/TPU/MPS/NPU core/CPU for training.
per_device_eval_batch_size (`int`, *optional*, defaults to 8):
The batch size per GPU/XPU/TPU/MPS/NPU core/CPU for evaluation.
gradient_accumulation_steps (`int`, *optional*, defaults to 1):
Number of updates steps to accumulate the gradients for, before performing a backward/update pass.
<Tip warning={true}>
When using gradient accumulation, one step is counted as one step with backward pass. Therefore, logging,
evaluation, save will be conducted every `gradient_accumulation_steps * xxx_step` training examples.
</Tip>
eval_accumulation_steps (`int`, *optional*):
Number of predictions steps to accumulate the output tensors for, before moving the results to the CPU. If
left unset, the whole predictions are accumulated on GPU/NPU/TPU before being moved to the CPU (faster but
requires more memory).
eval_delay (`float`, *optional*):
Number of epochs or steps to wait for before the first evaluation can be performed, depending on the
evaluation_strategy.
learning_rate (`float`, *optional*, defaults to 5e-5):
The initial learning rate for [`AdamW`] optimizer.
weight_decay (`float`, *optional*, defaults to 0):
The weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in [`AdamW`]
optimizer.
adam_beta1 (`float`, *optional*, defaults to 0.9):
The beta1 hyperparameter for the [`AdamW`] optimizer.
adam_beta2 (`float`, *optional*, defaults to 0.999):
The beta2 hyperparameter for the [`AdamW`] optimizer.
adam_epsilon (`float`, *optional*, defaults to 1e-8):
The epsilon hyperparameter for the [`AdamW`] optimizer.
max_grad_norm (`float`, *optional*, defaults to 1.0):
Maximum gradient norm (for gradient clipping).
num_train_epochs(`float`, *optional*, defaults to 3.0):
Total number of training epochs to perform (if not an integer, will perform the decimal part percents of
the last epoch before stopping training).
max_steps (`int`, *optional*, defaults to -1):
If set to a positive number, the total number of training steps to perform. Overrides `num_train_epochs`.
For a finite dataset, training is reiterated through the dataset (if all data is exhausted) until
`max_steps` is reached.
lr_scheduler_type (`str` or [`SchedulerType`], *optional*, defaults to `"linear"`):
The scheduler type to use. See the documentation of [`SchedulerType`] for all possible values.
lr_scheduler_kwargs ('dict', *optional*, defaults to {}):
The extra arguments for the lr_scheduler. See the documentation of each scheduler for possible values.
warmup_ratio (`float`, *optional*, defaults to 0.0):
Ratio of total training steps used for a linear warmup from 0 to `learning_rate`.
warmup_steps (`int`, *optional*, defaults to 0):
Number of steps used for a linear warmup from 0 to `learning_rate`. Overrides any effect of `warmup_ratio`.
log_level (`str`, *optional*, defaults to `passive`):
Logger log level to use on the main process. Possible choices are the log levels as strings: 'debug',
'info', 'warning', 'error' and 'critical', plus a 'passive' level which doesn't set anything and keeps the
current log level for the Transformers library (which will be `"warning"` by default).
log_level_replica (`str`, *optional*, defaults to `"warning"`):
Logger log level to use on replicas. Same choices as `log_level`"
log_on_each_node (`bool`, *optional*, defaults to `True`):
In multinode distributed training, whether to log using `log_level` once per node, or only on the main
node.
logging_dir (`str`, *optional*):
[TensorBoard](https://www.tensorflow.org/tensorboard) log directory. Will default to
*output_dir/runs/**CURRENT_DATETIME_HOSTNAME***.
logging_strategy (`str` or [`~trainer_utils.IntervalStrategy`], *optional*, defaults to `"steps"`):
The logging strategy to adopt during training. Possible values are:
- `"no"`: No logging is done during training.
- `"epoch"`: Logging is done at the end of each epoch.
- `"steps"`: Logging is done every `logging_steps`.
logging_first_step (`bool`, *optional*, defaults to `False`):
Whether to log the first `global_step` or not.
logging_steps (`int` or `float`, *optional*, defaults to 500):
Number of update steps between two logs if `logging_strategy="steps"`. Should be an integer or a float in
range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps.
logging_nan_inf_filter (`bool`, *optional*, defaults to `True`):
Whether to filter `nan` and `inf` losses for logging. If set to `True` the loss of every step that is `nan`
or `inf` is filtered and the average loss of the current logging window is taken instead.
<Tip>
`logging_nan_inf_filter` only influences the logging of loss values, it does not change the behavior the
gradient is computed or applied to the model.
</Tip>
save_strategy (`str` or [`~trainer_utils.IntervalStrategy`], *optional*, defaults to `"steps"`):
The checkpoint save strategy to adopt during training. Possible values are:
- `"no"`: No save is done during training.
- `"epoch"`: Save is done at the end of each epoch.
- `"steps"`: Save is done every `save_steps`.
save_steps (`int` or `float`, *optional*, defaults to 500):
Number of updates steps before two checkpoint saves if `save_strategy="steps"`. Should be an integer or a
float in range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps.
save_total_limit (`int`, *optional*):
If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in
`output_dir`. When `load_best_model_at_end` is enabled, the "best" checkpoint according to
`metric_for_best_model` will always be retained in addition to the most recent ones. For example, for
`save_total_limit=5` and `load_best_model_at_end`, the four last checkpoints will always be retained
alongside the best model. When `save_total_limit=1` and `load_best_model_at_end`, it is possible that two
checkpoints are saved: the last one and the best one (if they are different).
save_safetensors (`bool`, *optional*, defaults to `True`):
Use [safetensors](https://huggingface.co/docs/safetensors) saving and loading for state dicts instead of
default `torch.load` and `torch.save`.
save_on_each_node (`bool`, *optional*, defaults to `False`):
When doing multi-node distributed training, whether to save models and checkpoints on each node, or only on
the main one.
This should not be activated when the different nodes use the same storage as the files will be saved with
the same names for each node.
save_only_model (`bool`, *optional*, defaults to `False`):
When checkpointing, whether to only save the model, or also the optimizer, scheduler & rng state.
Note that when this is true, you won't be able to resume training from checkpoint.
This enables you to save storage by not storing the optimizer, scheduler & rng state.
You can only load the model using `from_pretrained` with this option set to `True`.
use_cpu (`bool`, *optional*, defaults to `False`):
Whether or not to use cpu. If set to False, we will use cuda or mps device if available.
seed (`int`, *optional*, defaults to 42):
Random seed that will be set at the beginning of training. To ensure reproducibility across runs, use the
[`~Trainer.model_init`] function to instantiate the model if it has some randomly initialized parameters.
data_seed (`int`, *optional*):
Random seed to be used with data samplers. If not set, random generators for data sampling will use the
same seed as `seed`. This can be used to ensure reproducibility of data sampling, independent of the model
seed.
jit_mode_eval (`bool`, *optional*, defaults to `False`):
Whether or not to use PyTorch jit trace for inference.
use_ipex (`bool`, *optional*, defaults to `False`):
Use Intel extension for PyTorch when it is available. [IPEX
installation](https://github.com/intel/intel-extension-for-pytorch).
bf16 (`bool`, *optional*, defaults to `False`):
Whether to use bf16 16-bit (mixed) precision training instead of 32-bit training. Requires Ampere or higher
NVIDIA architecture or using CPU (use_cpu) or Ascend NPU. This is an experimental API and it may change.
fp16 (`bool`, *optional*, defaults to `False`):
Whether to use fp16 16-bit (mixed) precision training instead of 32-bit training.
fp16_opt_level (`str`, *optional*, defaults to 'O1'):
For `fp16` training, Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. See details on
the [Apex documentation](https://nvidia.github.io/apex/amp).
fp16_backend (`str`, *optional*, defaults to `"auto"`):
This argument is deprecated. Use `half_precision_backend` instead.
half_precision_backend (`str`, *optional*, defaults to `"auto"`):
The backend to use for mixed precision training. Must be one of `"auto", "apex", "cpu_amp"`. `"auto"` will
use CPU/CUDA AMP or APEX depending on the PyTorch version detected, while the other choices will force the
requested backend.
bf16_full_eval (`bool`, *optional*, defaults to `False`):
Whether to use full bfloat16 evaluation instead of 32-bit. This will be faster and save memory but can harm
metric values. This is an experimental API and it may change.
fp16_full_eval (`bool`, *optional*, defaults to `False`):
Whether to use full float16 evaluation instead of 32-bit. This will be faster and save memory but can harm
metric values.
tf32 (`bool`, *optional*):
Whether to enable the TF32 mode, available in Ampere and newer GPU architectures. The default value depends
on PyTorch's version default of `torch.backends.cuda.matmul.allow_tf32`. For more details please refer to
the [TF32](https://huggingface.co/docs/transformers/performance#tf32) documentation. This is an
experimental API and it may change.
local_rank (`int`, *optional*, defaults to -1):
Rank of the process during distributed training.
ddp_backend (`str`, *optional*):
The backend to use for distributed training. Must be one of `"nccl"`, `"mpi"`, `"ccl"`, `"gloo"`, `"hccl"`.
tpu_num_cores (`int`, *optional*):
When training on TPU, the number of TPU cores (automatically passed by launcher script).
dataloader_drop_last (`bool`, *optional*, defaults to `False`):
Whether to drop the last incomplete batch (if the length of the dataset is not divisible by the batch size)
or not.
eval_steps (`int` or `float`, *optional*):
Number of update steps between two evaluations if `evaluation_strategy="steps"`. Will default to the same
value as `logging_steps` if not set. Should be an integer or a float in range `[0,1)`. If smaller than 1,
will be interpreted as ratio of total training steps.
dataloader_num_workers (`int`, *optional*, defaults to 0):
Number of subprocesses to use for data loading (PyTorch only). 0 means that the data will be loaded in the
main process.
past_index (`int`, *optional*, defaults to -1):
Some models like [TransformerXL](../model_doc/transformerxl) or [XLNet](../model_doc/xlnet) can make use of
the past hidden states for their predictions. If this argument is set to a positive int, the `Trainer` will
use the corresponding output (usually index 2) as the past state and feed it to the model at the next
training step under the keyword argument `mems`.
run_name (`str`, *optional*):
A descriptor for the run. Typically used for [wandb](https://www.wandb.com/) and
[mlflow](https://www.mlflow.org/) logging.
disable_tqdm (`bool`, *optional*):
Whether or not to disable the tqdm progress bars and table of metrics produced by
[`~notebook.NotebookTrainingTracker`] in Jupyter Notebooks. Will default to `True` if the logging level is
set to warn or lower (default), `False` otherwise.
remove_unused_columns (`bool`, *optional*, defaults to `True`):
Whether or not to automatically remove the columns unused by the model forward method.
label_names (`List[str]`, *optional*):
The list of keys in your dictionary of inputs that correspond to the labels.
Will eventually default to the list of argument names accepted by the model that contain the word "label",
except if the model used is one of the `XxxForQuestionAnswering` in which case it will also include the
`["start_positions", "end_positions"]` keys.
load_best_model_at_end (`bool`, *optional*, defaults to `False`):
Whether or not to load the best model found during training at the end of training. When this option is
enabled, the best checkpoint will always be saved. See
[`save_total_limit`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.save_total_limit)
for more.
<Tip>
When set to `True`, the parameters `save_strategy` needs to be the same as `evaluation_strategy`, and in
the case it is "steps", `save_steps` must be a round multiple of `eval_steps`.
</Tip>
metric_for_best_model (`str`, *optional*):
Use in conjunction with `load_best_model_at_end` to specify the metric to use to compare two different
models. Must be the name of a metric returned by the evaluation with or without the prefix `"eval_"`. Will
default to `"loss"` if unspecified and `load_best_model_at_end=True` (to use the evaluation loss).
If you set this value, `greater_is_better` will default to `True`. Don't forget to set it to `False` if
your metric is better when lower.
greater_is_better (`bool`, *optional*):
Use in conjunction with `load_best_model_at_end` and `metric_for_best_model` to specify if better models
should have a greater metric or not. Will default to:
- `True` if `metric_for_best_model` is set to a value that isn't `"loss"` or `"eval_loss"`.
- `False` if `metric_for_best_model` is not set, or set to `"loss"` or `"eval_loss"`.
ignore_data_skip (`bool`, *optional*, defaults to `False`):
When resuming training, whether or not to skip the epochs and batches to get the data loading at the same
stage as in the previous training. If set to `True`, the training will begin faster (as that skipping step
can take a long time) but will not yield the same results as the interrupted training would have.
fsdp (`bool`, `str` or list of [`~trainer_utils.FSDPOption`], *optional*, defaults to `''`):
Use PyTorch Distributed Parallel Training (in distributed training only).
A list of options along the following:
- `"full_shard"`: Shard parameters, gradients and optimizer states.
- `"shard_grad_op"`: Shard optimizer states and gradients.
- `"hybrid_shard"`: Apply `FULL_SHARD` within a node, and replicate parameters across nodes.
- `"hybrid_shard_zero2"`: Apply `SHARD_GRAD_OP` within a node, and replicate parameters across nodes.
- `"offload"`: Offload parameters and gradients to CPUs (only compatible with `"full_shard"` and
`"shard_grad_op"`).
- `"auto_wrap"`: Automatically recursively wrap layers with FSDP using `default_auto_wrap_policy`.
fsdp_config (`str` or `dict`, *optional*):
Config to be used with fsdp (Pytorch Distributed Parallel Training). The value is either a location of
fsdp json config file (e.g., `fsdp_config.json`) or an already loaded json file as `dict`.
A List of config and its options:
- min_num_params (`int`, *optional*, defaults to `0`):
FSDP's minimum number of parameters for Default Auto Wrapping. (useful only when `fsdp` field is
passed).
- transformer_layer_cls_to_wrap (`List[str]`, *optional*):
List of transformer layer class names (case-sensitive) to wrap, e.g, `BertLayer`, `GPTJBlock`,
`T5Block` .... (useful only when `fsdp` flag is passed).
- backward_prefetch (`str`, *optional*)
FSDP's backward prefetch mode. Controls when to prefetch next set of parameters (useful only when
`fsdp` field is passed).
A list of options along the following:
- `"backward_pre"` : Prefetches the next set of parameters before the current set of parameter's
gradient
computation.
- `"backward_post"` : This prefetches the next set of parameters after the current set of
parameter’s
gradient computation.
- forward_prefetch (`bool`, *optional*, defaults to `False`)
FSDP's forward prefetch mode (useful only when `fsdp` field is passed).
If `"True"`, then FSDP explicitly prefetches the next upcoming all-gather while executing in the
forward pass.
- limit_all_gathers (`bool`, *optional*, defaults to `False`)
FSDP's limit_all_gathers (useful only when `fsdp` field is passed).
If `"True"`, FSDP explicitly synchronizes the CPU thread to prevent too many in-flight
all-gathers.
- use_orig_params (`bool`, *optional*, defaults to `True`)
If `"True"`, allows non-uniform `requires_grad` during init, which means support for interspersed
frozen and trainable paramteres. Useful in cases such as parameter-efficient fine-tuning. Please
refer this
[blog](https://dev-discuss.pytorch.org/t/rethinking-pytorch-fully-sharded-data-parallel-fsdp-from-first-principles/1019
- sync_module_states (`bool`, *optional*, defaults to `True`)
If `"True"`, each individually wrapped FSDP unit will broadcast module parameters from rank 0 to
ensure they are the same across all ranks after initialization
- activation_checkpointing (`bool`, *optional*, defaults to `False`):
If `"True"`, activation checkpointing is a technique to reduce memory usage by clearing activations of
certain layers and recomputing them during a backward pass. Effectively, this trades extra
computation time for reduced memory usage.
- xla (`bool`, *optional*, defaults to `False`):
Whether to use PyTorch/XLA Fully Sharded Data Parallel Training. This is an experimental feature
and its API may evolve in the future.
- xla_fsdp_settings (`dict`, *optional*)
The value is a dictionary which stores the XLA FSDP wrapping parameters.
For a complete list of options, please see [here](
https://github.com/pytorch/xla/blob/master/torch_xla/distributed/fsdp/xla_fully_sharded_data_parallel.py).
- xla_fsdp_grad_ckpt (`bool`, *optional*, defaults to `False`):
Will use gradient checkpointing over each nested XLA FSDP wrapped layer. This setting can only be
used when the xla flag is set to true, and an auto wrapping policy is specified through
fsdp_min_num_params or fsdp_transformer_layer_cls_to_wrap.
deepspeed (`str` or `dict`, *optional*):
Use [Deepspeed](https://github.com/microsoft/deepspeed). This is an experimental feature and its API may
evolve in the future. The value is either the location of DeepSpeed json config file (e.g.,
`ds_config.json`) or an already loaded json file as a `dict`"
<Tip warning={true}>
If enabling any Zero-init, make sure that your model is not initialized until
*after* initializing the `TrainingArguments`, else it will not be applied.
</Tip>
accelerator_config (`str`, `dict`, or `AcceleratorConfig`, *optional*):
Config to be used with the internal `Accelerator` implementation. The value is either a location of
accelerator json config file (e.g., `accelerator_config.json`), an already loaded json file as `dict`,
or an instance of [`~trainer_pt_utils.AcceleratorConfig`].
A list of config and its options:
- split_batches (`bool`, *optional*, defaults to `False`):
Whether or not the accelerator should split the batches yielded by the dataloaders across the devices. If
`True` the actual batch size used will be the same on any kind of distributed processes, but it must be a
round multiple of the `num_processes` you are using. If `False`, actual batch size used will be the one set
in your script multiplied by the number of processes.
- dispatch_batches (`bool`, *optional*):
If set to `True`, the dataloader prepared by the Accelerator is only iterated through on the main process
and then the batches are split and broadcast to each process. Will default to `True` for `DataLoader` whose
underlying dataset is an `IterableDataset`, `False` otherwise.
- even_batches (`bool`, *optional*, defaults to `True`):
If set to `True`, in cases where the total batch size across all processes does not exactly divide the
dataset, samples at the start of the dataset will be duplicated so the batch can be divided equally among
all workers.
- use_seedable_sampler (`bool`, *optional*, defaults to `True`):
Whether or not use a fully seedable random sampler ([`accelerate.data_loader.SeedableRandomSampler`]). Ensures
training results are fully reproducable using a different sampling technique. While seed-to-seed results
may differ, on average the differences are neglible when using multiple different seeds to compare. Should
also be ran with [`~utils.set_seed`] for the best results.
label_smoothing_factor (`float`, *optional*, defaults to 0.0):
The label smoothing factor to use. Zero means no label smoothing, otherwise the underlying onehot-encoded
labels are changed from 0s and 1s to `label_smoothing_factor/num_labels` and `1 - label_smoothing_factor +
label_smoothing_factor/num_labels` respectively.
debug (`str` or list of [`~debug_utils.DebugOption`], *optional*, defaults to `""`):
Enable one or more debug features. This is an experimental feature.
Possible options are:
- `"underflow_overflow"`: detects overflow in model's input/outputs and reports the last frames that led to
the event
- `"tpu_metrics_debug"`: print debug metrics on TPU
The options should be separated by whitespaces.
optim (`str` or [`training_args.OptimizerNames`], *optional*, defaults to `"adamw_torch"`):
The optimizer to use: adamw_hf, adamw_torch, adamw_torch_fused, adamw_apex_fused, adamw_anyprecision or
adafactor.
optim_args (`str`, *optional*):
Optional arguments that are supplied to AnyPrecisionAdamW.
group_by_length (`bool`, *optional*, defaults to `False`):
Whether or not to group together samples of roughly the same length in the training dataset (to minimize
padding applied and be more efficient). Only useful if applying dynamic padding.
length_column_name (`str`, *optional*, defaults to `"length"`):
Column name for precomputed lengths. If the column exists, grouping by length will use these values rather
than computing them on train startup. Ignored unless `group_by_length` is `True` and the dataset is an
instance of `Dataset`.
report_to (`str` or `List[str]`, *optional*, defaults to `"all"`):
The list of integrations to report the results and logs to. Supported platforms are `"azure_ml"`,
`"clearml"`, `"codecarbon"`, `"comet_ml"`, `"dagshub"`, `"dvclive"`, `"flyte"`, `"mlflow"`, `"neptune"`,
`"tensorboard"`, and `"wandb"`. Use `"all"` to report to all integrations installed, `"none"` for no
integrations.
ddp_find_unused_parameters (`bool`, *optional*):
When using distributed training, the value of the flag `find_unused_parameters` passed to
`DistributedDataParallel`. Will default to `False` if gradient checkpointing is used, `True` otherwise.
ddp_bucket_cap_mb (`int`, *optional*):
When using distributed training, the value of the flag `bucket_cap_mb` passed to `DistributedDataParallel`.
ddp_broadcast_buffers (`bool`, *optional*):
When using distributed training, the value of the flag `broadcast_buffers` passed to
`DistributedDataParallel`. Will default to `False` if gradient checkpointing is used, `True` otherwise.
dataloader_pin_memory (`bool`, *optional*, defaults to `True`):
Whether you want to pin memory in data loaders or not. Will default to `True`.
dataloader_persistent_workers (`bool`, *optional*, defaults to `False`):
If True, the data loader will not shut down the worker processes after a dataset has been consumed once.
This allows to maintain the workers Dataset instances alive. Can potentially speed up training, but will
increase RAM usage. Will default to `False`.
dataloader_prefetch_factor (`int`, *optional*):
Number of batches loaded in advance by each worker.
2 means there will be a total of 2 * num_workers batches prefetched across all workers.
skip_memory_metrics (`bool`, *optional*, defaults to `True`):
Whether to skip adding of memory profiler reports to metrics. This is skipped by default because it slows
down the training and evaluation speed.
push_to_hub (`bool`, *optional*, defaults to `False`):
Whether or not to push the model to the Hub every time the model is saved. If this is activated,
`output_dir` will begin a git directory synced with the repo (determined by `hub_model_id`) and the content
will be pushed each time a save is triggered (depending on your `save_strategy`). Calling
[`~Trainer.save_model`] will also trigger a push.
<Tip warning={true}>
If `output_dir` exists, it needs to be a local clone of the repository to which the [`Trainer`] will be
pushed.
</Tip>
resume_from_checkpoint (`str`, *optional*):
The path to a folder with a valid checkpoint for your model. This argument is not directly used by
[`Trainer`], it's intended to be used by your training/evaluation scripts instead. See the [example
scripts](https://github.com/huggingface/transformers/tree/main/examples) for more details.
hub_model_id (`str`, *optional*):
The name of the repository to keep in sync with the local *output_dir*. It can be a simple model ID in
which case the model will be pushed in your namespace. Otherwise it should be the whole repository name,
for instance `"user_name/model"`, which allows you to push to an organization you are a member of with
`"organization_name/model"`. Will default to `user_name/output_dir_name` with *output_dir_name* being the
name of `output_dir`.
Will default to the name of `output_dir`.
hub_strategy (`str` or [`~trainer_utils.HubStrategy`], *optional*, defaults to `"every_save"`):
Defines the scope of what is pushed to the Hub and when. Possible values are:
- `"end"`: push the model, its configuration, the tokenizer (if passed along to the [`Trainer`]) and a
draft of a model card when the [`~Trainer.save_model`] method is called.
- `"every_save"`: push the model, its configuration, the tokenizer (if passed along to the [`Trainer`]) and
a draft of a model card each time there is a model save. The pushes are asynchronous to not block
training, and in case the save are very frequent, a new push is only attempted if the previous one is
finished. A last push is made with the final model at the end of training.
- `"checkpoint"`: like `"every_save"` but the latest checkpoint is also pushed in a subfolder named
last-checkpoint, allowing you to resume training easily with
`trainer.train(resume_from_checkpoint="last-checkpoint")`.
- `"all_checkpoints"`: like `"checkpoint"` but all checkpoints are pushed like they appear in the output
folder (so you will get one checkpoint folder per folder in your final repository)
hub_token (`str`, *optional*):
The token to use to push the model to the Hub. Will default to the token in the cache folder obtained with
`huggingface-cli login`.
hub_private_repo (`bool`, *optional*, defaults to `False`):
If True, the Hub repo will be set to private.
hub_always_push (`bool`, *optional*, defaults to `False`):
Unless this is `True`, the `Trainer` will skip pushing a checkpoint when the previous push is not finished.
gradient_checkpointing (`bool`, *optional*, defaults to `False`):
If True, use gradient checkpointing to save memory at the expense of slower backward pass.
gradient_checkpointing_kwargs (`dict`, *optional*, defaults to `None`):
Key word arguments to be passed to the `gradient_checkpointing_enable` method.
include_inputs_for_metrics (`bool`, *optional*, defaults to `False`):
Whether or not the inputs will be passed to the `compute_metrics` function. This is intended for metrics
that need inputs, predictions and references for scoring calculation in Metric class.
eval_do_concat_batches (`bool`, *optional*, defaults to `True`):
Whether to recursively concat inputs/losses/labels/predictions across batches. If `False`,
will instead store them as lists, with each batch kept separate.
auto_find_batch_size (`bool`, *optional*, defaults to `False`)
Whether to find a batch size that will fit into memory automatically through exponential decay, avoiding
CUDA Out-of-Memory errors. Requires accelerate to be installed (`pip install accelerate`)
full_determinism (`bool`, *optional*, defaults to `False`)
If `True`, [`enable_full_determinism`] is called instead of [`set_seed`] to ensure reproducible results in
distributed training. Important: this will negatively impact the performance, so only use it for debugging.
torchdynamo (`str`, *optional*):
If set, the backend compiler for TorchDynamo. Possible choices are `"eager"`, `"aot_eager"`, `"inductor"`,
`"nvfuser"`, `"aot_nvfuser"`, `"aot_cudagraphs"`, `"ofi"`, `"fx2trt"`, `"onnxrt"` and `"ipex"`.
ray_scope (`str`, *optional*, defaults to `"last"`):
The scope to use when doing hyperparameter search with Ray. By default, `"last"` will be used. Ray will
then use the last checkpoint of all trials, compare those, and select the best one. However, other options
are also available. See the [Ray documentation](
https://docs.ray.io/en/latest/tune/api_docs/analysis.html#ray.tune.ExperimentAnalysis.get_best_trial) for
more options.
ddp_timeout (`int`, *optional*, defaults to 1800):
The timeout for `torch.distributed.init_process_group` calls, used to avoid GPU socket timeouts when
performing slow operations in distributed runnings. Please refer the [PyTorch documentation]
(https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group) for more
information.
use_mps_device (`bool`, *optional*, defaults to `False`):
This argument is deprecated.`mps` device will be used if it is available similar to `cuda` device.
torch_compile (`bool`, *optional*, defaults to `False`):
Whether or not to compile the model using PyTorch 2.0
[`torch.compile`](https://pytorch.org/get-started/pytorch-2.0/).
This will use the best defaults for the [`torch.compile`
API](https://pytorch.org/docs/stable/generated/torch.compile.html?highlight=torch+compile#torch.compile).
You can customize the defaults with the argument `torch_compile_backend` and `torch_compile_mode` but we
don't guarantee any of them will work as the support is progressively rolled in in PyTorch.
This flag and the whole compile API is experimental and subject to change in future releases.
torch_compile_backend (`str`, *optional*):
The backend to use in `torch.compile`. If set to any value, `torch_compile` will be set to `True`.
Refer to the PyTorch doc for possible values and note that they may change across PyTorch versions.
This flag is experimental and subject to change in future releases.
torch_compile_mode (`str`, *optional*):
The mode to use in `torch.compile`. If set to any value, `torch_compile` will be set to `True`.
Refer to the PyTorch doc for possible values and note that they may change across PyTorch versions.
This flag is experimental and subject to change in future releases.
split_batches (`bool`, *optional*):
Whether or not the accelerator should split the batches yielded by the dataloaders across the devices
during distributed training. If
set to `True`, the actual batch size used will be the same on any kind of distributed processes, but it
must be a
round multiple of the number of processes you are using (such as GPUs).
include_tokens_per_second (`bool`, *optional*):
Whether or not to compute the number of tokens per second per device for training speed metrics.
This will iterate over the entire training dataloader once beforehand,
and will slow down the entire process.
include_num_input_tokens_seen (`bool`, *optional*):
Whether or not to track the number of input tokens seen throughout training.
May be slower in distributed training as gather operations must be called.
neftune_noise_alpha (`Optional[float]`):
If not `None`, this will activate NEFTune noise embeddings. This can drastically improve model performance
for instruction fine-tuning. Check out the [original paper](https://arxiv.org/abs/2310.05914) and the
[original code](https://github.com/neelsjain/NEFTune). Support transformers `PreTrainedModel` and also
`PeftModel` from peft.
optim_target_modules (`Union[str, List[str]]`, *optional*):
The target modules to optimize, i.e. the module names that you would like to train, right now this is used only for GaLore algorithm
https://arxiv.org/abs/2403.03507
See: https://github.com/jiaweizzhao/GaLore for more details. You need to make sure to pass a valid GaloRe
optimizer, e.g. one of: "galore_adamw", "galore_adamw_8bit", "galore_adafactor" and make sure that the target modules are `nn.Linear` modules
only.
"""GPT-4-128k
Poe
TrainingArguments 是用于控制训练循环的参数类，它通常用于 Hugging Face Transformers 库中的示例脚本。以下是详细解释每个参数的功能和用途：
基本训练设置
- output_dir (
str): 模型预测和检查点将被写入的输出目录。 - overwrite_output_dir (
bool, 可选, 默认为False): 如果设置为True，将覆盖输出目录的内容。这用于如果output_dir指向一个检查点目录，可以继续训练。 - do_train (
bool, 可选, 默认为False): 是否执行训练。这个参数不是直接由Trainer使用，而是用于控制训练/评估脚本的流程。 - do_eval (
bool, 可选): 是否在验证集上执行评估。如果evaluation_strategy不为"no"，则此设置将自动设为True。 - do_predict (
bool, 可选, 默认为False): 是否在测试集上进行预测。此参数也是用于控制脚本流程，不直接由Trainer使用。
训练性能优化
- per_device_train_batch_size (
int, 可选, 默认为 8): 每个训练设备的批处理大小。 - per_device_eval_batch_size (
int, 可选, 默认为 8): 每个评估设备的批处理大小。 - gradient_accumulation_steps (
int, 可选, 默认为 1): 累积梯度的更新步骤数，然后执行反向/更新传递。
评估和记录
- eval_accumulation_steps (
int, 可选): 在将结果移至 CPU 之前，累积输出张量的预测步骤数。 - evaluation_strategy (
str或IntervalStrategy, 可选, 默认为"no"): 训练期间采用的评估策略，可能的值有"no"（无评估），"steps"（每eval_steps步进行一次评估），"epoch"（每个时代结束时进行评估）。
优化器和调度器
- learning_rate (
float, 可选, 默认为 5e-5): 用于AdamW优化器的初始学习率。 - weight_decay (
float, 可选, 默认为 0): 应用于所有层（除偏置和LayerNorm层外）的权重衰减。 - adam_beta1 (
float, 可选, 默认为 0.9):AdamW优化器的 beta1 超参数。 - adam_beta2 (
float, 可选, 默认为 0.999):AdamW优化器的 beta2 超参数。 - adam_epsilon (
float, 可选, 默认为 1e-8):AdamW优化器的 epsilon 超参数。
其他重要设置
- num_train_epochs (
float, 可选, 默认为 3.0): 要执行的总训练时代数。 - max_steps (
int, 可选, 默认为 -1): 如果设置为正数，则执行的总训练步数，这将覆盖num_train_epochs。 - warmup_steps (
int, 可选, 默认为 0): 从 0 到learning_rate的线性预热步数。
日志和保存
- logging_dir (
str, 可选): TensorBoard 日志目录，默认路径为output_dir/runs/CURRENT_DATETIME_HOSTNAME。 - save_strategy (
str或IntervalStrategy, 可选, 默认为"steps"): 训练期间采用的检查点保存策略。
高级设置
- **use_cpu (
boolTrainingArguments` 是用于控制训练循环的参数类，它通常用于 Hugging Face Transformers 库中的示例脚本。以下是详细解释每个参数的功能和用途：
基本训练设置
- output_dir (
str): 模型预测和检查点将被写入的输出目录。 - overwrite_output_dir (
bool, 可选, 默认为False): 如果设置为True，将覆盖输出目录的内容。这用于如果output_dir指向一个检查点目录，可以继续训练。 - do_train (
bool, 可选, 默认为False): 是否执行训练。这个参数不是直接由Trainer使用，而是用于控制训练/评估脚本的流程。 - do_eval (
bool, 可选): 是否在验证集上执行评估。如果evaluation_strategy不为"no"，则此设置将自动设为True。 - do_predict (
bool, 可选, 默认为False): 是否在测试集上进行预测。此参数也是用于控制脚本流程，不直接由Trainer使用。
训练性能优化
- per_device_train_batch_size (
int, 可选, 默认为 8): 每个训练设备的批处理大小。 - per_device_eval_batch_size (
int, 可选, 默认为 8): 每个评估设备的批处理大小。 - gradient_accumulation_steps (
int, 可选, 默认为 1): 累积梯度的更新步骤数，然后执行反向/更新传递。
评估和记录
- eval_accumulation_steps (
int, 可选): 在将结果移至 CPU 之前，累积输出张量的预测步骤数。 - evaluation_strategy (
str或IntervalStrategy, 可选, 默认为"no"): 训练期间采用的评估策略，可能的值有"no"（无评估），"steps"（每eval_steps步进行一次评估），"epoch"（每个时代结束时进行评估）。
优化器和调度器
- learning_rate (
float, 可选, 默认为 5e-5): 用于AdamW优化器的初始学习率。 - weight_decay (
float, 可选, 默认为 0): 应用于所有层（除偏置和LayerNorm层外）的权重衰减。 - adam_beta1 (
float, 可选, 默认为 0.9):AdamW优化器的 beta1 超参数。 - adam_beta2 (
float, 可选, 默认为 0.999):AdamW优化器的 beta2 超参数。 - adam_epsilon (
float, 可选, 默认为 1e-8):AdamW优化器的 epsilon 超参数。
其他重要设置
- num_train_epochs (
float, 可选, 默认为 3.0): 要执行的总训练时代数。 - max_steps (
int, 可选, 默认为 -1): 如果设置为正数，则执行的总训练步数，这将覆盖num_train_epochs。 - warmup_steps (
int, 可选, 默认为 0): 从 0 到learning_rate的线性预热步数。
日志和保存
- logging_dir (
str, 可选): TensorBoard 日志目录，默认路径为output_dir/runs/CURRENT_DATETIME_HOSTNAME。 - save_strategy (
str或IntervalStrategy, 可选, 默认为"steps"): 训练期间采用的检查点保存策略。
高级设置
- use_cpu (
bool, 可选, 默认为False): 是否使用 CPU 进行训练。如果设为False，将默认使用 CUDA 或其他可用的硬件加速设备。 - seed (
int, 可选, 默认为 42): 训练开始时设置的随机种子，用来确保多次运行的可重复性。 - data_seed (
int, 可选): 用于数据采样的随机种子。如果未设置，则数据采样将使用与模型种子相同的种子，以确保数据采样的一致性。
模型保存与加载
- save_total_limit (
int, 可选): 如果设置了这个值，将限制保存的检查点总数。当启用load_best_model_at_end时，根据metric_for_best_model保存的“最佳”检查点将始终被保留，除此之外还会保留最新的几个检查点。 - save_safetensors (
bool, 可选, 默认为True): 使用 safetensors 来保存和加载状态字典，而不是使用默认的torch.load和torch.save。 - save_on_each_node (
bool, 可选, 默认为False): 在进行多节点分布式训练时，是否在每个节点上保存模型和检查点。 - save_only_model (
bool, 可选, 默认为False): 在进行检查点保存时，是否只保存模型，而不保存优化器、调度器及随机数生成状态。这样可以节省存储空间，但无法从检查点恢复训练。
混合精度和其他硬件优化
- fp16 (
bool, 可选, 默认为False): 是否使用 fp16 16位（混合精度）训练，代替 32位训练。 - bf16 (
bool, 可选, 默认为False): 是否使用 bf16 16位（混合精度）训练，需要支持的 NVIDIA 架构或使用 CPU。 - tf32 (
bool, 可选): 是否启用 TF32 模式，该模式在 Ampere 及更新的 GPU 架构上可用。默认值依赖于 PyTorch 版本的默认设置。
分布式训练和加速
- ddp_backend (
str, 可选): 用于分布式训练的后端。可能的值包括"nccl","mpi","ccl","gloo","hccl"等。 - tpu_num_cores (
int, 可选): 在 TPU 上训练时使用的 TPU 核心数。
高级调试和性能调整
- gradient_checkpointing (
bool, 可选, 默认为False): 是否使用梯度检查点来节省内存，代价是反向传播的速度会更慢。 - push_to_hub (
bool, 可选, 默认为False): 是否在每次保存模型时将其推送到 Hub。如果启用，output_dir将变为与仓库同步的 git 目录，并且每次触发保存时都会推送内容。
这些参数为使用 Hugging Face Transformers 进行模型训练提供了极大的灵活性和控制能力，允许用户根据具体的需求和硬件条件调整训练配置。
framework = "pt"
output_dir: str = field(
metadata={"help": "The output directory where the model predictions and checkpoints will be written."},
)
overwrite_output_dir: bool = field(
default=False,
metadata={
"help": (
"Overwrite the content of the output directory. "
"Use this to continue training if output_dir points to a checkpoint directory."
)
},
)
do_train: bool = field(default=False, metadata={"help": "Whether to run training."})
do_eval: bool = field(default=False, metadata={"help": "Whether to run eval on the dev set."})
do_predict: bool = field(default=False, metadata={"help": "Whether to run predictions on the test set."})
evaluation_strategy: Union[IntervalStrategy, str] = field(
default="no",
metadata={"help": "The evaluation strategy to use."},
)
prediction_loss_only: bool = field(
default=False,
metadata={"help": "When performing evaluation and predictions, only returns the loss."},
)
per_device_train_batch_size: int = field(
default=8, metadata={"help": "Batch size per GPU/TPU/MPS/NPU core/CPU for training."}
)
per_device_eval_batch_size: int = field(
default=8, metadata={"help": "Batch size per GPU/TPU/MPS/NPU core/CPU for evaluation."}
)
per_gpu_train_batch_size: Optional[int] = field(
default=None,
metadata={
"help": (
"Deprecated, the use of `--per_device_train_batch_size` is preferred. "
"Batch size per GPU/TPU core/CPU for training."
)
},
)
per_gpu_eval_batch_size: Optional[int] = field(
default=None,
metadata={
"help": (
"Deprecated, the use of `--per_device_eval_batch_size` is preferred. "
"Batch size per GPU/TPU core/CPU for evaluation."
)
},
)
gradient_accumulation_steps: int = field(
default=1,
metadata={"help": "Number of updates steps to accumulate before performing a backward/update pass."},
)
eval_accumulation_steps: Optional[int] = field(
default=None,
metadata={"help": "Number of predictions steps to accumulate before moving the tensors to the CPU."},
)
GPT-4
Poe
以下是对您提供的代码片段的逐行解释。这段代码看起来是用 Python 的 dataclasses 模块定义的一个配置类，用于管理和配置机器学习训练、评估和预测任务。代码使用了类型提示和默认值，还有详细的元数据说明，使得这些参数的用途更加清晰易懂。
python
复制
framework = "pt"
这行代码定义了一个变量 framework 并赋值为 "pt"，这通常代表使用 PyTorch 框架。在 Transformers 库中，"pt" 通常是指 PyTorch，而 "tf" 则代表 TensorFlow。
python
复制
output_dir: str = field(
metadata={"help": "The output directory where the model predictions and checkpoints will be written."},
)
这里定义了一个名为 output_dir 的数据类字段，其类型为 str（字符串）。通过 field() 函数，为这个字段添加了元数据，其中包含一个帮助字符串，解释这个字段的用途是指定模型预测和检查点将被写入的输出目录。
python
复制
overwrite_output_dir: bool = field(
default=False,
metadata={
"help": (
"Overwrite the content of the output directory. "
"Use this to continue training if output_dir points to a checkpoint directory."
)
},
)
这里定义了一个布尔类型的字段 overwrite_output_dir，默认值为 False。它的元数据提供了帮助信息，说明这个选项允许用户覆盖输出目录的内容，特别是当输出目录指向一个检查点目录时，可以用来继续之前的训练。
python
复制
do_train: bool = field(default=False, metadata={"help": "Whether to run training."})
do_eval: bool = field(default=False, metadata={"help": "Whether to run eval on the dev set."})
do_predict: bool = field(default=False, metadata={"help": "Whether to run predictions on the test set."})
这三行定义了控制训练、评估和预测的三个布尔类型字段：do_train、do_eval 和 do_predict。每个字段都有一个默认值 False，说明除非明确指定，否则不会执行相应的操作。
python
复制
evaluation_strategy: Union[IntervalStrategy, str] = field(
default="no",
metadata={"help": "The evaluation strategy to use."},
)
定义了一个名为 evaluation_strategy 的字段，其类型可以是 IntervalStrategy 类型或者字符串。默认值是 "no"，元数据中的帮助信息说明了这个字段用于指定评估策略。
python
复制
prediction_loss_only: bool = field(
default=False,
metadata={"help": "When performing evaluation and predictions, only returns the loss."},
)
prediction_loss_only 是一个布尔字段，控制在执行评估和预测时是否只返回损失值。
python
复制
per_device_train_batch_size: int = field(
default=8, metadata={"help": "Batch size per GPU/TPU/MPS/NPU core/CPU for training."}
)
per_device_eval_batch_size: int = field(
default=8, metadata={"help": "Batch size per GPU/TPU/MPS/NPU core/CPU for evaluation."}
)
这两行定义了训练和评估时每个设备的批量大小，包括 GPU、TPU、MPS、NPU 或 CPU。
python
复制
per_gpu_train_batch_size: Optional[int] = field(
default=None,
metadata={
"help": (
"Deprecated, the use of `--per_device_train_batch_size` is preferred. "
"Batch size per GPU/TPU core/CPU for training."
)
},
)
per_gpu_eval_batch_size: Optional[int] = field(
default=None,
metadata={
"help": (
"Deprecated, the use of `--per_device_eval_batch_size` is preferred. "
"Batch size per GPU/TPU core/CPU for evaluation."
)
},
)
这两行定义了已被弃用的字段 per_gpu_train_batch_size 和 per_gpu_eval_batch_size，这两个字段都被设置为 None 并标记为“已弃用”，推荐使用 per_device_train_batch_size 和 per_device_eval_batch_size。
python
复制
gradient_accumulation_steps: int = field(
default=1,
metadata={"help": "Number of updates steps to accumulate before performing a backward/update pass."},
)
此行定义了一个整数字段 gradient_accumulation_steps，默认值为 1。这个设置用于在执行反向传播更新之前累积的梯度更新步骤数。在小批量数据较小或计算资源有限时，通过累积梯度可以有效地模拟更大批量的训练，从而实现稳定的优化。
python
复制
eval_accumulation_steps: Optional[int] = field(
default=None,
metadata={"help": "Number of predictions steps to accumulate before moving the tensors to the CPU."},
)
最后一行定义了一个可选整数字段 eval_accumulation_steps，默认值为 None。这个字段用于控制在将数据传输到 CPU 之前，应积累多少预测步骤的结果。这在处理大量数据时尤其有用，可以优化内存使用和计算效率。
总体说明
这段代码通过定义一个数据类，为机器学习模型的训练、验证和测试提供了一系列可配置的参数。每个参数都有详细的描述，使得其他开发者或用户能够更容易地理解每个参数的作用，并根据需要进行调整。使用此类配置还有助于代码的模块化和重用，增强了代码的可维护性和可扩展性。
eval_delay: Optional[float] = field(
default=0,
metadata={
"help": (
"Number of epochs or steps to wait for before the first evaluation can be performed, depending on the"
" evaluation_strategy."
)
},
)
learning_rate: float = field(default=5e-5, metadata={"help": "The initial learning rate for AdamW."})
weight_decay: float = field(default=0.0, metadata={"help": "Weight decay for AdamW if we apply some."})
adam_beta1: float = field(default=0.9, metadata={"help": "Beta1 for AdamW optimizer"})
adam_beta2: float = field(default=0.999, metadata={"help": "Beta2 for AdamW optimizer"})
adam_epsilon: float = field(default=1e-8, metadata={"help": "Epsilon for AdamW optimizer."})
max_grad_norm: float = field(default=1.0, metadata={"help": "Max gradient norm."})
num_train_epochs: float = field(default=3.0, metadata={"help": "Total number of training epochs to perform."})
max_steps: int = field(
default=-1,
metadata={"help": "If > 0: set total number of training steps to perform. Override num_train_epochs."},
)
lr_scheduler_type: Union[SchedulerType, str] = field(
default="linear",
metadata={"help": "The scheduler type to use."},
)
lr_scheduler_kwargs: Optional[Union[dict, str]] = field(
default_factory=dict,
metadata={
"help": (
"Extra parameters for the lr_scheduler such as {'num_cycles': 1} for the cosine with hard restarts."
)
},
)
warmup_ratio: float = field(
default=0.0, metadata={"help": "Linear warmup over warmup_ratio fraction of total steps."}
)
warmup_steps: int = field(default=0, metadata={"help": "Linear warmup over warmup_steps."})
log_level: Optional[str] = field(
default="passive",
metadata={
"help": (
"Logger log level to use on the main node. Possible choices are the log levels as strings: 'debug',"
" 'info', 'warning', 'error' and 'critical', plus a 'passive' level which doesn't set anything and"
" lets the application set the level. Defaults to 'passive'."
),
"choices": trainer_log_levels.keys(),
},
)
log_level_replica: Optional[str] = field(
default="warning",
metadata={
"help": "Logger log level to use on replica nodes. Same choices and defaults as ``log_level``",
"choices": trainer_log_levels.keys(),
},
)
log_on_each_node: bool = field(
default=True,
metadata={
"help": (
"When doing a multinode distributed training, whether to log once per node or just once on the main"
" node."
)
},
)
logging_dir: Optional[str] = field(default=None, metadata={"help": "Tensorboard log dir."})
logging_strategy: Union[IntervalStrategy, str] = field(
default="steps",
metadata={"help": "The logging strategy to use."},
)以下是Transformers库中代码的逐行解释，这些代码用于设置模型训练的各种配置参数：
eval_delay:
python
复制
eval_delay: Optional[float] = field(
default=0,
metadata={
"help": (
"Number of epochs or steps to wait for before the first evaluation can be performed, depending on the"
" evaluation_strategy."
)
},
)
eval_delay 参数用于设定在第一次模型评估之前应等待的训练周期数或步数。具体等待的周期数或步数取决于 evaluation_strategy 的设置。
learning_rate:
python
复制
learning_rate: float = field(default=5e-5, metadata={"help": "The initial learning rate for AdamW."})
learning_rate 设置AdamW优化器的初始学习率。这里默认为0.00005。
weight_decay:
python
复制
weight_decay: float = field(default=0.0, metadata={"help": "Weight decay for AdamW if we apply some."})
weight_decay 参数用于设置AdamW优化器的权重衰减，这里默认值为0.0，表示不应用权重衰减。
adam_beta1:
python
复制
adam_beta1: float = field(default=0.9, metadata={"help": "Beta1 for AdamW optimizer"})
adam_beta1 设置AdamW优化器的Beta1参数，用于一阶矩估计的指数衰减率，默认为0.9。
adam_beta2:
python
复制
adam_beta2: float = field(default=0.999, metadata={"help": "Beta2 for AdamW optimizer"})
adam_beta2 设置AdamW优化器的Beta2参数，用于二阶矩估计的指数衰减率，默认为0.999。
adam_epsilon:
python
复制
adam_epsilon: float = field(default=1e-8, metadata={"help": "Epsilon for AdamW optimizer."})
adam_epsilon 设置AdamW优化器的epsilon参数，这是一个非常小的数，用于防止除零错误，默认为1e-8。
max_grad_norm:
python
复制
max_grad_norm: float = field(default=1.0, metadata={"help": "Max gradient norm."})
max_grad_norm 设置梯度裁剪的最大梯度范数，防止梯度爆炸，默认为1.0。
num_train_epochs:
python
复制
num_train_epochs: float = field(default=3.0, metadata={"help": "Total number of training epochs to perform."})
num_train_epochs 设置总共进行的训练周期数，默认为3。
max_steps:
python
复制
max_steps: int = field(
default=-1,
metadata={"help": "If > 0: set total number of training steps to perform. Override num_train_epochs."},
)
max_steps 设置训练的总步数。如果设置大于0的值，将覆盖 num_train_epochs 参数。
lr_scheduler_type:
python
复制
lr_scheduler_type: Union[SchedulerType, str] = field(
default="linear",
metadata={"help": "The scheduler type to use."},
)
lr_scheduler_type 设置学习率调度器的类型，默认为“linear”。
lr_scheduler_kwargs:
python
复制
lr_scheduler_kwargs: Optional[Union[dict, str]] = field(
default_factory=dict,
metadata={
"help": (
"Extra parameters for the lr_scheduler such as {'num_cycles': 1} for the cosine with hard restarts."
)
},
)
lr_scheduler_kwargs 允许为学习率调度器提供额外的参数，例如在使用周期性变化的学习率时指定周期数。
warmup_ratio:
python
复制
warmup_ratio: float = field(
default=0.0, metadata={"help": "Linear warmup over warmup_ratio fraction of total steps."}
)
warmup_ratio 设置线性预热期的比例，即在总步数中有多大比例的步数用于预热，这里默认为0.0，表示不进行预热。
warmup_steps:
python
复制
warmup_steps: int = field(default=0, metadata={"help": "Linear warmup over warmup_steps."})
warmup_steps 设置预热的具体步数，这里默认为0，表示不进行预热。
log_level:
python
复制
log_level: Optional[str] = field(
default="passive",
metadata={
"help": (
"Logger log level to use on the main node. Possible choices are the log levels as strings: 'debug',"
" 'info', 'warning', 'error' and 'critical', plus a 'passive' level which doesn't set anything and"
" lets the application set the level. Defaults to 'passive'."
),
"choices": trainer_log_levels.keys(),
},
)
log_level 设置主节点上的日志级别。默认为"passive"，这意味着不设置任何日志级别，留给应用程序来决定。其他选项包括"debug", "info", "warning", "error"和"critical"。
log_level_replica:
python
复制
log_level_replica: Optional[str] = field(
default="warning",
metadata={
"help": "Logger log level to use on replica nodes. Same choices and defaults as ``log_level``",
"choices": trainer_log_levels.keys(),
},
)
log_level_replica 设置副本节点上的日志级别，选项与log_level相同，但默认设置为"warning"。
log_on_each_node:
python
复制
log_on_each_node: bool = field(
default=True,
metadata={
"help": (
"When doing a multinode distributed training, whether to log once per node or just once on the main"
" node."
)
},
)
在进行多节点分布式训练时，log_on_each_node 设置是否在每个节点上记录日志。默认为True，即每个节点都进行日志记录。
logging_dir:
python
复制
logging_dir: Optional[str] = field(default=None, metadata={"help": "Tensorboard log dir."})
logging_dir 设置Tensorboard日志的目录。默认为None，即不设置目录。
logging_strategy:
python
复制
logging_strategy: Union[IntervalStrategy, str] = field(
default="steps",
metadata={"help": "The logging strategy to use."},
)
logging_strategy 设置日志记录策略，可以选择基于时间间隔或者步数。默认为"steps"，即根据步数来记录日志。
以上解释了这部分代码中各个参数的设置及其意义，这些参数主要用于配置模型训练过程中的优化器、学习率调度器、日志记录等重要方面。
logging_first_step: bool = field(default=False, metadata={"help": "Log the first global_step"})
logging_steps: float = field(
default=500,
metadata={
"help": (
"Log every X updates steps. Should be an integer or a float in range `[0,1)`. "
"If smaller than 1, will be interpreted as ratio of total training steps."
)
},
)
logging_nan_inf_filter: bool = field(default=True, metadata={"help": "Filter nan and inf losses for logging."})
save_strategy: Union[IntervalStrategy, str] = field(
default="steps",
metadata={"help": "The checkpoint save strategy to use."},
)
save_steps: float = field(
default=500,
metadata={
"help": (
"Save checkpoint every X updates steps. Should be an integer or a float in range `[0,1)`. "
"If smaller than 1, will be interpreted as ratio of total training steps."
)
},
)
save_total_limit: Optional[int] = field(
default=None,
metadata={
"help": (
"If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in"
" `output_dir`. When `load_best_model_at_end` is enabled, the 'best' checkpoint according to"
" `metric_for_best_model` will always be retained in addition to the most recent ones. For example,"
" for `save_total_limit=5` and `load_best_model_at_end=True`, the four last checkpoints will always be"
" retained alongside the best model. When `save_total_limit=1` and `load_best_model_at_end=True`,"
" it is possible that two checkpoints are saved: the last one and the best one (if they are different)."
" Default is unlimited checkpoints"
)
},
)
save_safetensors: Optional[bool] = field(
default=True,
metadata={
"help": "Use safetensors saving and loading for state dicts instead of default torch.load and torch.save."
},
)
save_on_each_node: bool = field(
default=False,
metadata={
"help": (
"When doing multi-node distributed training, whether to save models and checkpoints on each node, or"
" only on the main one"
)
},
)
save_only_model: bool = field(
default=False,
metadata={
"help": (
"When checkpointing, whether to only save the model, or also the optimizer, scheduler & rng state."
"Note that when this is true, you won't be able to resume training from checkpoint."
"This enables you to save storage by not storing the optimizer, scheduler & rng state."
"You can only load the model using from_pretrained with this option set to True."
)
},
)
no_cuda: bool = field(
default=False,
metadata={"help": "This argument is deprecated. It will be removed in version 5.0 of 🤗 Transformers."},
)
use_cpu: bool = field(
default=False,
metadata={
"help": " Whether or not to use cpu. If set to False, we will use cuda/tpu/mps/npu device if available."
},
)
use_mps_device: bool = field(
default=False,
metadata={
"help": "This argument is deprecated. `mps` device will be used if available similar to `cuda` device."
" It will be removed in version 5.0 of 🤗 Transformers"
},
)
seed: int = field(default=42, metadata={"help": "Random seed that will be set at the beginning of training."})
data_seed: Optional[int] = field(default=None, metadata={"help": "Random seed to be used with data samplers."})
jit_mode_eval: bool = field(
default=False, metadata={"help": "Whether or not to use PyTorch jit trace for inference"}
)
use_ipex: bool = field(
default=False,
metadata={
"help": (
"Use Intel extension for PyTorch when it is available, installation:"
" 'https://github.com/intel/intel-extension-for-pytorch'"
)
},
)
bf16: bool = field(
default=False,
metadata={
"help": (
"Whether to use bf16 (mixed) precision instead of 32-bit. Requires Ampere or higher NVIDIA"
" architecture or using CPU (use_cpu) or Ascend NPU. This is an experimental API and it may change."
)
},
)
fp16: bool = field(
default=False,
metadata={"help": "Whether to use fp16 (mixed) precision instead of 32-bit"},
)
fp16_opt_level: str = field(
default="O1",
metadata={
"help": (
"For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. "
"See details at https://nvidia.github.io/apex/amp.html"
)
},
)GPT-4
Poe
以下是代码逐行解释，主要用于编程和人工智能教学的目的。这段代码是使用 Python 的 dataclasses 模块定义的一组配置项，这些配置项通常用于控制机器学习模型的训练过程。
logging_first_step: bool = field(default=False, metadata={"help": "Log the first global_step"})
定义一个布尔型变量 logging_first_step，默认值为 False。该变量用于控制是否在训练开始时记录第一个全局步骤（global_step）的日志。元数据中提供了帮助信息，解释了此配置的作用。
sql_more
复制
logging_steps: float = field(
default=500,
metadata={
"help": (
"Log every X updates steps. Should be an integer or a float in range `[0,1)`. "
"If smaller than 1, will be interpreted as ratio of total training steps."
)
},
)
定义一个浮点型变量 logging_steps，默认值为 500。此变量决定每多少步骤记录一次训练过程的日志。如果这个值小于1，则被解释为总训练步数的比例。
logging_nan_inf_filter: bool = field(default=True, metadata={"help": "Filter nan and inf losses for logging."})
定义一个布尔型变量 logging_nan_inf_filter，默认值为 True。此变量用于控制是否在记录日志时过滤出现的 NaN 或无穷大的损失值。
复制
save_strategy: Union[IntervalStrategy, str] = field(
default="steps",
metadata={"help": "The checkpoint save strategy to use."},
)
定义一个变量 save_strategy，可以是 IntervalStrategy 枚举类型或字符串，用于指定模型检查点的保存策略，默认为 "steps"。
sql_more
复制
save_steps: float = field(
default=500,
metadata={
"help": (
"Save checkpoint every X updates steps. Should be an integer or a float in range `[0,1)`. "
"If smaller than 1, will be interpreted as ratio of total training steps."
)
},
)
定义一个浮点型变量 save_steps，默认值为 500。此变量决定每多少步骤保存一次模型检查点。如果这个值小于1，则被解释为总训练步数的比例。
sql_more
复制
save_total_limit: Optional[int] = field(
default=None,
metadata={
"help": (
"If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in"
" `output_dir`. When `load_best_model_at_end` is enabled, the 'best' checkpoint according to"
" `metric_for_best_model` will always be retained in addition to the most recent ones. For example,"
" for `save_total_limit=5` and `load_best_model_at_end=True`, the four last checkpoints will always be"
" retained alongside the best model. When `save_total_limit=1` and `load_best_model_at_end=True`,"
" it is possible that two checkpoints are saved: the last one and the best one (if they are different)."
" Default is unlimited checkpoints"
)
},
)
定义一个可选的整型变量 save_total_limit，默认值为 None。此变量用于限制保存的检查点总数。如果设置了此值，旧的检查点将会被删除以保持总数不超过限制。
save_safetensors: Optional[bool] = field(default=True, metadata={"help": "Use safetensors saving and loading for state dicts instead of default torch.load and torch.save."})
定义一个可选的布尔型变量 save_safetensors，默认值为 True。此变量控制是否使用 safetensors 方式来保存和加载状态字典，这种方式与默认的 torch.load 和 torch.save 不同。
save_on每个节点: bool = field(
default=False,
metadata={
"help": (
"When doing multi-node distributed training, whether to save models and checkpoints on each node, or"
" only on the main one"
)
},
)
复制
- 定义一个布尔型变量 `save_on_each_node`，默认为 `False`。该变量用于控制在进行多节点分布式训练时，是否在每个节点上保存模型和检查点，或只在主节点上保存。
smalltalk
复制
save_only_model: bool = field(
default=False,
metadata={
"help": (
"When checkpointing, whether to only save the model, or also the optimizer, scheduler & rng state."
"Note that when this is true, you won't be able to resume training from checkpoint."
"This enables you to save storage by not storing the optimizer, scheduler & rng state."
"You can only load the model using from_pretrained with this option set to True."
)
},
)
定义一个布尔型变量 save_only_model，默认为 False。此选项控制在保存检查点时是否仅保存模型本身，还是同时保存优化器、调度器和随机数生成器的状态。当此选项为 True 时，无法从检查点恢复训练。
no_cuda: bool = field(default=False, metadata={"help": "This argument is deprecated. It will be removed in version 5.0 of 🤗 Transformers."})
定义一个布尔型变量 no_cuda，默认值为 False。此变量指示是否禁用 CUDA 支持，但已被标记为弃用，并将在 Transformers 库的未来版本中移除。
use_cpu: bool = field(default=False, metadata={"help": " Whether or not to use cpu. If set to False, we will use cuda/tpu/mps/npu device if available."})
定义一个布尔型变量 use_cpu，默认值为 False。此变量用于控制是否强制使用 CPU 进行计算。如果设置为 False，将自动选择可用的其他设备（如 CUDA、TPU、MPS 或 NPU）。
use_mps_device: bool = field(default=False, metadata={"help": "This argument is deprecated. mpsdevice will be used if available similar tocuda device. It will be removed in version 5.0 of 🤗 Transformers"})
定义一个布尔型变量 use_mps_device，默认值为 False。此变量表示是否使用 MPS 设备（适用于苹果硬件的一种加速技术），但已被标记为弃用，并将在 Transformers 库的未来版本中移除。
seed: int = field(default=42, metadata={"help": "Random seed that will be set at the beginning of training."})
定义一个整型变量 seed，默认值为 42。此变量用于设置训练开始时的随机种子，以确保结果的可重复性。
- use_cpu (
bool, 可选, 默认为False): 是否使用 CPU 进行训练。如果设为False，将默认使用 CUDA 或其他可用的硬件加速设备。 - seed (
int, 可选, 默认为 42): 训练开始时设置的随机种子，用来确保多次运行的可重复性。 - data_seed (
int, 可选): 用于数据采样的随机种子。如果未设置，则数据采样将使用与模型种子相同的种子，以确保数据采样的一致性。
模型保存与加载
- save_total_limit (
int, 可选): 如果设置了这个值，将限制保存的检查点总数。当启用load_best_model_at_end时，根据metric_for_best_model保存的“最佳”检查点将始终被保留，除此之外还会保留最新的几个检查点。 - save_safetensors (
bool, 可选, 默认为True): 使用 safetensors 来保存和加载状态字典，而不是使用默认的torch.load和torch.save。 - save_on_each_node (
bool, 可选, 默认为False): 在进行多节点分布式训练时，是否在每个节点上保存模型和检查点。 - save_only_model (
bool, 可选, 默认为False): 在进行检查点保存时，是否只保存模型，而不保存优化器、调度器及随机数生成状态。这样可以节省存储空间，但无法从检查点恢复训练。
混合精度和其他硬件优化
- fp16 (
bool, 可选, 默认为False): 是否使用 fp16 16位（混合精度）训练，代替 32位训练。 - bf16 (
bool, 可选, 默认为False): 是否使用 bf16 16位（混合精度）训练，需要支持的 NVIDIA 架构或使用 CPU。 - tf32 (
bool, 可选): 是否启用 TF32 模式，该模式在 Ampere 及更新的 GPU 架构上可用。默认值依赖于 PyTorch 版本的默认设置。
分布式训练和加速
- ddp_backend (
str, 可选): 用于分布式训练的后端。可能的值包括"nccl","mpi","ccl","gloo","hccl"等。 - tpu_num_cores (
int, 可选): 在 TPU 上训练时使用的 TPU 核心数。
高级调试和性能调整
- gradient_checkpointing (
bool, 可选, 默认为False): 是否使用梯度检查点来节省内存，代价是反向传播的速度会更慢。 - push_to_hub (
bool, 可选, 默认为False): 是否在每次保存模型时将其推送到 Hub。如果启用，output_dir将变为与仓库同步的 git 目录，并且每次触发保存时都会推送内容。
这些参数为使用 Hugging Face Transformers 进行模型训练提供了极大的灵活性和控制能力，允许用户根据具体的需求和硬件条件调整训练配置。
问答
这种Union就是枚举类型吗，和其他的类型有什么区别
save_strategy: Union[IntervalStrategy, str] = field(
default="steps",
metadata={"help": "The checkpoint save strategy to use."},
)
GPT-4-128k
Poe
在 Python 中，Union 类型来自于类型提示系统（Type Hints），并非直接表示枚举类型，而是用来指示一个变量可以接受多种不同类型的值。使用 Union 是为了告诉开发者和代码编辑工具这个变量可以是几种类型中的任何一种。
Union 的作用和应用
Union 类型在 typing 模块中定义，用来组合多个类型，表示变量可以是这些类型中的任何一个。例如，Union[int, str] 表示变量可以是整数或字符串。
示例
在你的例子中：
python
复制
save_strategy: Union[IntervalStrategy, str] = field(
default="steps",
metadata={"help": "The checkpoint save strategy to use."},
)
这里的 save_strategy 可以是 IntervalStrategy 类型的一个实例，或者是一个字符串。IntervalStrategy 可能是一个枚举类或其他具体的类，用于定义不同的间隔策略，而 str 类型则允许用户直接通过字符串方式指定策略。
枚举类型（Enum）
枚举类型，通常通过 Python 的 Enum 类定义，用于创建一组命名常量。每个枚举成员都有一个名称和一个值，枚举的主要目的是确保某些特定的变量只能取有限集合中的值。
和 Union 的区别
- 枚举 (Enum): 用于定义一个变量的可能值的固定集合。每个枚举成员都是枚举类的唯一实例。
- Union: 并不限制值的集合，而是说明变量可以接受的多种类型。每个类型可以有无限多个可能的值。
使用场景
- 使用 枚举 当你想要变量的值只限于几个固定选项时。例如，定义一个颜色枚举，只包含红、绿、蓝。
- 使用 Union 当你的函数或方法可以接受多种类型的参数时。例如，一个函数可以接受数字或字符串作为输入。
总结，Union 和枚举类型服务于不同的目的：Union 提供类型灵活性，而枚举提供值的固定性和限制性。在你的代码示例中，使用 Union 可能是为了兼容性和灵活性，允许不同类型的值用于同一参数。
1392
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
