Created
November 1, 2024 17:14
-
-
Save VRehnberg/bb06bbde9c77738276896d8c5fcd4d24 to your computer and use it in GitHub Desktop.
(partial) EasyBuild log for failed build of /cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-lquq6o8j/files_pr21438/d/DeepSpeed/DeepSpeed-0.14.5-foss-2023a-CUDA-12.1.1.eb (PR(s) #21438) (easyblock PR(s) #3450)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 6.49s call unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fp32_optimizer | |
| 6.48s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype1] | |
| 6.48s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-1-dtype2] | |
| 6.48s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_4to2[True-False-1-dtype2] | |
| 6.48s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_Dict] | |
| 6.48s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_arg_none[mask1] | |
| 6.48s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[3] | |
| 6.48s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-33] | |
| 6.47s call unit/runtime/zero/test_zero.py::TestZero3DictFwd::test[list] | |
| 6.47s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyGrad::test[2] | |
| 6.47s call unit/runtime/half_precision/test_bf16.py::TestZeroDtypeCocktail::test[fp16-bf16] | |
| 6.47s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[False-2] | |
| 6.47s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-1-dtype2] | |
| 6.46s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_4to2[True-True-3-dtype1] | |
| 6.46s call unit/runtime/half_precision/test_bf16.py::TestZeroDtypeCocktail::test[bf16-fp16] | |
| 6.46s call unit/runtime/test_ds_config_dict.py::TestDeprecatedDeepScaleConfig::test | |
| 6.45s call unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[True] | |
| 6.45s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-32-18-1-False] | |
| 6.45s call unit/runtime/zero/test_zero.py::TestZeroFrozenWeights::test[1] | |
| 6.44s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_4to2[True-False-3-dtype0] | |
| 6.44s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_4to2[False-False-3-dtype1] | |
| 6.44s call unit/runtime/half_precision/test_fp16.py::TestZero2ReduceScatterOff::test | |
| 6.44s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype0] | |
| 6.43s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_quantized_linear[8bits-0] | |
| 6.42s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[1e-05-1e-05-1-False] | |
| 6.42s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_Tuple] | |
| 6.42s call unit/runtime/zero/test_zero.py::TestZeroPartitionCache::test_training_partition_cache[True] | |
| 6.41s call unit/runtime/half_precision/test_bf16.py::TestZeroAllowUntestedOptimizer::test | |
| 6.41s call unit/runtime/sparse_tensor/test_averaging_sparse_gradients.py::TestSparseAdam::test | |
| 6.41s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_4to2[False-True-1-dtype2] | |
| 6.40s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_quantized_linear[4bits-0] | |
| 6.40s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-True-False] | |
| 6.40s call unit/runtime/zero/test_zero_nesting_init.py::TestShutdownInNestingInit::test_shutdown_in_nesting_init | |
| 6.40s call unit/runtime/test_ds_config_dict.py::TestDistInit::test | |
| 6.39s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_4to2[False-True-3-dtype1] | |
| 6.39s call unit/runtime/half_precision/test_bf16.py::TestZeroDtypeCocktail::test[default-bf16] | |
| 6.39s call unit/runtime/test_data_efficiency.py::TestLegacyCurriculumScheduler::test_fixed_discrete | |
| 6.39s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_reduce_scatter[False] | |
| 6.39s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[False-True] | |
| 6.38s call unit/runtime/test_runtime_utils.py::TestCheckOverflow::test[True] | |
| 6.38s call unit/runtime/half_precision/test_fp16.py::TestZeroStaticScale::test[False-2] | |
| 6.37s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningManyParams::test[False] | |
| 6.37s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestUnfused::test_some_overflow | |
| 6.37s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-False] | |
| 6.37s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-3-dtype0] | |
| 6.37s call unit/comm/test_dist.py::TestDistInitNoEnv::test | |
| 6.36s call unit/runtime/half_precision/test_bf16.py::TestZeroDtypeCocktail::test[fp16-fp32] | |
| 6.36s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_quantized_linear[4bits-1] | |
| 6.36s call unit/runtime/test_data_efficiency.py::TestDataEfficiency::test_curriculum_learning | |
| 6.36s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_namedtuple] | |
| 6.35s call unit/runtime/half_precision/test_bf16.py::TestZero2ReduceScatterOff::test | |
| 6.35s call unit/runtime/half_precision/test_fp16.py::TestFP16AdamTypes::test[False-Adam] | |
| 6.35s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output3] | |
| 6.35s call unit/runtime/zero/test_zero.py::TestZeroFrozenWeights::test[3] | |
| 6.34s call unit/runtime/zero/test_zero.py::TestZero3RepeatForwardLoop::test[False] | |
| 6.34s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-1-dtype0] | |
| 6.34s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype1] | |
| 6.34s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-1-dtype1] | |
| 6.34s call unit/runtime/half_precision/test_fp16.py::TestZeroStaticScale::test[False-1] | |
| 6.33s call unit/runtime/zero/test_zero.py::TestZeroFrozenWeights::test[2] | |
| 6.33s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-15] | |
| 6.33s call unit/runtime/test_ds_initialize.py::TestClientOptimizer::test[Callable] | |
| 6.32s call unit/runtime/half_precision/test_bf16.py::TestZeroDtypeCocktail::test[fp16-fp16] | |
| 6.32s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_reduce_scatter[True] | |
| 6.32s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_contiguous_gradients[True] | |
| 6.32s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-False-False] | |
| 6.31s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[0] | |
| 6.31s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-True-True] | |
| 6.31s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-3-dtype2] | |
| 6.31s call unit/runtime/zero/test_zero.py::TestZeroOffloadOptim::test[False] | |
| 6.31s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyGrad::test[3] | |
| 6.31s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_prefetching[True] | |
| 6.31s call unit/comm/test_dist.py::TestInit::test | |
| 6.30s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[FusedAdam-3] | |
| 6.30s call unit/runtime/test_runtime_utils.py::TestClipGradNorm::test_clipped_val | |
| 6.30s call unit/runtime/zero/test_zero_nesting_init.py::TestNestingInit::test_nesting_init | |
| 6.28s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[log-10] | |
| 6.28s call unit/runtime/comm/test_coalesced_collectives.py::TestReduceScatterCoalescedTensorSmallerThanWorldSize::test | |
| 6.28s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[0-False] | |
| 6.27s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs3[mask0] | |
| 6.26s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output2] | |
| 6.26s call unit/runtime/test_ds_config_dict.py::TestBasicConfig::test_accelerator | |
| 6.26s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[OneCycle-params2] | |
| 6.26s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_offload_optimizer[False] | |
| 6.26s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_save_exclude_custom_frozen_weights[1] | |
| 6.26s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[0] | |
| 6.26s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[0.9] | |
| 6.25s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_save_exclude_custom_frozen_weights[2] | |
| 6.25s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-32-8-2-True] | |
| 6.25s call unit/runtime/half_precision/test_fp16.py::TestZero3LazyScatter::test | |
| 6.25s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-True-True] | |
| 6.24s call unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test | |
| 6.24s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[True] | |
| 6.24s call unit/linear/test_ctx.py::TestInitTransformers::test_config_init | |
| 6.23s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-32-16-1-True] | |
| 6.23s call unit/runtime/half_precision/test_fp16.py::TestZeroStaticScale::test[False-3] | |
| 6.23s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Optimizer-_LRScheduler] | |
| 6.22s call unit/comm/test_dist.py::TestDistributedFixture::test[2-16] | |
| 6.22s call unit/ops/adagrad/test_cpu_adagrad.py::TestCPUAdagradGPUError::test_cpu_adagrad_gpu_error | |
| 6.22s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[non_tensor4] | |
| 6.22s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpoint::test_load_module_only[0] | |
| 6.22s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-3-dtype0] | |
| 6.22s call unit/utils/test_init_on_device.py::TestOnDevice::test_on_device[cuda:0] | |
| 6.21s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[3] | |
| 6.21s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype1] | |
| 6.21s call unit/runtime/test_multi_output_model.py::TestThreeOutputModel::test | |
| 6.21s call unit/runtime/half_precision/test_fp16.py::TestAdamwFP16EmptyGrad::test | |
| 6.20s call unit/checkpoint/test_zero_optimizer.py::TestZeRONonDistributed::test_chmod_exception_handling[2] | |
| 6.20s call unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[False] | |
| 6.20s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-False-False] | |
| 6.20s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs1_outputs1[mask0] | |
| 6.20s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_zero_grad[False] | |
| 6.19s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_NamedTuple] | |
| 6.19s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_contiguous_gradients[False] | |
| 6.18s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[False-3] | |
| 6.17s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-1-dtype1] | |
| 6.17s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[True-True] | |
| 6.17s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[Adam-1] | |
| 6.17s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-False-False] | |
| 6.17s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[LRRangeTest-params3] | |
| 6.16s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-False-False] | |
| 6.16s call unit/runtime/test_ds_config_dict.py::TestBasicConfig::test_check_version | |
| 6.16s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-False-False] | |
| 6.16s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-True-True] | |
| 6.15s call unit/runtime/zero/test_zero_context_return.py::TestReturnParam::test_stage_3_output_type[tensor] | |
| 6.15s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[non_tensor3] | |
| 6.15s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[0] | |
| 6.15s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype0] | |
| 6.15s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_fp16_enabled[False] | |
| 6.15s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[2] | |
| 6.15s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-False-True] | |
| 6.15s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-1-dtype2] | |
| 6.14s call unit/runtime/zero/test_zero_context_return.py::TestReturnParam::test_stage_3_output_type[dict] | |
| 6.14s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[FusedAdam-1] | |
| 6.14s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-True-False] | |
| 6.14s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestFused::test_no_overflow | |
| 6.14s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_fp16_enabled[True] | |
| 6.13s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestFused::test_all_overflow | |
| 6.13s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-15] | |
| 6.13s call unit/runtime/test_ds_initialize.py::TestClientOptimizer::test[Optimizer] | |
| 6.13s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_same_lrscheler_and_callable[_LRScheduler] | |
| 6.13s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable[None] | |
| 6.13s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs1_outputs1[mask1] | |
| 6.12s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype1] | |
| 6.12s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-1-dtype0] | |
| 6.11s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-True-False] | |
| 6.11s call unit/runtime/zero/test_zero.py::TestZero3RepeatForwardLoop::test[True] | |
| 6.11s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-True] | |
| 6.11s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs2[mask1] | |
| 6.10s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-False-True] | |
| 6.10s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[None] | |
| 6.10s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype2] | |
| 6.10s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[0.1] | |
| 6.10s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-1-dtype2] | |
| 6.10s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-33] | |
| 6.10s call unit/checkpoint/test_shared_weights.py::TestCheckpointSharedWeights::test_checkpoint_shared_weights | |
| 6.10s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[0] | |
| 6.10s call unit/runtime/zero/test_zero.py::TestZeroAdamOptimizerStepCount::test[3] | |
| 6.10s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyPartition::test[False-3] | |
| 6.09s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_disable_autocast_linear[True] | |
| 6.09s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[0.001-0.1-0-21-21] | |
| 6.09s call unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[True] | |
| 6.09s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-33-17-2-False] | |
| 6.09s call unit/runtime/comm/test_coalesced_collectives.py::TestReduceScatterCoalesced::test_single_input | |
| 6.08s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Callable-None] | |
| 6.08s call unit/runtime/test_ds_initialize.py::TestNoOptim::test[0] | |
| 6.08s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-1-dtype2] | |
| 6.07s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[None] | |
| 6.07s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[2] | |
| 6.07s call unit/runtime/test_ds_config_dict.py::TestInitNoOptimizer::test | |
| 6.06s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-1-dtype2] | |
| 6.06s call unit/checkpoint/test_zero_optimizer.py::TestZeRONonDistributed::test_chmod_exception_handling[1] | |
| 6.06s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[non_tensor_output1] | |
| 6.06s call unit/monitor/test_monitor.py::TestCSVMonitor::test_csv_monitor | |
| 6.06s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype0] | |
| 6.06s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[1.0] | |
| 6.06s call unit/runtime/half_precision/test_bf16.py::TestZeroSupportedClientOptimizer::test[Adam] | |
| 6.05s call unit/runtime/test_ds_config_dict.py::TestArgs::test_no_args | |
| 6.05s call unit/runtime/test_runtime_utils.py::TestClipGradNorm::test_gather | |
| 6.05s call unit/runtime/test_lr_schedulers.py::TestSchedulerOptimizerParity::test[WarmupLR-params0] | |
| 6.04s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[2] | |
| 6.04s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[1e-05-0.1-0-10-0] | |
| 6.04s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_disable_autocast_linear[False] | |
| 6.04s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[IGNORE] | |
| 6.04s call unit/runtime/half_precision/test_fp16.py::TestFP16AdamTypes::test[True-AdamW] | |
| 6.04s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestUnfused::test_no_overflow | |
| 6.04s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-10] | |
| 6.04s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.01-0.01-19-True] | |
| 6.03s call unit/runtime/zero/test_zero.py::TestZero3InitForParentWeightInitialization::test | |
| 6.03s call unit/runtime/comm/test_coalesced_collectives.py::TestAllToAllQuantReduceFallback::test_1d_tensor | |
| 6.03s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[1e-05-0.01-0.001-10-100] | |
| 6.03s call unit/runtime/half_precision/test_fp16.py::TestAdamFP16ZeroOneCycleCompatibility::test[False-2] | |
| 6.03s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs1[mask0] | |
| 6.03s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[WARN] | |
| 6.02s call unit/compression/test_compression.py::TestCompression::test_linear_layer_compress | |
| 6.02s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-3-dtype0] | |
| 6.02s call unit/runtime/zero/test_zero.py::TestZeroPartitionCache::test_training_partition_cache[False] | |
| 6.02s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_half_int8_quantization | |
| 6.01s call unit/runtime/zero/test_zero.py::TestZeroAdamOptimizerStepCount::test[2] | |
| 6.01s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-3-dtype2] | |
| 6.01s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-3-dtype1] | |
| 6.01s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-3-dtype1] | |
| 6.01s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_ext_param_getattr | |
| 6.01s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[non_tensor4] | |
| 6.01s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_float_int4_quantization | |
| 6.01s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs3[mask0] | |
| 6.00s call unit/runtime/zero/test_zero.py::TestZero3DictFwd::test[tuple] | |
| 6.00s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-True-False] | |
| 6.00s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_float_int8_quantization | |
| 6.00s call unit/monitor/test_monitor.py::TestCSVMonitor::test_empty_csv_monitor | |
| 6.00s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.01-0.01-19-False] | |
| 6.00s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.0001-1e-05-1-True] | |
| 5.99s call unit/runtime/test_multi_output_model.py::TestTwoOutputModel::test | |
| 5.99s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[True-1] | |
| 5.99s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-1-dtype1] | |
| 5.99s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[non_tensor4] | |
| 5.98s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_half_int4_quantization | |
| 5.98s call unit/compression/test_compression.py::TestCompression::test_conv1d_convertion | |
| 5.98s call unit/runtime/test_ds_config_dict.py::TestArgs::test_none_args | |
| 5.98s call unit/runtime/test_lr_schedulers.py::TestSchedulerOptimizerParity::test[WarmupDecayLR-params1] | |
| 5.97s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[False-2] | |
| 5.97s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output1] | |
| 5.97s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-1-dtype0] | |
| 5.97s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[True-2] | |
| 5.96s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-33] | |
| 5.96s call unit/runtime/half_precision/test_bf16.py::TestZeroSupportedClientOptimizer::test[FusedAdam] | |
| 5.96s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype1] | |
| 5.96s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-3-dtype1] | |
| 5.96s call unit/comm/test_dist.py::TestDistArgs::test[hello-icosahedron-1138-purple] | |
| 5.96s call unit/runtime/test_pld.py::TestNonPLDModel::test_non_pld_model | |
| 5.96s call unit/comm/test_dist.py::TestDistInit::test_already_init[True] | |
| 5.95s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs1[mask1] | |
| 5.95s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[True-False] | |
| 5.95s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[WarmupDecayLR-params1] | |
| 5.95s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[1-False] | |
| 5.95s call unit/runtime/comm/test_coalesced_collectives.py::TestAllToAllQuantReduceFallback::test_non_divisible | |
| 5.95s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[2] | |
| 5.95s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs2[mask0] | |
| 5.95s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs2[mask1] | |
| 5.94s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-15] | |
| 5.94s call unit/runtime/test_ds_config_dict.py::TestConfigLoad::test_json | |
| 5.94s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable_onecyclelr_steplr[None] | |
| 5.94s call unit/runtime/test_lr_schedulers.py::TestWarmupCosineLR::test_lr[100-10-0.1-0.2] | |
| 5.94s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-3-dtype0] | |
| 5.93s call unit/runtime/zero/test_zero_context_ancestry.py::TestDSInitWZinit::test | |
| 5.93s call unit/runtime/half_precision/test_fp16.py::TestAdamFP16ZeroOneCycleCompatibility::test[False-3] | |
| 5.93s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0.001-101] | |
| 5.93s call unit/runtime/test_runtime_utils.py::TestCheckOverflow::test[False] | |
| 5.93s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[Adam-3] | |
| 5.93s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype2] | |
| 5.92s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[Adam-2] | |
| 5.92s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-True-True] | |
| 5.92s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.0001-0.001-10-True] | |
| 5.92s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_arg_none[mask0] | |
| 5.92s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[False-1] | |
| 5.92s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output1] | |
| 5.92s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-True-True] | |
| 5.91s call unit/comm/test_dist.py::TestDistInit::test_already_init[False] | |
| 5.91s call unit/runtime/zero/test_zero.py::TestZero3DictFwd::test[dict] | |
| 5.91s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[WarmupLR-params0] | |
| 5.91s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-1-dtype0] | |
| 5.91s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-19] | |
| 5.91s call unit/elasticity/test_elastic.py::TestElasticConfigChanged::test | |
| 5.91s call unit/runtime/half_precision/test_fp16.py::TestFP16AdamTypes::test[True-Adam] | |
| 5.90s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-3-dtype2] | |
| 5.90s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs3[mask1] | |
| 5.90s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs3[mask1] | |
| 5.90s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[non_tensor3] | |
| 5.90s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_missing_amp_autocast[False] | |
| 5.90s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs2[mask0] | |
| 5.90s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-False-True] | |
| 5.90s call unit/runtime/zero/test_zero.py::TestIncorectAllgatherBucketSize::test[1000] | |
| 5.89s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-19] | |
| 5.89s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[None] | |
| 5.89s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-3-dtype1] | |
| 5.88s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[True] | |
| 5.88s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs1_outputs1[mask1] | |
| 5.88s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-3-dtype1] | |
| 5.88s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-True-False] | |
| 5.88s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_missing_amp_autocast[True] | |
| 5.87s call unit/comm/test_dist.py::TestDistInit::test_no_init[True] | |
| 5.87s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-3-dtype2] | |
| 5.87s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[2] | |
| 5.87s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-False-True] | |
| 5.86s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[4-False] | |
| 5.86s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype0] | |
| 5.86s call unit/runtime/half_precision/test_fp16.py::TestFP16AdamTypes::test[False-AdamW] | |
| 5.86s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[True] | |
| 5.86s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[non_tensor3] | |
| 5.86s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs1_outputs1[mask0] | |
| 5.86s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[non_tensor3] | |
| 5.86s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output2] | |
| 5.86s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs1[mask0] | |
| 5.85s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output3] | |
| 5.85s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[non_tensor3] | |
| 5.85s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[FAIL] | |
| 5.85s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[None] | |
| 5.85s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-10] | |
| 5.85s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-3-dtype0] | |
| 5.85s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_same_lrscheler_and_callable[None] | |
| 5.85s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs2[mask0] | |
| 5.84s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-1-dtype2] | |
| 5.84s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-1-dtype2] | |
| 5.84s call unit/runtime/zero/test_zero_context_return.py::TestReturnParam::test_stage_3_output_type[None] | |
| 5.83s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs1_outputs1[mask1] | |
| 5.83s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.001-0.001-10-False] | |
| 5.83s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-3-dtype0] | |
| 5.83s call unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_existing_latest | |
| 5.82s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype2] | |
| 5.82s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable_onecyclelr_steplr[_LRScheduler] | |
| 5.82s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs3[mask1] | |
| 5.82s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs1[mask1] | |
| 5.82s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable_onecyclelr_steplr[Callable] | |
| 5.82s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0-210] | |
| 5.82s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable[_LRScheduler] | |
| 5.81s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-10] | |
| 5.81s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[non_tensor4] | |
| 5.81s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-2-dtype0] | |
| 5.80s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[True-3] | |
| 5.80s call unit/elasticity/test_elastic.py::TestNonElasticBatchParams::test | |
| 5.80s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs1_outputs1[mask0] | |
| 5.80s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[1-True] | |
| 5.80s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Callable-_LRScheduler] | |
| 5.80s call unit/runtime/test_ds_config_dict.py::TestConfigLoad::test_dict | |
| 5.79s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[non_tensor3] | |
| 5.79s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[None] | |
| 5.79s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[True] | |
| 5.78s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestFused::test_some_overflow | |
| 5.78s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[False-1] | |
| 5.78s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[non_tensor4] | |
| 5.78s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-False-True] | |
| 5.78s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_arg_none[mask0] | |
| 5.78s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[4-True] | |
| 5.77s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-True-False] | |
| 5.77s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[non_tensor_output3] | |
| 5.77s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[None] | |
| 5.77s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[True] | |
| 5.76s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs3[mask0] | |
| 5.76s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_arg_none[mask1] | |
| 5.76s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs1[mask1] | |
| 5.75s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Optimizer-None] | |
| 5.75s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[True-1] | |
| 5.74s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_offload_optimizer[True] | |
| 5.74s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[2] | |
| 5.74s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[True] | |
| 5.74s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Optimizer-Callable] | |
| 5.73s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_throughput_calculation | |
| 5.73s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestUnfused::test_all_overflow | |
| 5.73s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-True-True] | |
| 5.73s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-False-resulting_optimizer0] | |
| 5.73s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_same_lrscheler_and_callable[Callable] | |
| 5.72s call unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_missing_latest | |
| 5.71s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-False-False] | |
| 5.71s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Callable-Callable] | |
| 5.71s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-3-dtype2] | |
| 5.70s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[non_tensor_output2] | |
| 5.70s call unit/comm/test_dist.py::TestGroupedDistTest::test_two[1138] | |
| 5.69s call unit/runtime/test_ds_config_dict.py::TestNoModel::test | |
| 5.69s call unit/comm/test_dist.py::TestDistributedFixture::test[4-32] | |
| 5.69s call unit/comm/test_dist.py::TestGroupedDistTest::test_one[1138] | |
| 5.69s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[2] | |
| 5.69s call unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test_flops_profiler_in_inference | |
| 5.68s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[None-Callable] | |
| 5.67s call unit/runtime/test_ds_initialize.py::TestClientOptimizer::test[None] | |
| 5.67s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[None] | |
| 5.66s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable[Callable] | |
| 5.66s call unit/runtime/zero/test_zero.py::TestZeroOffloadOptim::test[True] | |
| 5.66s call unit/runtime/zero/test_zero.py::TestIncorectAllgatherBucketSize::test[1001] | |
| 5.66s call unit/runtime/zero/test_zero_context.py::TestGatherUpdate::test | |
| 5.65s call unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_2 | |
| 5.65s call unit/checkpoint/test_zero_optimizer.py::TestZeRONonDistributed::test_chmod_exception_handling[3] | |
| 5.65s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[True-2] | |
| 5.64s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0-211] | |
| 5.64s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_arg_none[mask1] | |
| 5.64s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyPartition::test[True-3] | |
| 5.64s call unit/runtime/zero/test_zero_context.py::TestScatterGather::test | |
| 5.63s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[non_tensor4] | |
| 5.63s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-False-False] | |
| 5.62s call unit/comm/test_dist.py::TestDistributedFixture::test[2-32] | |
| 5.61s call unit/comm/test_dist.py::TestDistributedFixture::test[4-16] | |
| 5.61s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-2-dtype2] | |
| 5.61s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-2-dtype1] | |
| 5.60s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs2[mask1] | |
| 5.59s call unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_1 | |
| 5.54s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[None] | |
| 5.52s call unit/runtime/zero/test_zero_dynamic_class.py::TestNewClassDeclaredNestingInit::test_new_class_declared_nesting_init | |
| 5.52s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-1-dtype1] | |
| 5.51s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-1-dtype0] | |
| 5.51s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-1-dtype2] | |
| 5.50s call unit/runtime/zero/test_zero_context.py::TestMiCSGatheredParametersFree::test | |
| 5.48s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_subclass_param | |
| 5.47s call unit/runtime/zero/test_zero_dynamic_class.py::TestNewClassDeclaredInsideNestingInit::test_new_class_declared_inside_nesting_init | |
| 5.44s setup unit/comm/test_dist.py::TestDistributedFixture::test[4-32] | |
| 5.37s setup unit/comm/test_dist.py::TestDistributedFixture::test[2-16] | |
| 5.37s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_scattered_init_dist | |
| 5.28s call unit/runtime/zero/test_zero_context_ancestry.py::TestSerialParamInit::test_subclass_param_init | |
| 5.23s setup unit/comm/test_dist.py::TestDistributedFixture::test[2-32] | |
| 5.21s call unit/runtime/zero/test_zero_context.py::TestZeroGatheredParametersFree::test | |
| 5.16s setup unit/comm/test_dist.py::TestDistributedFixture::test[4-16] | |
| 5.12s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_scatter_halftype | |
| 4.53s call unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[False] | |
| 4.34s call unit/comm/test_dist.py::TestDistInit::test_no_init[False] | |
| 2.13s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-full-dtype0] | |
| 2.12s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-True-resulting_optimizer6] | |
| 1.99s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-full-dtype1] | |
| 1.94s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-full-dtype2] | |
| 1.92s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-local-dtype2] | |
| 1.92s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-full-dtype1] | |
| 1.91s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-full-False] | |
| 1.90s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-full-dtype0] | |
| 1.88s call unit/ops/adam/test_hybrid_adam.py::TestHybridAdam::test_hybrid_adam_equal[8-fp16] | |
| 1.88s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-local-False] | |
| 1.88s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-False-resulting_optimizer2] | |
| 1.88s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-full-False] | |
| 1.86s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-local-dtype2] | |
| 1.85s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-local-dtype1] | |
| 1.83s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-full-dtype2] | |
| 1.83s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-local-True] | |
| 1.82s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-full-True] | |
| 1.82s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-True-resulting_optimizer4] | |
| 1.82s call unit/ops/lion/test_cpu_lion.py::TestCPULion::test_fused_lion_equal[64-fp16] | |
| 1.82s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-local-dtype0] | |
| 1.82s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-local-False] | |
| 1.81s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-local-True] | |
| 1.80s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-full-True] | |
| 1.79s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-local-dtype1] | |
| 1.79s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-full-dtype1] | |
| 1.78s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-local-dtype0] | |
| 1.78s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-1-full-dtype2] | |
| 1.76s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-fp32-zero3] | |
| 1.71s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[64-fp16] | |
| 1.64s call unit/ops/lion/test_lion.py::TestLionConfigs::test[Lion-True-DeepSpeedCPULion] | |
| 1.64s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-True-resulting_optimizer14] | |
| 1.60s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-False-resulting_optimizer10] | |
| 1.59s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-bf16-zero3] | |
| 1.55s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-bf16-zero2] | |
| 1.54s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-full-dtype0] | |
| 1.49s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-2-full-dtype2] | |
| 1.48s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-2-full-dtype1] | |
| 1.47s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-2-full-dtype0] | |
| 1.47s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-1-full-dtype0] | |
| 1.46s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-1-full-dtype1] | |
| 1.45s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-local-dtype0] | |
| 1.40s call unit/runtime/test_mup_optimizers.py::TestMuPOptimizers::test[False-MuAdam-Adam] | |
| 1.39s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-fp16-zero3] | |
| 1.37s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-bf16-zero3] | |
| 1.36s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-1-full-False] | |
| 1.36s call unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[True-True-True] | |
| 1.36s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-2-full-False] | |
| 1.36s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-2-full-True] | |
| 1.35s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-1-full-True] | |
| 1.35s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-local-dtype1] | |
| 1.35s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-local-dtype2] | |
| 1.32s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-full-dtype2] | |
| 1.32s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-False-False-resulting_optimizer8] | |
| 1.32s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-bf16-zero3] | |
| 1.32s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-bf16-zero3] | |
| 1.31s call unit/ops/aio/test_aio.py::TestRead::test_async_read[True-False-True-False] | |
| 1.28s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp16-zero3] | |
| 1.28s call unit/launcher/test_run.py::test_parser_multinode | |
| 1.26s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_bf16_fragments[False] | |
| 1.23s call unit/ops/aio/test_aio.py::TestRead::test_async_read[False-True-False-True] | |
| 1.22s call unit/runtime/zero/test_zero_leaf_module.py::TestSetZ3LeafModule::test_choose_module_by_rank | |
| 1.21s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp32-zero3] | |
| 1.20s call unit/ops/aio/test_aio.py::TestRead::test_parallel_read[True-True-True] | |
| 1.19s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-bf16-zero1] | |
| 1.19s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-2-full-dtype0] | |
| 1.18s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-fp32-zero3] | |
| 1.18s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-True-False-resulting_optimizer3] | |
| 1.18s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-full-False] | |
| 1.17s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-fp32-zero3] | |
| 1.16s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-full-True] | |
| 1.15s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-bf16-zero1] | |
| 1.15s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-local-False] | |
| 1.14s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-local-True] | |
| 1.10s call unit/runtime/zero/test_zero_leaf_module.py::TestSetZ3LeafModule::test_no_grad_input_error | |
| 1.08s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-fp16-zero3] | |
| 1.07s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-fp16-zero3] | |
| 1.04s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-False-True-True] | |
| 1.02s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-2-full-dtype1] | |
| 1.02s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-bf16-zero2] | |
| 1.02s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-fp16-zero1] | |
| 1.01s call unit/ops/adam/test_hybrid_adam.py::TestHybridAdam::test_hybrid_adam_equal[8-bf16] | |
| 1.00s call unit/ops/aio/test_aio.py::TestRead::test_parallel_read[False-False-True] | |
| 1.00s call unit/runtime/test_mup_optimizers.py::TestMuPOptimizers::test[False-MuSGD-SGD] | |
| (2601 durations < 1s hidden. Use -vv to show these durations.) | |
| [36m[1m===================================================================================== short test summary info =====================================================================================[0m | |
| [31mFAILED[0m tests/unit/inference/quantization/test_intX_quantization.py::[1mTestQuantizedInt::test_zero3_int4_post_init_quant_nvme_offload[0m - AssertionError | |
| [31mFAILED[0m tests/unit/inference/quantization/test_intX_quantization.py::[1mTestQuantizedInt::test_zero3_int4_quantized_initialization_nvme_offload[0m - AssertionError | |
| [31m===================================================== [31m[1m2 failed[0m, [32m1052 passed[0m, [33m129 skipped[0m, [33m4426 deselected[0m, [33m172 warnings[0m[31m in 6045.42s (1:40:45)[0m[31m =====================================================[0m | |
| (at easybuild/tools/run.py:695 in parse_cmd_output) | |
| == 2024-11-01 18:14:00,357 build_log.py:267 INFO ... (took 1 hour 48 mins 15 secs) | |
| == 2024-11-01 18:14:00,357 build_log.py:267 INFO ... (took 1 hour 48 mins 35 secs) | |
| == 2024-11-01 18:14:00,357 filetools.py:2025 INFO Removing lock /apps/Test/software/.locks/_apps_Test_software_DeepSpeed_0.14.5-foss-2023a-CUDA-12.1.1.lock... | |
| == 2024-11-01 18:14:00,364 filetools.py:385 INFO Path /apps/Test/software/.locks/_apps_Test_software_DeepSpeed_0.14.5-foss-2023a-CUDA-12.1.1.lock successfully removed. | |
| == 2024-11-01 18:14:00,364 filetools.py:2029 INFO Lock removed: /apps/Test/software/.locks/_apps_Test_software_DeepSpeed_0.14.5-foss-2023a-CUDA-12.1.1.lock | |
| == 2024-11-01 18:14:00,364 easyblock.py:4297 WARNING build failed (first 300 chars): cmd "export PATH=/cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-ehjcjfxy/tmpo3g409wd/bin:$PATH PYTHONPATH=/cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-ehjcjfxy/tmpo3g409wd/lib/python3.11/site-packages:$PYTHONPATH && ln -s $PWD/tests/ ../tests && cd ../ && pytest tests/unit/ -k "not TestTensorBoard and not Te | |
| == 2024-11-01 18:14:00,365 easyblock.py:326 INFO Closing log for application name DeepSpeed version 0.14.5 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment