Created
October 29, 2024 15:34
-
-
Save VRehnberg/4028207276394d53328a44514c2b03f4 to your computer and use it in GitHub Desktop.
(partial) EasyBuild log for failed build of /cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-0e6vf5o0/files_pr21438/d/DeepSpeed/DeepSpeed-0.14.5-foss-2023a-CUDA-12.1.1.eb (PR(s) #21438) (easyblock PR(s) #3450)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 6.33s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-False-True] | |
| 6.33s call unit/runtime/test_lr_schedulers.py::TestWarmupCosineLR::test_lr[200-20-0.1-0.2] | |
| 6.32s call unit/comm/test_dist.py::TestInit::test | |
| 6.32s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0.001-100] | |
| 6.31s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyGrad::test[2] | |
| 6.31s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[False-False] | |
| 6.31s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[None] | |
| 6.31s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[None] | |
| 6.30s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_namedtuple] | |
| 6.30s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_prefetching[True] | |
| 6.30s call unit/runtime/half_precision/test_fp16.py::TestLambFP32GradClip::test | |
| 6.30s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-3-dtype2] | |
| 6.30s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[None-None] | |
| 6.30s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs3[mask0] | |
| 6.30s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype2] | |
| 6.29s call unit/runtime/sparse_tensor/test_averaging_sparse_gradients.py::TestSparseAdam::test | |
| 6.28s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[None] | |
| 6.27s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-False-True] | |
| 6.27s call unit/runtime/test_multi_output_model.py::TestThreeOutputModel::test | |
| 6.26s call unit/runtime/zero/test_zero.py::TestZero3RepeatForwardLoop::test[False] | |
| 6.26s call unit/runtime/zero/test_zero.py::TestZeroToFP32::test_1_param_group[False-2] | |
| 6.26s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-False-False] | |
| 6.26s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-False-False] | |
| 6.25s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[non_tensor3] | |
| 6.25s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[0.001-0.1-0-21-21] | |
| 6.24s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpoint::test_load_module_only[0] | |
| 6.24s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyGrad::test[1] | |
| 6.24s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-1-dtype0] | |
| 6.23s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_NamedTuple] | |
| 6.23s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype0] | |
| 6.23s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[True] | |
| 6.23s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[non_tensor_output2] | |
| 6.23s call unit/runtime/test_runtime_utils.py::TestCheckOverflow::test[True] | |
| 6.23s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[2] | |
| 6.22s call unit/runtime/half_precision/test_fp16.py::TestAdamFP32EmptyGrad::test | |
| 6.22s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_fp16_enabled[True] | |
| 6.22s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-3-dtype2] | |
| 6.22s call unit/runtime/half_precision/test_fp16.py::TestLambFP16::test_empty_grad | |
| 6.21s call unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fp32_optimizer | |
| 6.21s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[3] | |
| 6.21s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[2] | |
| 6.21s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-3-dtype1] | |
| 6.21s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-True-True] | |
| 6.20s call unit/runtime/half_precision/test_fp16.py::TestLambFP16::test__basic | |
| 6.20s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[True] | |
| 6.19s call unit/compression/test_compression.py::TestCompression::test_linear_layer_compress | |
| 6.19s call unit/runtime/zero/test_zero_context_return.py::TestReturnParam::test_ext_param_return | |
| 6.18s call unit/runtime/test_runtime_utils.py::TestClipGradNorm::test_clipped_val | |
| 6.18s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs3[mask0] | |
| 6.18s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[Adam-1] | |
| 6.18s call unit/runtime/test_ds_config_dict.py::TestDeprecatedDeepScaleConfig::test | |
| 6.18s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_param_persistence_threshold[10] | |
| 6.17s call unit/elasticity/test_elastic.py::TestNonElasticBatchParams::test | |
| 6.17s call unit/runtime/half_precision/test_fp16.py::TestZeroStaticScale::test[False-2] | |
| 6.17s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[WarmupDecayLR-params1] | |
| 6.17s call unit/runtime/comm/test_coalesced_collectives.py::TestAllToAllQuantReduceFallback::test_non_divisible | |
| 6.17s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Optimizer-Callable] | |
| 6.17s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_offload_optimizer[False] | |
| 6.17s call unit/runtime/zero/test_zero.py::TestZero3RepeatForwardLoop::test[True] | |
| 6.17s call unit/runtime/half_precision/test_fp16.py::TestAdamFP16ZeroOneCycleCompatibility::test[False-1] | |
| 6.16s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-False-False] | |
| 6.16s call unit/runtime/half_precision/test_bf16.py::TestZeroSupportedClientOptimizer::test[Adam] | |
| 6.16s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype0] | |
| 6.15s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-3-dtype0] | |
| 6.15s call unit/runtime/half_precision/test_bf16.py::TestZeroEmptyGrad::test | |
| 6.15s call unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[True] | |
| 6.15s call unit/runtime/test_ds_initialize.py::TestNoOptim::test[0] | |
| 6.15s call unit/runtime/zero/test_zero.py::TestZeroUnbalancedGradients::test[2] | |
| 6.15s call unit/runtime/half_precision/test_fp16.py::TestZeroStaticScale::test[False-1] | |
| 6.14s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_save_exclude_custom_frozen_weights[1] | |
| 6.14s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[IGNORE] | |
| 6.14s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-1-dtype1] | |
| 6.14s call unit/comm/test_dist.py::TestDistArgs::test[hello-icosahedron-1138-purple] | |
| 6.14s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[non_tensor4] | |
| 6.14s call unit/runtime/test_lr_schedulers.py::TestSchedulerOptimizerParity::test[LRRangeTest-params3] | |
| 6.14s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_zero_grad[True] | |
| 6.14s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-1-dtype1] | |
| 6.13s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[non_tensor3] | |
| 6.13s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-False-True] | |
| 6.13s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_List] | |
| 6.13s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0-210] | |
| 6.13s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_save_exclude_custom_frozen_weights[2] | |
| 6.13s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Callable-None] | |
| 6.13s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-1-dtype0] | |
| 6.13s call unit/runtime/half_precision/test_fp16.py::TestFP16AdamTypes::test[True-Adam] | |
| 6.13s call unit/runtime/zero/test_zero_leaf_module.py::TestSetZ3LeafModule::test_choose_module_by_counter | |
| 6.12s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_missing_amp_autocast[True] | |
| 6.12s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_zero_grad[False] | |
| 6.11s call unit/runtime/zero/test_zero_context_return.py::TestReturnParam::test_stage_3_output_type[tensor] | |
| 6.11s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[False-True] | |
| 6.11s call unit/runtime/half_precision/test_fp16.py::TestAdamwFP16EmptyGrad::test | |
| 6.11s call unit/runtime/zero/test_zero_dynamic_class.py::TestNewClassDeclaredNestingInit::test_new_class_declared_nesting_init | |
| 6.11s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs3[mask1] | |
| 6.11s call unit/runtime/zero/test_zero.py::TestZero3InitForParentWeightInitialization::test | |
| 6.10s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output1] | |
| 6.10s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.0001-0.001-10-True] | |
| 6.10s call unit/monitor/test_monitor.py::TestCSVMonitor::test_empty_csv_monitor | |
| 6.09s call unit/runtime/half_precision/test_bf16.py::TestZeroSupportedClientOptimizer::test[FusedAdam] | |
| 6.09s call unit/linear/test_ctx.py::TestInitTransformers::test_config_init | |
| 6.09s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[0] | |
| 6.09s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[None] | |
| 6.09s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-1-dtype1] | |
| 6.09s call unit/runtime/test_ds_config_dict.py::TestDistInit::test | |
| 6.09s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-True-True] | |
| 6.09s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestFused::test_no_overflow | |
| 6.09s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Callable-_LRScheduler] | |
| 6.09s call unit/comm/test_dist.py::TestDistInit::test_already_init[False] | |
| 6.08s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-False] | |
| 6.08s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_prefetching[False] | |
| 6.07s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs2[mask0] | |
| 6.07s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs2[mask1] | |
| 6.06s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_arg_none[mask1] | |
| 6.06s call unit/runtime/half_precision/test_fp16.py::TestAdamFP16ZeroOneCycleCompatibility::test[False-2] | |
| 6.06s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[0] | |
| 6.05s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[LRRangeTest-params3] | |
| 6.04s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[False-1] | |
| 6.04s call unit/runtime/zero/test_zero.py::TestZeroPartitionCache::test_training_partition_cache[True] | |
| 6.04s call unit/runtime/half_precision/test_bf16.py::TestAdamBF16ZeroOneCycleCompatibility::test | |
| 6.04s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-15] | |
| 6.03s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[None-_LRScheduler] | |
| 6.03s call unit/runtime/comm/test_coalesced_collectives.py::TestReduceScatterCoalesced::test_single_input | |
| 6.03s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-3-dtype1] | |
| 6.03s call unit/utils/test_init_on_device.py::TestOnDevice::test_on_device[cuda:0] | |
| 6.03s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs1_outputs1[mask1] | |
| 6.03s call unit/runtime/test_lr_schedulers.py::TestSchedulerOptimizerParity::test[WarmupLR-params0] | |
| 6.02s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-3-dtype0] | |
| 6.02s call unit/runtime/test_multi_output_model.py::TestTwoOutputModel::test | |
| 6.02s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-19] | |
| 6.02s call unit/runtime/zero/test_zero.py::TestZeroUnbalancedGradients::test[3] | |
| 6.01s call unit/checkpoint/test_zero_optimizer.py::TestZeRONonDistributed::test_chmod_exception_handling[3] | |
| 6.01s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-1-dtype0] | |
| 6.01s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-33] | |
| 6.01s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable_onecyclelr_steplr[Callable] | |
| 6.01s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[FusedAdam-3] | |
| 6.00s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-False-True] | |
| 6.00s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs1_outputs1[mask1] | |
| 6.00s call unit/runtime/zero/test_zero.py::TestZero3DictFwd::test[dict] | |
| 6.00s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[False-1] | |
| 6.00s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-10] | |
| 5.99s call unit/comm/test_dist.py::TestDistributedFixture::test[2-32] | |
| 5.99s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-10] | |
| 5.99s call unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test | |
| 5.99s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-3-dtype0] | |
| 5.98s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-1-dtype1] | |
| 5.98s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable[_LRScheduler] | |
| 5.98s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_float_int8_quantization | |
| 5.98s setup unit/comm/test_dist.py::TestDistributedFixture::test[4-16] | |
| 5.98s call unit/runtime/test_ds_initialize.py::TestClientOptimizer::test[Callable] | |
| 5.97s call unit/checkpoint/test_zero_optimizer.py::TestZeRONonDistributed::test_chmod_exception_handling[2] | |
| 5.97s call unit/runtime/zero/test_zero.py::TestZero3DictFwd::test[list] | |
| 5.97s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[0.001-0.1-0.1-21-21] | |
| 5.97s call unit/moe/test_moe.py::TestTopk::test | |
| 5.97s call unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[True] | |
| 5.96s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[log-10] | |
| 5.96s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-3-dtype1] | |
| 5.95s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.01-0.01-19-True] | |
| 5.95s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-True-True] | |
| 5.95s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[True-False] | |
| 5.95s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs1[mask1] | |
| 5.95s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[2] | |
| 5.94s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[1e-05-0.01-0.001-10-100] | |
| 5.94s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[non_tensor3] | |
| 5.94s call unit/runtime/half_precision/test_fp16.py::TestZeroStaticScale::test[False-3] | |
| 5.94s call unit/runtime/test_runtime_utils.py::TestClipGradNorm::test_gather | |
| 5.93s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-True-True] | |
| 5.93s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-False-False] | |
| 5.93s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-3-dtype0] | |
| 5.93s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_float_int4_quantization | |
| 5.93s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[OneCycle-params2] | |
| 5.92s call unit/runtime/test_lr_schedulers.py::TestSchedulerOptimizerParity::test[WarmupDecayLR-params1] | |
| 5.92s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-3-dtype2] | |
| 5.92s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[True] | |
| 5.92s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[None] | |
| 5.92s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs2[mask0] | |
| 5.92s call unit/runtime/half_precision/test_fp16.py::TestAdamwFP16Basic::test | |
| 5.92s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[0.9] | |
| 5.91s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_arg_none[mask1] | |
| 5.91s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[Adam-2] | |
| 5.91s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs1_outputs1[mask0] | |
| 5.91s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-19] | |
| 5.90s call unit/runtime/half_precision/test_bf16.py::TestZeroAllowUntestedOptimizer::test | |
| 5.90s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-False-False] | |
| 5.90s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs2[mask1] | |
| 5.89s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-3-dtype0] | |
| 5.89s call unit/runtime/test_ds_config_dict.py::TestConfigLoad::test_dict | |
| 5.89s call unit/comm/test_dist.py::TestDistInit::test_already_init[None] | |
| 5.89s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-1-dtype0] | |
| 5.88s call unit/runtime/test_ds_initialize.py::TestConfigOptimizer::test[False] | |
| 5.88s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[1e-05-0.01-0.001-10-101] | |
| 5.88s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[1.0] | |
| 5.88s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[1-False] | |
| 5.88s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_half_int4_quantization | |
| 5.88s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[FAIL] | |
| 5.88s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype1] | |
| 5.88s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs3[mask0] | |
| 5.87s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.001-0.001-10-False] | |
| 5.87s call unit/elasticity/test_elastic.py::TestNonElasticBatchParamsWithOverride::test | |
| 5.87s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.0001-1e-05-1-True] | |
| 5.87s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[False-2] | |
| 5.87s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[WarmupLR-params0] | |
| 5.87s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[FusedAdam-1] | |
| 5.87s call unit/comm/test_dist.py::TestDistInitNoEnv::test | |
| 5.87s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[1e-05-0.1-0-10-0] | |
| 5.87s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-33] | |
| 5.87s call unit/checkpoint/test_zero_optimizer.py::TestZeRONonDistributed::test_chmod_exception_handling[1] | |
| 5.87s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-3-dtype0] | |
| 5.86s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-1-dtype2] | |
| 5.86s call unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[False] | |
| 5.86s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-10] | |
| 5.86s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-1-dtype0] | |
| 5.86s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_ext_param_getattr | |
| 5.85s call unit/runtime/test_lr_schedulers.py::TestWarmupCosineLR::test_lr[100-10-0.1-0.2] | |
| 5.85s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-19] | |
| 5.85s call unit/runtime/zero/test_zero.py::TestZeroPartitionCache::test_training_partition_cache[False] | |
| 5.84s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[log-15] | |
| 5.83s call unit/monitor/test_monitor.py::TestCSVMonitor::test_csv_monitor | |
| 5.83s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-1-dtype2] | |
| 5.83s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-15] | |
| 5.83s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-True-False] | |
| 5.82s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[4-False] | |
| 5.81s call unit/runtime/zero/test_zero.py::TestZeroOffloadOptim::test[False] | |
| 5.81s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-33] | |
| 5.81s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_disable_autocast_linear[False] | |
| 5.81s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype0] | |
| 5.81s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[True-True] | |
| 5.80s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestFused::test_some_overflow | |
| 5.80s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-True-True] | |
| 5.80s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[WARN] | |
| 5.80s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable_onecyclelr_steplr[None] | |
| 5.80s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Callable-Callable] | |
| 5.80s call unit/runtime/comm/test_coalesced_collectives.py::TestReduceScatterCoalescedTensorSmallerThanWorldSize::test | |
| 5.80s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs2[mask1] | |
| 5.79s call unit/runtime/test_ds_config_dict.py::TestBasicConfig::test_accelerator | |
| 5.79s call unit/runtime/zero/test_zero_nesting_init.py::TestNestingInit::test_nesting_init | |
| 5.79s call unit/runtime/half_precision/test_fp16.py::TestFP16AdamTypes::test[True-AdamW] | |
| 5.79s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-3-dtype0] | |
| 5.79s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-1-dtype2] | |
| 5.79s call unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_2 | |
| 5.79s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0.001-101] | |
| 5.79s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-32-8-2-True] | |
| 5.79s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyPartition::test[True-3] | |
| 5.79s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[False-2] | |
| 5.78s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[None-Callable] | |
| 5.78s call unit/elasticity/test_elastic.py::TestElasticConfigChanged::test | |
| 5.78s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype2] | |
| 5.78s call unit/comm/test_dist.py::TestDistInit::test_no_init[True] | |
| 5.78s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_arg_none[mask0] | |
| 5.78s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-True-True] | |
| 5.78s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable[None] | |
| 5.78s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_offload_optimizer[True] | |
| 5.78s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[2] | |
| 5.78s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestFused::test_all_overflow | |
| 5.78s call unit/runtime/test_ds_initialize.py::TestConfigOptimizer::test[True] | |
| 5.78s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-1-dtype2] | |
| 5.77s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_disable_autocast_linear[True] | |
| 5.76s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-True-False] | |
| 5.76s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[non_tensor4] | |
| 5.76s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[0.1] | |
| 5.76s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output2] | |
| 5.76s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_arg_none[mask1] | |
| 5.75s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable_onecyclelr_steplr[_LRScheduler] | |
| 5.75s call unit/runtime/comm/test_coalesced_collectives.py::TestReduceScatterCoalesced::test_two_inputs | |
| 5.75s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestUnfused::test_all_overflow | |
| 5.75s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0-211] | |
| 5.74s call unit/comm/test_dist.py::TestDistInit::test_already_init[True] | |
| 5.74s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-33-17-2-False] | |
| 5.74s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[Adam-3] | |
| 5.74s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs3[mask1] | |
| 5.74s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[None] | |
| 5.73s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[non_tensor3] | |
| 5.73s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-True-False] | |
| 5.72s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_same_lrscheler_and_callable[None] | |
| 5.72s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs1[mask1] | |
| 5.72s call unit/comm/test_dist.py::TestDistributedFixture::test[4-16] | |
| 5.72s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[True] | |
| 5.72s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-1-dtype2] | |
| 5.72s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unknown_tag_validation | |
| 5.71s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[non_tensor4] | |
| 5.71s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype2] | |
| 5.71s call unit/comm/test_dist.py::TestDistInit::test_no_init[None] | |
| 5.71s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-False-True] | |
| 5.71s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-1-dtype1] | |
| 5.71s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable[Callable] | |
| 5.71s call unit/runtime/test_ds_config_dict.py::TestArgs::test_no_args | |
| 5.71s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Optimizer-None] | |
| 5.70s call unit/runtime/test_pld.py::TestNonPLDModel::test_non_pld_model | |
| 5.70s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[1-True] | |
| 5.69s call unit/runtime/test_ds_config_dict.py::TestInitNoOptimizer::test | |
| 5.69s call unit/runtime/zero/test_zero_dynamic_class.py::TestNewClassDeclaredInsideNestingInit::test_new_class_declared_inside_nesting_init | |
| 5.69s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[2] | |
| 5.69s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs1[mask0] | |
| 5.69s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_arg_none[mask0] | |
| 5.69s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[True] | |
| 5.68s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-False-resulting_optimizer0] | |
| 5.68s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs1_outputs1[mask0] | |
| 5.68s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[non_tensor4] | |
| 5.68s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[4-True] | |
| 5.68s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[non_tensor_output1] | |
| 5.68s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestUnfused::test_some_overflow | |
| 5.68s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[True] | |
| 5.67s call unit/runtime/test_ds_config_dict.py::TestBasicConfig::test_check_version | |
| 5.67s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[non_tensor4] | |
| 5.67s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[non_tensor3] | |
| 5.67s call unit/runtime/test_ds_initialize.py::TestClientOptimizer::test[None] | |
| 5.67s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-3-dtype2] | |
| 5.66s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[non_tensor3] | |
| 5.66s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[2] | |
| 5.66s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs1[mask0] | |
| 5.65s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_arg_none[mask0] | |
| 5.65s call unit/runtime/zero/test_zero.py::TestZeroOffloadOptim::test[True] | |
| 5.64s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestUnfused::test_no_overflow | |
| 5.64s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs2[mask0] | |
| 5.64s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[True-3] | |
| 5.64s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[2] | |
| 5.63s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_same_lrscheler_and_callable[Callable] | |
| 5.63s call unit/runtime/test_ds_config_dict.py::TestConfigLoad::test_json | |
| 5.63s call unit/runtime/test_ds_config_dict.py::TestArgs::test_none_args | |
| 5.62s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[None] | |
| 5.62s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[non_tensor4] | |
| 5.62s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output1] | |
| 5.61s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-1-dtype2] | |
| 5.61s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[False-3] | |
| 5.60s call unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test_flops_profiler_in_inference | |
| 5.60s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output3] | |
| 5.60s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs1[mask1] | |
| 5.60s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[True-1] | |
| 5.59s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Optimizer-_LRScheduler] | |
| 5.59s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[None] | |
| 5.59s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[non_tensor_output3] | |
| 5.59s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_same_lrscheler_and_callable[_LRScheduler] | |
| 5.58s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs1_outputs1[mask0] | |
| 5.58s call unit/ops/accelerators/test_accelerator_backward.py::TestCUDABackward::test_backward[8-1600-128-25-3-True-True-0.05] | |
| 5.58s call unit/runtime/zero/test_zero.py::TestIncorectAllgatherBucketSize::test[1001] | |
| 5.58s call unit/runtime/test_ds_config_dict.py::TestNoModel::test | |
| 5.57s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs1[mask0] | |
| 5.56s call unit/runtime/test_ds_config_dict.py::TestConfigLoad::test_hjson | |
| 5.56s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs1_outputs1[mask1] | |
| 5.56s call unit/comm/test_dist.py::TestDistributedFixture::test[4-32] | |
| 5.55s call unit/runtime/test_ds_initialize.py::TestClientOptimizer::test[Optimizer] | |
| 5.55s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output2] | |
| 5.53s call unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_missing_latest | |
| 5.52s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_throughput_calculation | |
| 5.52s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[True-2] | |
| 5.51s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-2-dtype1] | |
| 5.51s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-2-dtype2] | |
| 5.49s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyPartition::test[False-3] | |
| 5.49s call unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_1 | |
| 5.46s call unit/runtime/zero/test_zero_context.py::TestGatherUpdate::test | |
| 5.44s call unit/comm/test_dist.py::TestDistributedFixture::test[2-16] | |
| 5.43s call unit/runtime/zero/test_zero_context_ancestry.py::TestDSInitWZinit::test | |
| 5.41s setup unit/comm/test_dist.py::TestDistributedFixture::test[4-32] | |
| 5.41s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-2-dtype0] | |
| 5.40s call unit/ops/accelerators/test_accelerator_backward.py::TestCUDABackward::test_backward[8-1600-128-2-3-True-True-0.05] | |
| 5.39s setup unit/comm/test_dist.py::TestDistributedFixture::test[2-16] | |
| 5.39s call unit/ops/accelerators/test_accelerator_forward.py::TestCUDAForwardSmallBatchSize::test_forward_with_small_bsz[8-7-1024-512-16-3-False-True] | |
| 5.36s call unit/runtime/zero/test_zero_context_ancestry.py::TestSerialParamInit::test_subclass_param_init | |
| 5.29s call unit/runtime/zero/test_zero_context.py::TestScatterGather::test | |
| 5.27s call unit/runtime/zero/test_zero_context.py::TestZeroGatheredParametersFree::test | |
| 5.27s call unit/ops/accelerators/test_accelerator_forward.py::TestCUDAForwardSmallBatchSize::test_forward_with_small_bsz[8-7-1024-512-16-3-True-True] | |
| 5.24s call unit/runtime/zero/test_zero_context.py::TestMiCSGatheredParametersFree::test | |
| 5.21s call unit/ops/accelerators/test_accelerator_forward.py::TestCUDAForwardSmallBatchSize::test_forward_with_small_bsz[8-3-1024-512-16-3-True-False] | |
| 5.21s call unit/ops/accelerators/test_accelerator_forward.py::TestCUDAForwardSmallBatchSize::test_forward_with_small_bsz[8-3-1024-512-16-3-False-False] | |
| 5.20s call unit/ops/accelerators/test_accelerator_backward.py::TestCUDABackward::test_backward[64-160-128-2-24-False-True-0.2] | |
| 5.18s setup unit/comm/test_dist.py::TestDistributedFixture::test[2-32] | |
| 5.05s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_scattered_init_dist | |
| 5.04s call unit/ops/accelerators/test_accelerator_backward.py::TestCUDABackward::test_backward[8-160-128-2-3-True-True-0.1] | |
| 5.01s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_scatter_halftype | |
| 4.99s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_subclass_param | |
| 4.41s call unit/comm/test_dist.py::TestDistInit::test_no_init[False] | |
| 4.34s call unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[False] | |
| 2.64s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-full-False] | |
| 2.39s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-full-dtype0] | |
| 2.38s call unit/ops/aio/test_aio.py::TestRead::test_parallel_read[False-True-True] | |
| 2.31s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-full-False] | |
| 2.22s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-full-dtype1] | |
| 2.14s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-full-dtype2] | |
| 2.09s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-1-full-dtype1] | |
| 1.98s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-full-dtype0] | |
| 1.96s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-local-True] | |
| 1.96s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-full-dtype1] | |
| 1.95s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-full-True] | |
| 1.94s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-True-resulting_optimizer14] | |
| 1.93s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-local-dtype2] | |
| 1.93s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-local-dtype1] | |
| 1.90s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-full-True] | |
| 1.89s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-full-dtype2] | |
| 1.88s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-local-dtype1] | |
| 1.88s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-local-dtype0] | |
| 1.87s call unit/ops/adam/test_hybrid_adam.py::TestHybridAdam::test_hybrid_adam_equal[8-fp16] | |
| 1.87s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-local-True] | |
| 1.86s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-local-dtype1] | |
| 1.84s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-local-False] | |
| 1.84s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-local-dtype2] | |
| 1.84s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-local-False] | |
| 1.80s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-local-dtype0] | |
| 1.78s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-True-resulting_optimizer6] | |
| 1.77s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-False-resulting_optimizer10] | |
| 1.73s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-1-full-dtype1] | |
| 1.72s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-2-full-dtype0] | |
| 1.72s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[64-fp16] | |
| 1.71s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-1-full-dtype0] | |
| 1.68s call unit/ops/lion/test_cpu_lion.py::TestCPULion::test_fused_lion_equal[64-fp16] | |
| 1.67s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_bf16_fragments[False] | |
| 1.65s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1048576-fp32] | |
| 1.63s call unit/ops/lion/test_lion.py::TestLionConfigs::test[Lion-True-DeepSpeedCPULion] | |
| 1.59s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-local-dtype2] | |
| 1.58s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1024-fp16] | |
| 1.56s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-False-resulting_optimizer2] | |
| 1.56s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1048576-bf16] | |
| 1.55s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[22-fp16] | |
| 1.55s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1048576-fp32] | |
| 1.54s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[128-fp32] | |
| 1.54s call unit/ops/lion/test_cpu_lion.py::TestCPULion::test_fused_lion_equal[1048576-bf16] | |
| 1.54s call unit/ops/lion/test_cpu_lion.py::TestCPULion::test_fused_lion_equal[1048576-fp32] | |
| 1.52s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1048576-bf16] | |
| 1.52s call unit/ops/lion/test_cpu_lion.py::TestCPULion::test_fused_lion_equal[1048576-fp16] | |
| 1.51s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1048576-fp16] | |
| 1.51s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[128-bf16] | |
| 1.50s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[22-fp32] | |
| 1.49s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[22-bf16] | |
| 1.49s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-2-full-dtype2] | |
| 1.49s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[128-bf16] | |
| 1.48s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-full-dtype0] | |
| 1.48s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1024-bf16] | |
| 1.48s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[128-fp32] | |
| 1.47s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[64-fp32] | |
| 1.47s call unit/ops/lion/test_cpu_lion.py::TestCPULion::test_fused_lion_equal[1024-fp32] | |
| 1.47s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[64-bf16] | |
| 1.47s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-local-dtype0] | |
| 1.47s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-2-full-True] | |
| 1.47s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1024-bf16] | |
| 1.47s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-full-dtype1] | |
| 1.47s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[22-fp32] | |
| 1.47s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1024-fp32] | |
| 1.46s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[128-fp16] | |
| 1.46s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1024-fp32] | |
| 1.45s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[22-bf16] | |
| 1.44s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-2-full-dtype1] | |
| 1.41s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-1-full-dtype2] | |
| 1.40s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-local-True] | |
| 1.39s call unit/ops/aio/test_aio.py::TestRead::test_async_read[True-False-True-True] | |
| 1.38s call unit/ops/aio/test_aio.py::TestRead::test_async_read[False-False-True-True] | |
| 1.37s call unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[False-True-True] | |
| 1.37s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-1-full-False] | |
| 1.37s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp32-zero3] | |
| 1.37s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-1-full-True] | |
| 1.35s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-2-full-False] | |
| 1.30s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-full-dtype2] | |
| 1.24s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-True-False-True] | |
| 1.24s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[64-fp32] | |
| 1.22s call unit/ops/aio/test_aio.py::TestRead::test_async_read[False-True-True-False] | |
| 1.21s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-full-False] | |
| 1.20s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-full-True] | |
| 1.19s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-local-False] | |
| 1.18s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-True-True-True] | |
| 1.17s call unit/runtime/zero/test_zero_leaf_module.py::TestSetZ3LeafModule::test_choose_module_by_rank | |
| 1.16s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-fp32-zero3] | |
| 1.16s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-False-True-resulting_optimizer12] | |
| 1.15s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-bf16-zero3] | |
| 1.11s call unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[True-True-False] | |
| 1.11s call unit/runtime/zero/test_zero_leaf_module.py::TestSetZ3LeafModule::test_no_grad_input_error | |
| 1.10s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-fp32-zero3] | |
| 1.10s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-fp32-zero3] | |
| 1.09s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-fp16-zero3] | |
| 1.09s call unit/ops/aio/test_aio.py::TestRead::test_async_read[True-False-False-True] | |
| 1.09s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-bf16-zero3] | |
| 1.09s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-fp16-zero3] | |
| 1.09s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp16-zero3] | |
| 1.08s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-bf16-zero3] | |
| 1.08s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-fp16-zero3] | |
| 1.08s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-bf16-zero3] | |
| 1.07s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-False-False-False] | |
| 1.06s call unit/ops/adam/test_hybrid_adam.py::TestHybridAdam::test_hybrid_adam_equal[16-fp16] | |
| 1.05s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp32-zero1] | |
| 1.03s call unit/ops/aio/test_aio.py::TestRead::test_async_read[True-True-False-False] | |
| (2579 durations < 1s hidden. Use -vv to show these durations.) | |
| [36m[1m====================================================================================================================================== short test summary info =======================================================================================================================================[0m | |
| [31mFAILED[0m tests/unit/inference/quantization/test_intX_quantization.py::[1mTestQuantizedInt::test_zero3_int4_post_init_quant_nvme_offload[0m - AssertionError | |
| [31mFAILED[0m tests/unit/inference/quantization/test_intX_quantization.py::[1mTestQuantizedInt::test_zero3_int4_quantized_initialization_nvme_offload[0m - AssertionError | |
| [31mFAILED[0m tests/unit/ops/accelerators/test_accelerator_backward.py::[1mTestCUDABackward::test_backward[64-160-128-2-24-False-True-0.2][0m - ImportError: dynamic module does not define module export function (PyInit_transformer_op) | |
| [31mFAILED[0m tests/unit/ops/accelerators/test_accelerator_backward.py::[1mTestCUDABackward::test_backward[64-1600-128-2-4-False-True-0.2][0m - ImportError: dynamic module does not define module export function (PyInit_transformer_op) | |
| [31mFAILED[0m tests/unit/ops/accelerators/test_accelerator_backward.py::[1mTestCUDABackward::test_backward[8-1600-128-25-3-True-True-0.05][0m - ImportError: dynamic module does not define module export function (PyInit_transformer_op) | |
| [31mFAILED[0m tests/unit/ops/accelerators/test_accelerator_backward.py::[1mTestCUDABackward::test_backward[8-160-128-2-3-True-True-0.1][0m - ImportError: dynamic module does not define module export function (PyInit_transformer_op) | |
| [31mFAILED[0m tests/unit/ops/accelerators/test_accelerator_backward.py::[1mTestCUDABackward::test_backward[8-1600-128-2-3-True-True-0.05][0m - ImportError: dynamic module does not define module export function (PyInit_transformer_op) | |
| [31mFAILED[0m tests/unit/ops/accelerators/test_accelerator_forward.py::[1mTestCUDAForwardSmallBatchSize::test_forward_with_small_bsz[8-3-1024-512-16-3-True-False][0m - ImportError: dynamic module does not define module export function (PyInit_transformer_op) | |
| [31mFAILED[0m tests/unit/ops/accelerators/test_accelerator_forward.py::[1mTestCUDAForwardSmallBatchSize::test_forward_with_small_bsz[8-7-1024-512-16-3-True-True][0m - ImportError: dynamic module does not define module export function (PyInit_transformer_op) | |
| [31mFAILED[0m tests/unit/ops/accelerators/test_accelerator_forward.py::[1mTestCUDAForwardSmallBatchSize::test_forward_with_small_bsz[8-3-1024-512-16-3-False-False][0m - ImportError: dynamic module does not define module export function (PyInit_transformer_op) | |
| [31mFAILED[0m tests/unit/ops/accelerators/test_accelerator_forward.py::[1mTestCUDAForwardSmallBatchSize::test_forward_with_small_bsz[8-7-1024-512-16-3-False-True][0m - ImportError: dynamic module does not define module export function (PyInit_transformer_op) | |
| [31m====================================================================================================== [31m[1m11 failed[0m, [32m1043 passed[0m, [33m129 skipped[0m, [33m4426 deselected[0m, [33m172 warnings[0m[31m in 5927.76s (1:38:47)[0m[31m ======================================================================================================[0m | |
| (at easybuild/tools/run.py:695 in parse_cmd_output) | |
| == 2024-10-29 16:34:53,874 build_log.py:267 INFO ... (took 1 hour 46 mins 28 secs) | |
| == 2024-10-29 16:34:53,874 build_log.py:267 INFO ... (took 1 hour 46 mins 46 secs) | |
| == 2024-10-29 16:34:53,874 filetools.py:2025 INFO Removing lock /apps/Test/software/.locks/_apps_Test_software_DeepSpeed_0.14.5-foss-2023a-CUDA-12.1.1.lock... | |
| == 2024-10-29 16:34:53,880 filetools.py:385 INFO Path /apps/Test/software/.locks/_apps_Test_software_DeepSpeed_0.14.5-foss-2023a-CUDA-12.1.1.lock successfully removed. | |
| == 2024-10-29 16:34:53,880 filetools.py:2029 INFO Lock removed: /apps/Test/software/.locks/_apps_Test_software_DeepSpeed_0.14.5-foss-2023a-CUDA-12.1.1.lock | |
| == 2024-10-29 16:34:53,880 easyblock.py:4297 WARNING build failed (first 300 chars): cmd "export PYTHONPATH=/cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-ji7_gp2a/tmpctesi1ko/lib/python3.11/site-packages:$PYTHONPATH && PATH="$PATH:$PWD/bin" pytest tests/unit/ -k "not TestTensorBoard and not TestWandb and not TestCometMonitor" " exited with exit code 1 and output: | |
| [1m=================== | |
| == 2024-10-29 16:34:53,880 easyblock.py:326 INFO Closing log for application name DeepSpeed version 0.14.5 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment