Created
November 8, 2024 16:38
-
-
Save VRehnberg/349c993a939ba87d69f1674fde393e8a to your computer and use it in GitHub Desktop.
(partial) EasyBuild log for failed build of /cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-gasnzvdk/files_pr21438/d/DeepSpeed/DeepSpeed-0.14.5-foss-2023a-CUDA-12.1.1.eb (PR(s) #21438) (easyblock PR(s) #3450)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 6.47s call unit/runtime/half_precision/test_bf16.py::TestZeroDtypeCocktail::test[default-bf16] | |
| 6.47s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[False-False] | |
| 6.47s call unit/runtime/zero/test_zero.py::TestZeroToFP32::test_1_param_group[False-2] | |
| 6.47s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpoint::test_hybrid_optimizer_state[2] | |
| 6.47s call unit/runtime/test_ds_initialize.py::TestNoOptim::test[0] | |
| 6.47s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[log-15] | |
| 6.47s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-3-dtype0] | |
| 6.46s call unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fp32_optimizer | |
| 6.46s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-3-dtype1] | |
| 6.46s call unit/runtime/half_precision/test_fp16.py::TestAdamFP16ZeroOneCycleCompatibility::test[False-3] | |
| 6.45s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[1e-05-0.1-0-10-0] | |
| 6.45s call unit/runtime/zero/test_zero.py::TestParamPartitioningSkipInit::test | |
| 6.45s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-False-False] | |
| 6.45s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-1-dtype2] | |
| 6.45s call unit/runtime/test_lr_schedulers.py::TestWarmupCosineLR::test_lr[200-20-0.1-0.2] | |
| 6.44s call unit/runtime/test_data_efficiency.py::TestLegacyCurriculumScheduler::test_fixed_discrete | |
| 6.44s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-3-dtype1] | |
| 6.44s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype2] | |
| 6.43s call unit/runtime/half_precision/test_fp16.py::TestFP16AdamTypes::test[False-Adam] | |
| 6.43s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_fixed_dp[False-True-True] | |
| 6.43s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-1-dtype1] | |
| 6.43s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-3-dtype2] | |
| 6.43s call unit/runtime/half_precision/test_bf16.py::TestZeroDtypeCocktail::test[fp16-fp16] | |
| 6.43s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype2] | |
| 6.43s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype2] | |
| 6.42s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-15] | |
| 6.42s call unit/runtime/half_precision/test_fp16.py::TestAdamFP32EmptyGrad::test | |
| 6.41s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_half_int4_quantization | |
| 6.41s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_quantized_linear[8bits-1] | |
| 6.41s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningManyParams::test[True] | |
| 6.41s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_zero_grad[False] | |
| 6.40s call unit/runtime/zero/test_zero.py::TestZeroFrozenWeights::test[2] | |
| 6.40s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-3-dtype1] | |
| 6.40s call unit/runtime/test_runtime_utils.py::TestCheckOverflow::test[False] | |
| 6.39s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_namedtuple] | |
| 6.39s call unit/runtime/test_ds_config_dict.py::TestDistInit::test | |
| 6.38s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[2] | |
| 6.38s call unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_existing_latest | |
| 6.37s call unit/runtime/zero/test_zero.py::TestZeroToFP32::test_2_param_groups[False-2] | |
| 6.37s call unit/runtime/zero/test_zero.py::TestZeroToFP32::test_1_param_group[True-2] | |
| 6.37s call unit/runtime/zero/test_zero_leaf_module.py::TestSetZ3LeafModule::test_choose_module_by_counter | |
| 6.37s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-3-dtype2] | |
| 6.36s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs3[mask1] | |
| 6.36s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-1-dtype0] | |
| 6.35s call unit/runtime/half_precision/test_bf16.py::TestZeroDtypeCocktail::test[fp16-bf16] | |
| 6.35s call unit/runtime/half_precision/test_bf16.py::TestZeroSupportedClientOptimizer::test[FusedAdam] | |
| 6.35s call unit/runtime/zero/test_zero.py::TestZero3DictFwd::test[list] | |
| 6.35s call unit/runtime/half_precision/test_fp16.py::TestLambFP16::test_empty_grad | |
| 6.35s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[False-2] | |
| 6.34s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-True-True] | |
| 6.34s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-1-dtype1] | |
| 6.34s call unit/runtime/zero/test_zero_context_return.py::TestReturnParam::test_stage_3_output_type[tensor] | |
| 6.34s call unit/runtime/zero/test_zero.py::TestZero3RepeatForwardLoop::test[False] | |
| 6.33s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-False-True] | |
| 6.33s call unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_unfused_optimizer | |
| 6.33s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-10] | |
| 6.33s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_save_exclude_frozen_weights[1] | |
| 6.32s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype1] | |
| 6.32s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_reduce_scatter[False] | |
| 6.32s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[1-True] | |
| 6.32s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[True] | |
| 6.30s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-False-False] | |
| 6.30s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-False] | |
| 6.30s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_disable_autocast_linear[True] | |
| 6.30s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[0.9] | |
| 6.30s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_fp16_enabled[False] | |
| 6.30s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs1_outputs1[mask1] | |
| 6.30s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_offload_optimizer[False] | |
| 6.30s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype0] | |
| 6.29s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_arg_none[mask0] | |
| 6.29s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[FusedAdam-1] | |
| 6.29s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.01-0.01-19-True] | |
| 6.29s call unit/monitor/test_monitor.py::TestCSVMonitor::test_empty_csv_monitor | |
| 6.29s call unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype1] | |
| 6.29s call unit/runtime/zero/test_zero.py::TestZeroAdamOptimizerStepCount::test[1] | |
| 6.29s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-True-False] | |
| 6.29s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_prefetching[False] | |
| 6.28s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyGrad::test[1] | |
| 6.28s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyGrad::test[3] | |
| 6.28s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype1] | |
| 6.27s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[log-10] | |
| 6.27s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[0] | |
| 6.27s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-1-dtype2] | |
| 6.27s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[0.1] | |
| 6.27s call unit/runtime/zero/test_zero.py::TestZeroUnbalancedGradients::test[2] | |
| 6.27s call unit/runtime/half_precision/test_fp16.py::TestAdamFP16ZeroOneCycleCompatibility::test[False-2] | |
| 6.27s call unit/runtime/half_precision/test_bf16.py::TestAdamBF16ZeroOneCycleCompatibility::test | |
| 6.27s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype0] | |
| 6.27s call unit/runtime/half_precision/test_fp16.py::TestZeroEmptyGrad::test[2] | |
| 6.26s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_fp16_enabled[True] | |
| 6.25s call unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fused_optimizer | |
| 6.25s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_reduce_scatter[True] | |
| 6.24s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0-210] | |
| 6.24s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-1-dtype1] | |
| 6.24s call unit/elasticity/test_elastic.py::TestNonElasticBatchParamsWithOverride::test | |
| 6.24s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-True-False] | |
| 6.23s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[0] | |
| 6.23s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-3-dtype0] | |
| 6.23s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_save_exclude_custom_frozen_weights[2] | |
| 6.23s call unit/runtime/test_pld.py::TestPLDModel::test_pld_model[1.0] | |
| 6.23s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-3-dtype1] | |
| 6.22s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[FusedAdam-2] | |
| 6.22s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[True-True] | |
| 6.22s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_param_persistence_threshold[0] | |
| 6.22s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-3-dtype0] | |
| 6.21s call unit/runtime/zero/test_zero.py::TestZero3DictFwd::test[dict] | |
| 6.21s call unit/runtime/test_lr_schedulers.py::TestWarmupCosineLR::test_lr[100-10-0.1-0.2] | |
| 6.20s call unit/comm/test_dist.py::TestDistInit::test_no_init[None] | |
| 6.20s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-True] | |
| 6.19s call unit/linear/test_ctx.py::TestInitTransformers::test_config_init | |
| 6.19s call unit/runtime/zero/test_zero_context_return.py::TestReturnParam::test_stage_3_output_type[dict] | |
| 6.19s call unit/runtime/test_runtime_utils.py::TestCheckOverflow::test[True] | |
| 6.19s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-1-dtype0] | |
| 6.19s call unit/runtime/test_runtime_utils.py::TestClipGradNorm::test_clipped_val | |
| 6.18s call unit/runtime/zero/test_zero.py::TestZeroOffloadOptim::test[False] | |
| 6.18s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_prefetching[True] | |
| 6.18s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[1] | |
| 6.18s call unit/elasticity/test_elastic.py::TestElasticConfigChanged::test | |
| 6.17s setup unit/comm/test_dist.py::TestDistributedFixture::test[2-32] | |
| 6.17s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[None-None] | |
| 6.17s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_Tuple] | |
| 6.16s call unit/runtime/comm/test_coalesced_collectives.py::TestReduceScatterCoalesced::test_single_input | |
| 6.16s call unit/runtime/test_ds_config_dict.py::TestBasicConfig::test_check_version | |
| 6.16s call unit/runtime/half_precision/test_fp16.py::TestZero3LazyScatter::test | |
| 6.16s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_zero_grad[True] | |
| 6.15s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-1-dtype1] | |
| 6.15s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0.001-100] | |
| 6.14s call unit/runtime/test_ds_config_dict.py::TestNoModel::test | |
| 6.14s call unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[True] | |
| 6.13s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-33] | |
| 6.13s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-3-dtype0] | |
| 6.13s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-1-dtype2] | |
| 6.13s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpoint::test_load_module_only[0] | |
| 6.12s call unit/runtime/comm/test_coalesced_collectives.py::TestReduceScatterCoalesced::test_two_inputs | |
| 6.12s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-False-3-dtype2] | |
| 6.11s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[None] | |
| 6.11s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[2] | |
| 6.11s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_model_class[EltwiseMultiplicationTestNetwork_List] | |
| 6.10s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[0] | |
| 6.10s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_param_persistence_threshold[10] | |
| 6.09s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestFused::test_all_overflow | |
| 6.09s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[3] | |
| 6.09s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_arg_none[mask1] | |
| 6.07s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[False-True] | |
| 6.07s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-3-dtype1] | |
| 6.07s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-False-False] | |
| 6.07s call unit/runtime/zero/test_zero.py::TestZeroPartitionCache::test_training_partition_cache[True] | |
| 6.07s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-19] | |
| 6.07s call unit/checkpoint/test_zero_optimizer.py::TestZeRONonDistributed::test_chmod_exception_handling[2] | |
| 6.07s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.0001-0.001-10-True] | |
| 6.06s call unit/runtime/zero/test_zero_context_return.py::TestReturnParam::test_ext_param_return | |
| 6.06s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[non_tensor_output3] | |
| 6.06s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-1-dtype1] | |
| 6.06s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[IGNORE] | |
| 6.05s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[WarmupDecayLR-params1] | |
| 6.05s call unit/runtime/half_precision/test_fp16.py::TestFP16AdamTypes::test[False-AdamW] | |
| 6.05s call unit/utils/test_init_on_device.py::TestOnDevice::test_on_device[cuda:0] | |
| 6.05s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[non_tensor3] | |
| 6.04s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-1-dtype2] | |
| 6.04s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-32-8-2-True] | |
| 6.03s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output2] | |
| 6.03s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[2] | |
| 6.02s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_ext_param_getattr | |
| 6.02s call unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[False] | |
| 6.01s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[True] | |
| 6.00s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs1_outputs1[mask0] | |
| 6.00s call unit/runtime/comm/test_coalesced_collectives.py::TestReduceScatterCoalescedTensorSmallerThanWorldSize::test | |
| 6.00s call unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_2 | |
| 6.00s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_disable_autocast_linear[False] | |
| 6.00s call unit/ops/adagrad/test_cpu_adagrad.py::TestCPUAdagradGPUError::test_cpu_adagrad_gpu_error | |
| 6.00s call unit/runtime/zero/test_zero.py::TestIncorectAllgatherBucketSize::test[1000] | |
| 6.00s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_float_int4_quantization | |
| 6.00s call unit/moe/test_moe.py::TestTopk::test | |
| 6.00s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.0001-1e-05-1-True] | |
| 6.00s call unit/runtime/test_ds_config_dict.py::TestInitNoOptimizer::test | |
| 6.00s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype2] | |
| 6.00s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[2] | |
| 6.00s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs3[mask1] | |
| 6.00s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-True-False] | |
| 6.00s call unit/runtime/test_multi_output_model.py::TestTwoOutputModel::test | |
| 5.99s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[log-19] | |
| 5.99s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestUnfused::test_some_overflow | |
| 5.99s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_arg_none[mask1] | |
| 5.98s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[None] | |
| 5.98s call unit/runtime/test_ds_initialize.py::TestConfigOptimizer::test[False] | |
| 5.98s call unit/runtime/test_lr_schedulers.py::TestSchedulerOptimizerParity::test[WarmupLR-params0] | |
| 5.97s call unit/runtime/test_ds_config_dict.py::TestArgs::test_no_args | |
| 5.97s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[0.001-0.1-0.1-21-21] | |
| 5.97s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-3-dtype0] | |
| 5.97s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_arg_none[mask1] | |
| 5.96s call unit/comm/test_dist.py::TestGroupedDistTest::test_two[1138] | |
| 5.96s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[WARN] | |
| 5.95s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-1-dtype0] | |
| 5.95s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_float_int8_quantization | |
| 5.95s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-3-dtype1] | |
| 5.95s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-3-dtype1] | |
| 5.95s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.01-0.01-19-False] | |
| 5.94s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[non_tensor3] | |
| 5.94s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output1] | |
| 5.94s call unit/comm/test_dist.py::TestDistInitNoEnv::test | |
| 5.94s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-False-False] | |
| 5.94s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs1[mask1] | |
| 5.94s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs1_outputs1[mask1] | |
| 5.94s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-True-1-dtype0] | |
| 5.93s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-33] | |
| 5.93s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_arg_none[mask0] | |
| 5.93s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Callable-Callable] | |
| 5.93s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-19] | |
| 5.93s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output3] | |
| 5.93s call unit/runtime/zero/test_zero_nesting_init.py::TestNestingInit::test_nesting_init | |
| 5.92s call unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[True] | |
| 5.92s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs1[mask1] | |
| 5.92s call unit/runtime/comm/test_coalesced_collectives.py::TestAllToAllQuantReduceFallback::test_non_divisible | |
| 5.92s call unit/runtime/test_autocast.py::TestAutoCastDisable::test_missing_amp_autocast[True] | |
| 5.92s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0-211] | |
| 5.92s call unit/runtime/test_pld.py::TestNonPLDModel::test_non_pld_model | |
| 5.92s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[log-19] | |
| 5.92s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[non_tensor3] | |
| 5.92s call unit/runtime/test_autocast.py::TestAutoCastEnable::test_autocast_linear[True-False] | |
| 5.92s call unit/runtime/test_ds_config_dict.py::TestConfigLoad::test_hjson | |
| 5.91s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-1-dtype0] | |
| 5.91s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-15] | |
| 5.91s call unit/runtime/zero/test_zero_context_return.py::TestReturnParam::test_stage_3_output_type[None] | |
| 5.91s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[WarmupLR-params0] | |
| 5.91s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[True-3] | |
| 5.91s call unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_1 | |
| 5.91s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[False-1] | |
| 5.91s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[FAIL] | |
| 5.91s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[OneCycle-params2] | |
| 5.90s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[1-False] | |
| 5.90s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs2[mask1] | |
| 5.90s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs1[mask0] | |
| 5.90s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[non_tensor4] | |
| 5.90s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-True-True] | |
| 5.90s call unit/runtime/test_lr_schedulers.py::TestGetLrBeforeTrain::test[LRRangeTest-params3] | |
| 5.90s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-False-3-dtype1] | |
| 5.89s call unit/checkpoint/test_zero_optimizer.py::TestZeRONonDistributed::test_chmod_exception_handling[3] | |
| 5.89s call unit/runtime/test_lr_schedulers.py::TestSchedulerOptimizerParity::test[WarmupDecayLR-params1] | |
| 5.89s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype2] | |
| 5.89s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[non_tensor3] | |
| 5.88s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[Adam-1] | |
| 5.88s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[0.001-0.001-10-False] | |
| 5.88s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[4-True] | |
| 5.88s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_mom[0.08-0.09-0.001-101] | |
| 5.88s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-True-True] | |
| 5.87s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-False-3-dtype0] | |
| 5.87s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[True] | |
| 5.86s call unit/runtime/test_ds_config_dict.py::TestConfigLoad::test_dict | |
| 5.86s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestUnfused::test_all_overflow | |
| 5.86s call unit/comm/test_dist.py::TestDistInit::test_already_init[None] | |
| 5.86s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-1-dtype1] | |
| 5.86s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs2[mask1] | |
| 5.86s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[None] | |
| 5.86s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_same_lrscheler_and_callable[Callable] | |
| 5.86s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[True-2] | |
| 5.86s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-32-16-1-True] | |
| 5.85s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[1e-05-0.01-0.001-10-101] | |
| 5.85s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[True-2] | |
| 5.85s call unit/runtime/comm/test_coalesced_collectives.py::TestAllToAllQuantReduceFallback::test_1d_tensor | |
| 5.85s call unit/monitor/test_monitor.py::TestCSVMonitor::test_csv_monitor | |
| 5.85s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestFused::test_no_overflow | |
| 5.85s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-33-17-2-False] | |
| 5.84s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-False-False] | |
| 5.84s call unit/compression/test_compression.py::TestCompression::test_conv1d_convertion | |
| 5.84s call unit/inference/quantization/test_intX_quantization.py::TestQuantizedInt::test_half_int8_quantization | |
| 5.84s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-1-dtype0] | |
| 5.84s call unit/runtime/test_lr_schedulers.py::TestOneCycle::test_lr[0.001-0.1-0-21-21] | |
| 5.84s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable_onecyclelr_steplr[Callable] | |
| 5.84s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[True-1] | |
| 5.84s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_scattered_init_dist | |
| 5.83s call unit/runtime/test_lr_schedulers.py::TestSchedulerOptimizerParity::test[LRRangeTest-params3] | |
| 5.83s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[None] | |
| 5.83s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs3[mask0] | |
| 5.83s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[Adam-2] | |
| 5.83s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-33] | |
| 5.83s call unit/runtime/half_precision/test_fp16.py::TestAdamFP16ZeroOneCycleCompatibility::test[False-1] | |
| 5.83s call unit/runtime/zero/test_zero_nesting_init.py::TestShutdownInNestingInit::test_shutdown_in_nesting_init | |
| 5.83s call unit/runtime/test_lr_schedulers.py::TestLrRange::test[1e-05-1e-05-1-False] | |
| 5.83s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestUnfused::test_no_overflow | |
| 5.83s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_schedule[linear-10] | |
| 5.83s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-1-dtype0] | |
| 5.83s call unit/comm/test_dist.py::TestGroupedDistTest::test_one[1138] | |
| 5.82s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable_onecyclelr_steplr[_LRScheduler] | |
| 5.82s call unit/runtime/half_precision/test_fp16.py::TestAdamwFP16Basic::test | |
| 5.82s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[True] | |
| 5.81s call unit/checkpoint/test_zero_optimizer.py::TestSaveTensorClone::test_save_tensor_clone[False-2] | |
| 5.81s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unknown_tag_validation | |
| 5.81s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-15] | |
| 5.81s call unit/runtime/test_multi_output_model.py::TestThreeOutputModel::test | |
| 5.80s call unit/runtime/test_ds_config_dict.py::TestArgs::test_none_args | |
| 5.80s call unit/runtime/half_precision/test_fp16.py::TestFP16AdamTypes::test[True-Adam] | |
| 5.79s call unit/runtime/test_ds_initialize.py::TestClientOptimizer::test[Callable] | |
| 5.79s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[non_tensor4] | |
| 5.79s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs1[mask1] | |
| 5.79s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs2[mask0] | |
| 5.79s call unit/runtime/zero/test_zero_context.py::TestScatterGather::test | |
| 5.79s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_inputs2_outputs1[mask0] | |
| 5.78s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[True-True-3-dtype1] | |
| 5.78s call unit/runtime/test_ds_config_dict.py::TestConfigLoad::test_json | |
| 5.78s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[non_tensor_output2] | |
| 5.78s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[None] | |
| 5.77s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_same_lrscheler_and_callable[None] | |
| 5.77s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output3] | |
| 5.77s call unit/comm/test_dist.py::TestDistInit::test_already_init[False] | |
| 5.77s call unit/runtime/test_lr_schedulers.py::TestLrSchedule::test_lr_warmup_decay_schedule[linear-10] | |
| 5.77s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_output[non_tensor4] | |
| 5.77s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[True] | |
| 5.77s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to2[False-True-3-dtype2] | |
| 5.77s call unit/runtime/test_ds_config_dict.py::TestBatchConfig::test[2-32-18-1-False] | |
| 5.76s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable[_LRScheduler] | |
| 5.76s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[True-1] | |
| 5.76s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable[None] | |
| 5.76s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-3-dtype0] | |
| 5.76s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-True-False] | |
| 5.76s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs1[mask0] | |
| 5.75s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs2[mask0] | |
| 5.75s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_throughput_calculation | |
| 5.74s call unit/runtime/zero/test_zero.py::TestIncorectAllgatherBucketSize::test[1001] | |
| 5.74s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[False-True-3-dtype2] | |
| 5.74s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[non_tensor4] | |
| 5.74s call unit/runtime/test_lr_schedulers.py::TestSchedulerOptimizerParity::test[OneCycle-params2] | |
| 5.74s call unit/comm/test_dist.py::TestDistInit::test_no_init[True] | |
| 5.73s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensorOutputOrdering::test_ckpt_non_tensor_output_ordering[non_tensor_output1] | |
| 5.73s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs1_outputs1[mask0] | |
| 5.73s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[2] | |
| 5.73s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Optimizer-Callable] | |
| 5.73s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[None] | |
| 5.73s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output1] | |
| 5.73s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[None] | |
| 5.72s call unit/runtime/zero/test_zero.py::TestZero3ParamPartitioningBase::test_offload_optimizer[True] | |
| 5.72s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs2[mask0] | |
| 5.72s call unit/comm/test_dist.py::TestDistArgs::test[hello-icosahedron-1138-purple] | |
| 5.71s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithGrad::test_ckpt_arg_none[mask0] | |
| 5.71s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Callable-_LRScheduler] | |
| 5.71s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[non_tensor4] | |
| 5.71s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs1_outputs1[mask1] | |
| 5.71s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_output[2] | |
| 5.70s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestCheckpointNonTensor::test_ckpt_non_tensor_input[non_tensor4] | |
| 5.70s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-False-False] | |
| 5.69s call unit/runtime/zero/test_zero_context.py::TestGatherUpdate::test | |
| 5.68s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-1-dtype2] | |
| 5.68s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Callable-None] | |
| 5.68s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype1-True-True] | |
| 5.67s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[None-Callable] | |
| 5.66s setup unit/checkpoint/test_universal_checkpoint.py::TestZeROUniversalCheckpointDP::test_dp_world_size_2to4[True-False-3-dtype2] | |
| 5.66s call unit/runtime/activation_checkpointing/test_activation_checkpointing.py::TestActivationCheckpoint::test_ckpt_inputs2_outputs3[mask1] | |
| 5.66s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_input[2] | |
| 5.66s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype0-False-True] | |
| 5.66s call unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_missing_latest | |
| 5.66s call unit/checkpoint/test_zero_optimizer.py::TestZeRONonDistributed::test_chmod_exception_handling[1] | |
| 5.64s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Optimizer-None] | |
| 5.64s call unit/runtime/half_precision/test_fp16.py::TestZeroAllowUntestedOptimizer::test[False-1] | |
| 5.64s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[2] | |
| 5.64s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithGrad::test_ckpt_non_tensor_input[non_tensor3] | |
| 5.64s call unit/runtime/zero/test_zero.py::TestZeroOffloadOptim::test[True] | |
| 5.63s call unit/runtime/half_precision/test_fp16.py::TestZeroSupportedClientOptimizer::test[Adam-3] | |
| 5.63s call unit/runtime/test_data.py::TestDataLoaderDropLast::test[4-False] | |
| 5.63s call unit/comm/test_dist.py::TestDistributedFixture::test[4-32] | |
| 5.62s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable_onecyclelr_steplr[None] | |
| 5.62s call unit/runtime/test_ds_initialize.py::TestClientOptimizer::test[Optimizer] | |
| 5.61s call unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test | |
| 5.61s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-2-dtype0] | |
| 5.61s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-2-dtype1] | |
| 5.61s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_same_lrscheler_and_callable[_LRScheduler] | |
| 5.61s call unit/runtime/test_ds_initialize.py::TestClientOptimizer::test[None] | |
| 5.61s call unit/runtime/test_ds_config_dict.py::TestBasicConfig::test_accelerator | |
| 5.60s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs2[mask1] | |
| 5.60s call unit/runtime/half_precision/test_dynamic_loss_scale.py::TestFused::test_some_overflow | |
| 5.60s call unit/runtime/test_ds_initialize.py::TestClientLrScheduler::test[Optimizer-_LRScheduler] | |
| 5.60s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestActivationCheckpointWithoutGrad::test_ckpt_inputs2_outputs3[mask0] | |
| 5.59s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[None] | |
| 5.59s call unit/comm/test_dist.py::TestDistributedFixture::test[2-16] | |
| 5.59s call unit/runtime/test_ds_initialize.py::TestClientLrSchedulerInit::test_diff_lrscheler_and_callable[Callable] | |
| 5.58s call unit/runtime/half_precision/test_bf16.py::TestZeroAllowUntestedOptimizer::test | |
| 5.57s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorWithoutGrad::test_ckpt_non_tensor_output[True] | |
| 5.56s call unit/comm/test_dist.py::TestDistributedFixture::test[2-32] | |
| 5.56s call unit/runtime/activation_checkpointing/test_activation_checkpointing_non_reentrant.py::TestCheckpointNonTensorOutputOrderingWithoutGrad::test_ckpt_non_tensor_output_ordering[non_tensor_output2] | |
| 5.54s call unit/runtime/zero/test_zero.py::TestEmptyParameterGroup::test_empty_param_groups[dtype2-False-True] | |
| 5.51s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-2-dtype2] | |
| 5.51s call unit/runtime/compile/test_compile_zero.py::TestZeRO::test_compile_zero[nvme-1-dtype2] | |
| 5.48s call unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test_flops_profiler_in_inference | |
| 5.47s call unit/comm/test_dist.py::TestDistributedFixture::test[4-16] | |
| 5.46s setup unit/comm/test_dist.py::TestDistributedFixture::test[2-16] | |
| 5.44s call unit/runtime/zero/test_zero_context_ancestry.py::TestDSInitWZinit::test | |
| 5.43s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_scatter_halftype | |
| 5.37s call unit/runtime/zero/test_zero.py::TestZeroPartitionCache::test_training_partition_cache[False] | |
| 5.33s setup unit/comm/test_dist.py::TestDistributedFixture::test[4-32] | |
| 5.31s call unit/runtime/zero/test_zero_context.py::TestZeroGatheredParametersFree::test | |
| 5.29s call unit/runtime/zero/test_zero_context_ancestry.py::TestSerialParamInit::test_subclass_param_init | |
| 5.29s call unit/runtime/zero/test_zero_dynamic_class.py::TestNewClassDeclaredNestingInit::test_new_class_declared_nesting_init | |
| 5.24s setup unit/comm/test_dist.py::TestDistributedFixture::test[4-16] | |
| 5.23s call unit/runtime/zero/test_zero_context.py::TestSerialContext::test_subclass_param | |
| 5.22s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-False-resulting_optimizer0] | |
| 5.22s call unit/runtime/zero/test_zero_dynamic_class.py::TestNewClassDeclaredInsideNestingInit::test_new_class_declared_inside_nesting_init | |
| 5.13s call unit/runtime/zero/test_zero_context.py::TestMiCSGatheredParametersFree::test | |
| 4.64s call unit/comm/test_dist.py::TestDistInit::test_no_init[False] | |
| 4.55s call unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[False] | |
| 2.88s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-full-False] | |
| 2.71s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-full-dtype2] | |
| 2.68s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-full-dtype0] | |
| 2.63s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-local-False] | |
| 2.43s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-2-full-dtype2] | |
| 2.41s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-full-dtype1] | |
| 2.40s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-local-False] | |
| 2.32s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-full-dtype2] | |
| 2.29s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-full-False] | |
| 2.24s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-1-full-dtype0] | |
| 2.23s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-full-dtype1] | |
| 2.20s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-local-dtype2] | |
| 2.20s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-full-dtype0] | |
| 2.19s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-local-dtype1] | |
| 2.19s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-full-True] | |
| 2.14s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-1-full-True] | |
| 2.13s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-2-full-False] | |
| 2.13s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-local-dtype1] | |
| 2.13s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-local-dtype2] | |
| 2.12s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[nvme-3-local-dtype0] | |
| 2.11s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-full-True] | |
| 2.11s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-3-local-dtype0] | |
| 2.10s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-3-local-True] | |
| 2.08s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-True-resulting_optimizer14] | |
| 2.08s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[nvme-3-local-True] | |
| 1.96s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-1-full-dtype2] | |
| 1.86s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-local-False] | |
| 1.83s call unit/ops/adam/test_hybrid_adam.py::TestHybridAdam::test_hybrid_adam_equal[8-fp16] | |
| 1.82s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-True-resulting_optimizer6] | |
| 1.81s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-False-resulting_optimizer10] | |
| 1.80s call unit/ops/lion/test_cpu_lion.py::TestCPULion::test_fused_lion_equal[64-fp16] | |
| 1.77s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp32-zero3] | |
| 1.76s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-1-full-False] | |
| 1.76s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-2-full-dtype1] | |
| 1.74s call unit/runtime/zero/test_zero_leaf_module.py::TestSetZ3LeafModule::test_no_grad_input_error | |
| 1.74s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-False-resulting_optimizer2] | |
| 1.71s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-1-full-dtype1] | |
| 1.69s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[cpu-2-full-dtype0] | |
| 1.67s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[cpu-2-full-True] | |
| 1.67s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[64-fp16] | |
| 1.66s call unit/ops/lion/test_lion.py::TestLionConfigs::test[Lion-True-DeepSpeedCPULion] | |
| 1.64s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-True-True-False] | |
| 1.63s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-local-dtype0] | |
| 1.57s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-local-dtype2] | |
| 1.56s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-full-dtype1] | |
| 1.50s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-2-full-dtype2] | |
| 1.46s call unit/ops/aio/test_aio.py::TestRead::test_async_read[True-False-True-False] | |
| 1.43s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp16-zero2] | |
| 1.42s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-fp16-zero3] | |
| 1.30s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-full-dtype0] | |
| 1.30s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-True-False-False] | |
| 1.29s call unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[True-True-False] | |
| 1.29s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-full-dtype2] | |
| 1.26s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-False-True-True] | |
| 1.25s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-bf16-zero3] | |
| 1.25s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-False-True-False] | |
| 1.24s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-3-local-dtype1] | |
| 1.24s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_bf16_fragments[False] | |
| 1.23s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-True-False-resulting_optimizer11] | |
| 1.23s call unit/runtime/zero/test_zero_leaf_module.py::TestSetZ3LeafModule::test_choose_module_by_rank | |
| 1.22s call unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[False-True-False] | |
| 1.21s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-local-True] | |
| 1.21s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-full-False] | |
| 1.20s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-3-full-True] | |
| 1.19s call unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[True-True-True] | |
| 1.19s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-False-True-True] | |
| 1.17s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp32-zero1] | |
| 1.16s call unit/ops/aio/test_aio.py::TestRead::test_async_read[True-True-True-True] | |
| 1.15s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp16-zero1] | |
| 1.15s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-fp16-zero3] | |
| 1.14s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentUpdate::test_zero_fragments[none-1-full-dtype0] | |
| 1.12s call unit/ops/aio/test_aio.py::TestRead::test_async_read[False-True-True-True] | |
| 1.11s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-fp32-zero3] | |
| 1.10s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-fp32-zero3] | |
| 1.10s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-fp16-zero3] | |
| 1.10s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-fp32-zero3] | |
| 1.10s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[bf16-bf16-zero3] | |
| 1.10s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-bf16-zero3] | |
| 1.09s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-bf16-zero3] | |
| 1.09s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp16-fp16-zero3] | |
| 1.08s call unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentGet::test_zero_fragments[none-1-full-True] | |
| 1.06s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-True-resulting_optimizer4] | |
| 1.05s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-True-False-resulting_optimizer9] | |
| 1.05s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[fp32-fp16-zero2] | |
| 1.03s call unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[True-False-True] | |
| 1.02s call unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-True-True-False] | |
| 1.01s call unit/runtime/test_ds_initialize.py::TestOptimizerImplementation::test[None-bf16-zero2] | |
| (2716 durations < 1s hidden. Use -vv to show these durations.) | |
| [36m[1m===================================================================================== short test summary info =====================================================================================[0m | |
| [31mFAILED[0m tests/unit/runtime/compile/test_compile_zero.py::[1mTestZeRO::test_compile_zero[none-2-dtype0][0m - Failed: Worker 0 killed by signal 11 | |
| [31mFAILED[0m tests/unit/runtime/compile/test_compile_zero.py::[1mTestZeRO::test_compile_zero[cpu-1-dtype1][0m - Failed: Worker 0 killed by signal 11 | |
| [31mFAILED[0m tests/unit/runtime/compile/test_compile_zero.py::[1mTestZeRO::test_compile_zero[nvme-3-dtype0][0m - Failed: Worker 1 killed by signal 11 | |
| [31m===================================================== [31m[1m3 failed[0m, [32m934 passed[0m, [33m246 skipped[0m, [33m4426 deselected[0m, [33m140 warnings[0m[31m in 4970.16s (1:22:50)[0m[31m ======================================================[0m | |
| (at easybuild/tools/run.py:695 in parse_cmd_output) | |
| == 2024-11-08 17:38:32,889 build_log.py:267 INFO ... (took 1 hour 30 mins 32 secs) | |
| == 2024-11-08 17:38:32,890 build_log.py:267 INFO ... (took 1 hour 30 mins 52 secs) | |
| == 2024-11-08 17:38:32,890 filetools.py:2025 INFO Removing lock /apps/Test/software/.locks/_apps_Test_software_DeepSpeed_0.14.5-foss-2023a-CUDA-12.1.1.lock... | |
| == 2024-11-08 17:38:32,894 filetools.py:385 INFO Path /apps/Test/software/.locks/_apps_Test_software_DeepSpeed_0.14.5-foss-2023a-CUDA-12.1.1.lock successfully removed. | |
| == 2024-11-08 17:38:32,894 filetools.py:2029 INFO Lock removed: /apps/Test/software/.locks/_apps_Test_software_DeepSpeed_0.14.5-foss-2023a-CUDA-12.1.1.lock | |
| == 2024-11-08 17:38:32,894 easyblock.py:4297 WARNING build failed (first 300 chars): cmd "export PATH=/cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-8uwzpoz6/tmp95rtur4l/bin:$PATH PYTHONPATH=/cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-8uwzpoz6/tmp95rtur4l/lib/python3.11/site-packages:$PYTHONPATH LD_LIBRARY_PATH=/cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-8uwzpoz6/tmp95rtur4l/lib/python3.11/site | |
| == 2024-11-08 17:38:32,894 easyblock.py:326 INFO Closing log for application name DeepSpeed version 0.14.5 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment