Skip to content

Instantly share code, notes, and snippets.

View jrzaurin's full-sized avatar

Javier jrzaurin

169 followers · 2 following
View GitHub Profile
@jrzaurin
jrzaurin / lightgbm_vs_dl_bank_marketing.csv
Last active June 13, 2021 09:17
LightGBM vs DL for the Bank Marketing dataset
model f1 auc runtime best_epoch_or_ntrees
tabresnet 0.4300 0.6501 92.52 11.0
tabtransformer 0.4200 0.6440 31.69 4.0
tabmlp 0.3856 0.6281 9.57 7.0
lightgbm 0.3852 0.6265 0.46 57.0
tabnet 0.3087 0.5943 77.88 13.0
@jrzaurin
jrzaurin / bank_marketing_tabtransformer.csv
Last active June 13, 2021 09:15
Results for the Bank Marketing dataset with TabTransformer
We can make this file beautiful and searchable if this error is corrected: It looks like row 6 should actually have 28 columns, instead of 18 in line 5.
embed_dropout,full_embed_dropout,shared_embed,add_shared_embed,frac_shared_embed,input_dim,n_heads,n_blocks,dropout,ff_hidden_dim,transformer_activation,mlp_hidden_dims,mlp_activation,mlp_batchnorm,mlp_batchnorm_last,mlp_linear_first,with_wide,lr,batch_size,weight_decay,optimizer,lr_scheduler,base_lr,max_lr,div_factor,final_div_factor,n_cycles,val_loss_or_metric
0.0,False,False,False,8,32,8,6,0.1,,relu,None,relu,False,False,False,False,0.001,512,0.0,Adam,ReduceLROnPlateau,0.001,0.01,25,10000.0,5.0,0.2646
0.0,False,False,False,8,32,8,6,0.1,,relu,None,relu,False,False,False,False,0.001,512,0.0,AdamW,ReduceLROnPlateau,0.001,0.01,25,10000.0,5.0,0.2647
0.0,False,True,False,4,16,4,6,0.1,,relu,None,relu,False,False,False,False,0.01,128,0.0,RAdam,ReduceLROnPlateau,0.001,0.01,25,10000.0,5.0,0.2668
0.0,False,False,False,8,32,8,6,0.1,,relu,None,relu,False,False,False,False,0.01,1024,0.0,RAdam,ReduceLROnPlateau,0.001,0.01,25,10000.0,5.0,0.2672
0.0,False,False,False,8,32,8,6,0.1,,relu,None,relu,False,False,False,False,0.0
@jrzaurin
jrzaurin / bank_marketing_tabnet.csv
Last active June 13, 2021 09:15
Results for the Bank Marketing dataset with Tabnet
n_steps step_dim attn_dim ghost_bn virtual_batch_size momentum gamma dropout embed_dropout lr batch_size weight_decay lambda_sparse optimizer lr_scheduler base_lr max_lr div_factor final_div_factor n_cycles val_loss_or_metric
5 16 16 True 128 0.75 1.5 0.0 0.0 0.03 512 0.0 0.0001 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2714
5 16 16 True 64 0.25 1.5 0.0 0.0 0.03 512 0.0 0.0001 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2722
5 64 64 False 128 0.98 1.5 0.2 0.0 0.03 128 0.0 0.0001 Adam ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2727
5 64 64 False 128 0.98 1.5 0.2 0.0 0.03 128 0.0 0.0001 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2738
5 16 16 True 128 0.98 2.0 0.0 0.0 0.03 512 0.0 0.0001 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2738
@jrzaurin
jrzaurin / bank_marketing_tabresnet.csv
Last active June 13, 2021 09:15
Results for the Bank Marketing dataset with TabResnet
blocks_dims blocks_dropout mlp_hidden_dims mlp_activation mlp_dropout mlp_batchnorm mlp_batchnorm_last mlp_linear_first embed_dropout lr batch_size weight_decay optimizer lr_scheduler base_lr max_lr div_factor final_div_factor n_cycles val_loss_or_metric
same 0.5 None relu 0.1 False False False 0.0 0.0004 64 0.0 Adam OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2660
[50,50,50,50] 0.2 None relu 0.1 False False False 0.0 0.001 512 0.0 Adam ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2660
same 0.5 None relu 0.1 False False False 0.0 0.0004 64 0.0 RAdam OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2663
same 0.5 None relu 0.1 False False False 0.0 0.0004 128 0.0 RAdam OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2664
same 0.5 None relu 0.1 False False False 0.0 0.0004 128 0.0 Adam OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2667
@jrzaurin
jrzaurin / bank_marketing_tabmlp.csv
Last active June 13, 2021 09:14
Results for the Bank Marketing dataset with TabMlp
mlp_hidden_dims mlp_activation mlp_dropout mlp_batchnorm mlp_batchnorm_last mlp_linear_first embed_dropout lr batch_size weight_decay optimizer lr_scheduler base_lr max_lr div_factor final_div_factor n_cycles val_loss_or_metric
[100,50] relu 0.1 True True False 0.1 0.001 512 0.0 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2638
[100,50] relu 0.1 True False True 0.1 0.001 512 0.0 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2639
[100,50] relu 0.1 True True False 0.1 0.001 512 0.0 Adam ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2643
[100,50] relu 0.1 False False False 0.1 0.001 512 0.0 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2643
[100,50] relu 0.1 True False False 0.1 0.001 512 0.0 Adam ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2646
@jrzaurin
jrzaurin / lightgbm_vs_dl_adult.csv
Last active June 13, 2021 09:14
LightGBM vs DL for the Adult Census dataset
model acc runtime best_epoch_or_ntrees
lightgbm 0.8782 0.92 408.0
tabmlp 0.8722 205.36 62.0
tabtransformer 0.8718 288.64 32.0
tabnet 0.8704 422.30 26.0
tabresnet 0.8698 388.93 25.0
@jrzaurin
jrzaurin / adult_tabtransformer.csv
Last active June 13, 2021 09:13
Results for the Adult Census dataset with TabTransformer
We can make this file beautiful and searchable if this error is corrected: It looks like row 6 should actually have 28 columns, instead of 19 in line 5.
embed_dropout,full_embed_dropout,shared_embed,add_shared_embed,frac_shared_embed,input_dim,n_heads,n_blocks,dropout,ff_hidden_dim,transformer_activation,mlp_hidden_dims,mlp_activation,mlp_batchnorm,mlp_batchnorm_last,mlp_linear_first,with_wide,lr,batch_size,weight_decay,optimizer,lr_scheduler,base_lr,max_lr,div_factor,final_div_factor,n_cycles,val_loss_or_metric
0.0,False,False,False,8,16,4,4,0.1,,relu,None,relu,False,False,False,False,0.01,128,0.0,RAdam,ReduceLROnPlateau,0.001,0.01,25,10000.0,5.0,0.2879
0.0,False,False,False,8,16,4,4,0.1,,relu,same,relu,False,False,False,False,0.01,128,0.0,RAdam,ReduceLROnPlateau,0.001,0.01,25,10000.0,5.0,0.2884
0.0,False,False,False,8,16,4,4,0.1,,relu,None,relu,False,False,False,True,0.01,128,0.0,RAdam,ReduceLROnPlateau,0.001,0.01,25,10000.0,5.0,0.2888
0.0,False,False,False,8,16,4,8,0.2,,relu,None,relu,False,False,False,True,0.001,128,0.0,AdamW,ReduceLROnPlateau,0.001,0.01,25,10000.0,5.0,0.2892
0.0,False,False,False,8,16,2,4,0.1,,relu,None,relu,False,False,False,False,0.01,
@jrzaurin
jrzaurin / adult_tabnet.csv
Last active June 13, 2021 09:12
Results for the Adult Census using Tabnet
n_steps step_dim attn_dim ghost_bn virtual_batch_size momentum gamma dropout embed_dropout lr batch_size weight_decay lambda_sparse optimizer lr_scheduler base_lr max_lr div_factor final_div_factor n_cycles val_loss_or_metric
5 32 32 False 128 0.98 1.5 0.1 0.0 0.03 128 0.0 0.0001 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2916
5 64 64 False 128 0.98 1.5 0.2 0.0 0.03 128 0.0 0.0001 Adam ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2938
5 32 32 False 128 0.98 1.5 0.1 0.0 0.03 128 0.0 0.0001 Adam ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2939
5 64 64 False 128 0.98 1.5 0.2 0.0 0.03 128 0.0 0.0001 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2945
5 64 64 False 128 0.98 1.5 0.2 0.0 0.05 128 0.0 0.0001 RAdam ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2962
@jrzaurin
jrzaurin / adult_tabresnet.csv
Last active June 13, 2021 22:04
Results for the Adult Census dataset with TabResnet
blocks_dims blocks_dropout mlp_hidden_dims mlp_activation mlp_dropout mlp_batchnorm mlp_batchnorm_last mlp_linear_first embed_dropout lr batch_size weight_decay optimizer lr_scheduler base_lr max_lr div_factor final_div_factor n_cycles val_loss_or_metric
same 0.5 None relu 0.1 False False False 0.1 0.0004 32 0.0 Adam OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2850
same 0.5 None relu 0.1 False False False 0.0 0.0004 32 0.0 Adam OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2853
same 0.5 None relu 0.1 False False False 0.1 0.0004 128 0.0 AdamW OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2854
same 0.5 None relu 0.1 False False False 0.1 0.0004 64 0.0 AdamW OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2855
same 0.5 None relu 0.1 False False False 0.1 0.0004 32 0.0 AdamW OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2856
@jrzaurin
jrzaurin / adult_tabmlp.csv
Last active June 13, 2021 22:04
Results for the Adult Census dataset with TabMlp
mlp_hidden_dims mlp_activation mlp_dropout mlp_batchnorm mlp_batchnorm_last mlp_linear_first embed_dropout lr batch_size weight_decay optimizer lr_scheduler base_lr max_lr div_factor final_div_factor n_cycles val_loss_or_metric
[400,200] relu 0.5 False False False 0.1 0.001 128 0.0 AdamW ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2857
[400,200] relu 0.5 False False False 0.0 0.0005 128 0.0 Adam CyclicLR 0.0005 0.01 25 10000.0 10.0 0.2860
[100,50] relu 0.2 False False False 0.0 0.0004 128 0.0 Adam OneCycleLR 0.001 0.01 25 1000.0 5.0 0.2860
[400,200] relu 0.5 False False False 0.1 0.001 128 0.0 Adam ReduceLROnPlateau 0.001 0.01 25 10000.0 5.0 0.2861
[400,200] relu 0.5 False False False 0.0 0.0005 128 0.0 RAdam CyclicLR 0.0005 0.01 25 10000.0 10.0 0.2862