05.10 - s3vectors - vstart vector tests error dump

05.10-logs.md

on folio07, ran s3vector test suite on my local version of remote branch wip-s3vector and my feature branch after changes.

1. started vstart with MON=1 OSD=1 MDS=0 MGR=0 RGW=1 ../src/vstart.sh -n -d --rgw_port 12000
2. under src/test/rgw/s3vector, changed port to 12000 in the conf file and ran S3VTESTS_CONF=s3vtests.conf.SAMPLE pytest after setup.
3. got the same errors in both branches (i don't know if there's anything wrong in how i am running tests)

pytest error sumary - first failure in test_put_vectors (consistent across both wip_s3vector and my feature branch)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED s3vector_test.py::test_put_vectors - botocore.exceptions.EndpointConne...
FAILED s3vector_test.py::test_update_vectors - botocore.exceptions.EndpointCo...
FAILED s3vector_test.py::test_put_vectors_dimension_mismatch - botocore.excep...
FAILED s3vector_test.py::test_get_vectors - botocore.exceptions.EndpointConne...
FAILED s3vector_test.py::test_get_vectors_without_data - botocore.exceptions....
FAILED s3vector_test.py::test_list_vectors - botocore.exceptions.EndpointConn...
FAILED s3vector_test.py::test_list_vectors_with_data - botocore.exceptions.En...
FAILED s3vector_test.py::test_list_vectors_without_data - botocore.exceptions...
FAILED s3vector_test.py::test_list_vectors_pagination - botocore.exceptions.E...
FAILED s3vector_test.py::test_list_vectors_exact_pagination - botocore.except...
FAILED s3vector_test.py::test_delete_vectors - botocore.exceptions.EndpointCo...
FAILED s3vector_test.py::test_query_vectors - botocore.exceptions.EndpointCon...
FAILED s3vector_test.py::test_query_vectors_with_distance - botocore.exceptio...
====== 13 failed, 11 passed, 1 skipped, 24 warnings in 118.00s (0:01:57) =======

rgw logs - crashes abruptly after a vector put operation in all cases (from local wip-s3vector-run)

2026-05-10T18:14:11.997+0000 7f60ff9a66c0 15 req 489896397665188422 0.000000000s s3:s3vector_put_vectors compare=0
2026-05-10T18:14:11.997+0000 7f60ff9a66c0 20 req 489896397665188422 0.000000000s s3:s3vector_put_vectors rgw::auth::s3::LocalEngine granted access
2026-05-10T18:14:11.997+0000 7f60ff9a66c0 20 req 489896397665188422 0.000000000s s3:s3vector_put_vectors rgw::auth::s3::AWSAuthStrategy granted access
2026-05-10T18:14:11.997+0000 7f60ff9a66c0  2 req 489896397665188422 0.000000000s s3:s3vector_put_vectors normalizing buckets and tenants
2026-05-10T18:14:11.997+0000 7f60ff9a66c0 10 req 489896397665188422 0.000000000s s->object=<NULL> s->bucket=PutVectors
2026-05-10T18:14:11.997+0000 7f60ff9a66c0 20 lua background: INFO: lua script bytecode not found : script.prerequest.
2026-05-10T18:14:11.997+0000 7f60ff9a66c0 10 req 489896397665188422 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry)
2026-05-10T18:14:11.997+0000 7f60ff9a66c0  2 req 489896397665188422 0.000000000s s3:s3vector_put_vectors init permissions
2026-05-10T18:14:11.997+0000 7f60ff9a66c0  2 req 489896397665188422 0.000000000s s3:s3vector_put_vectors recalculating target
2026-05-10T18:14:11.997+0000 7f60ff9a66c0  2 req 489896397665188422 0.000000000s s3:s3vector_put_vectors reading permissions
2026-05-10T18:14:11.997+0000 7f60ff9a66c0  2 req 489896397665188422 0.000000000s s3:s3vector_put_vectors init op
2026-05-10T18:14:11.997+0000 7f60f61936c0  5 req 489896397665188422 0.000000000s s3:s3vector_put_vectors NOTICE: call to do_aws4_auth_completion
2026-05-10T18:14:11.997+0000 7f60f61936c0 10 req 489896397665188422 0.000000000s s3:s3vector_put_vectors v4 auth ok -- do_aws4_auth_completion
2026-05-10T18:14:12.003+0000 7f60f61936c0  2 req 489896397665188422 0.006000001s s3:s3vector_put_vectors verifying op mask
2026-05-10T18:14:12.003+0000 7f60f61936c0 20 req 489896397665188422 0.006000001s s3:s3vector_put_vectors required_mask= 2 user.op_mask=7
2026-05-10T18:14:12.003+0000 7f60f61936c0  2 req 489896397665188422 0.006000001s s3:s3vector_put_vectors verifying op permissions
2026-05-10T18:14:12.003+0000 7f60f61936c0 10 req 489896397665188422 0.006000001s s3:s3vector_put_vectors INFO: verifying permission for s3vector PutVectors
2026-05-10T18:14:12.003+0000 7f60f61936c0  2 req 489896397665188422 0.006000001s s3:s3vector_put_vectors verifying op params
2026-05-10T18:14:12.003+0000 7f60f61936c0  2 req 489896397665188422 0.006000001s s3:s3vector_put_vectors pre-executing
2026-05-10T18:14:12.003+0000 7f60f61936c0  2 req 489896397665188422 0.006000001s s3:s3vector_put_vectors check rate limiting
2026-05-10T18:14:12.003+0000 7f60f61936c0 10 req 489896397665188422 0.006000001s cache get: name=default.rgw.log++script.postauth. : hit (negative entry)
2026-05-10T18:14:12.003+0000 7f60f61936c0  2 req 489896397665188422 0.006000001s s3:s3vector_put_vectors executing
2026-05-10T18:14:12.003+0000 7f60f61936c0 20 req 489896397665188422 0.006000001s s3:s3vector_put_vectors s3vector --- RadosVectorBucket::load_bucket called
2026-05-10T18:14:12.003+0000 7f60f61936c0 10 req 489896397665188422 0.006000001s s3:s3vector_put_vectors cache get: name=default.rgw.meta+vector+kaboomcbeiyb-14 : hit (requested=0x11, cached=0x13)
2026-05-10T18:14:12.004+0000 7f60f61936c0 20 req 489896397665188422 0.007000001s s3:s3vector_put_vectors INFO: executing s3vector PutVectors with: {"indexName":"test-index","vectorBucketName":"kaboomcbeiyb-14","vectors":[{"key":"vec-0","data":{"float32":[0.74270534515380859,-0.26057958602905273,-0.42312303185462952,-0.65077030658721924,-0.06395440548658371,1.051909327507019,0.66426968574523926,-0.29754447937011719,0.73840594291687012,0.53306013345718384,-0.49847987294197083,-0.18246395885944366,1.5752303600311279,-0.33463379740715027,2.3601915836334229,0.4418589174747467,-0.19860979914665222,0.3025972843170166,-0.085659660398960114,-0.12233930081129074,-0.99463111162185669,0.11428080499172211,-0.086625963449478149,-0.36116510629653931,0.13190475106239319,-0.95056146383285522,-0.90818601846694946,-0.50164890289306641,0.32055783271789551,-1.3031398057937622,-1.8900182247161865,0.15605098009109497,-0.21202906966209412,-0.42217013239860535,0.53652572631835938,0.2360905259847641,1.0757938623428345,-1.3048919439315796,0.66526263952255249,-0.86045849323272705,-0.084552615880966187,0.86831873655319214,-1.0855808258056641,-1.6343122720718384,0.95265859365463257,-0.44566258788108826,-1.4495583772659302,0.49472278356552124,0.141840860247612,-1.3737761974334717,1.088009238243103,0.077662289142608643,0.91995567083358765,-1.3196800947189331,-0.058039262890815735,0.36681163311004639,0.0020676366984844208,-0.5081600546836853,0.55301135778427124,1.4056230783462524,0.92215597629547119,0.84615498781204224,-1.328568696975708,1.3628076314926147,1.0971460342407227,-0.6782461404800415,0.019124796614050865,-1.297571063041687,-0.59601682424545288,1.1113256216049194,-0.94964563846588135,1.2974060773849487,-0.40093111991882324,0.20452465116977692,-0.46095564961433411,-2.4689326286315918,-0.23579609394073486,0.56255805492401123,-0.016080809757113457,0.58253359794616699,1.2376577854156494,-0.21276234090328217,0.095415979623794556,0.18010415136814117,0.48874187469482422,-0.99043565988540649,1.2537965774536133,0.74987554550170898,-0.5245169997215271,-0.091057784855365753,0.14258408546447754,-0.27769339084625244,-1.3373312950134277,0.35815250873565674,0.61938637495040894,-0.38669377565383911,-0.15372470021247864,-0.15769892930984497,-0.53655785322189331,-1.9800230264663696,1.1850749254226685,-0.30871275067329407,-0.46692976355552673,0.22337976098060608,0.21227826178073883,-1.6751916408538818,0.36347952485084534,-1.1417449712753296,-0.17912769317626953,-1.9511328935623169,-1.0967243909835815,-0.48749855160713196,-0.20059838891029358,0.08180622011423111,0.20066322386264801,1.1426290273666382,2.371720552444458,0.033026318997144699,-1.5122945308685303,0.8856852650642395,-0.35716459155082703,-1.3697710037231445,-0.3273347020149231,-1.1498961448669434,1.3770555257797241,0.86132586002349854,-0.17901559174060822,-0.22225148975849152]}},{"key":"vec-1","data":{"float32":[0.53821456432342529,0.19514927268028259,1.1460603475570679,-0.61523580551147461,-0.89520293474197388,2.5175564289093018,1.0457079410552979,1.7082880735397339,1.7546358108520508,2.2327051162719727,1.1583229303359985,1.4113439321517944,0.72321903705596924,1.6352502107620239,1.643608570098877,-1.1431884765625,2.3204500675201416,0.36595705151557922,2.4473624229431152,-0.9164460301399231,0.22651095688343048,0.3745180070400238,0.57146203517913818,-1.8902252912521362,0.29857879877090454,-0.15651294589042664,0.58032453060150146,0.38825243711471558,1.3368101119995117,0.90879029035568237,-0.18516521155834198,1.7731808423995972,0.79649847745895386,1.1936022043228149,2.4614675045013428,0.56162196397781372,1.6265022754669189,1.305419921875,-0.04361874982714653,0.71764802932739258,-0.79747861623764038,1.9147394895553589,2.7309057712554932,3.1615891456604004,3.801497220993042,2.6910417079925537,0.3460332453250885,1.3081140518188477,1.0151368379592896,-0.28310465812683105,0.31689593195915222,0.79227691888809204,0.044447153806686401,2.1729536056518555,-0.53237044811248779,0.8233223557472229,0.96450996398925781,2.3042042255401611,0.20712064206600189,1.6174185276031494,2.1229796409606934,0.30244100093841553,0.75783920288085938,0.55013018846511841,-0.22012622654438019,1.076433539390564,0.69049721956253052,2.3368914127349854,2.0724167823791504,0.45227059721946716,1.6661022901535034,1.4011673927307129,0.14513236284255981,0.59808659553527832,-0.47612527012825012,0.84509861469268799,1.1392439603805542,1.383757472038269,0.26685532927513123,3.4164652824401855,2.2405524253845215,0.79214012622833252,0.9511837363243103,1.9993098974227905,2.1168761253356934,2.176257848739624,2.2314088344573975,1.1478061676025391,1.433594822883606,3.3441522121429443,0.00032472962629981339,-0.54771077632904053,0.7999577522277832,0.54784262180328369,2.5082931518554688,1.2540453672409058,-0.16992028057575226,0.58119857311248779,-1.3334032297134399,0.79693090915679932,0.12691076099872589,0.84573590755462646,1.1734797954559326,0.69613581895828247,0.67618048191070557,0.85728549957275391,0.30958414077758789,1.6990257501602173,-0.063933856785297394,1.3782435655593872,0.94393932819366455,0.038182687014341354,2.5521948337554932,2.0824825763702393,0.96566808223724365,2.5386645793914795,0.58417433500289917,1.1981542110443115,0.60901987552642822,0.6435888409614563,3.2443845272064209,0.79199934005737305,0.66211938858032227,2.5669558048248291,1.7251209020614624,0.54081612825393677,1.0432641506195068,1.3853549957275391]}},{"key":"vec-2","data":{"float32":[2.1945805549621582,1.7928832769393921,0.87540131807327271,1.2124145030975342,2.3355295658111572,2.9147164821624756,1.8434712886810303,4.0410265922546387,3.6866662502288818,1.1991873979568481,2.2535085678100586,3.2986075878143311,-0.061496078968048096,1.7297924757003784,0.66759729385375977,2.6380760669708252,3.5476894378662109,0.9633522629737854,1.929690957069397,3.0931508541107178,1.7533235549926758,2.6042747497558594,1.3011263608932495,1.3402045965194702,2.87333083152771,2.9674072265625,3.2079012393951416,2.791759729385376,1.6047952175140381,2.4605045318603516,1.386744499206543,2.059450626373291,1.698777437210083,1.8302042484283447,4.2290754318237305,0.89178657531738281,2.3620080947875977,2.087285041809082,1.7424598932266235,1.2189050912857056,2.2438197135925293,2.2078609466552734,1.2328141927719116,2.8388180732727051,1.1539956331253052,2.9529922008514404,2.1119182109832764,3.2145674228668213,1.7625240087509155,1.0898340940475464,1.8989386558532715,1.8214157819747925,-0.56355971097946167,2.122715950012207,1.6769778728485107,2.5648713111877441,4.8391456604003906,2.9285337924957275,1.1604577302932739,2.2165946960449219,3.5908591747283936,0.5532565712928772,1.4920871257781982,0.97674983739852905,1.324041485786438,1.1863024234771729,2.4953451156616211,2.1618542671203613,1.2666079998016357,1.9516054391860962,1.0726842880249023,1.63670814037323,2.28631591796875,2.1310892105102539,1.3687120676040649,2.245215892791748,1.6707518100738525,2.0814085006713867,-0.80452042818069458,1.8463019132614136,2.8708994388580322,0.18413560092449188,3.1189219951629639,2.5387551784515381,2.8295199871063232,0.63947981595993042,1.4893808364868164,2.5047233104705811,0.26595926284790039,2.7929081916809082,3.5981781482696533,0.62929576635360718,1.9418885707855225,0.54796814918518066,0.1239934116601944,3.1173348426818848,3.5433773994445801,4.0139670372009277,2.5199244022369385,2.4619009494781494,2.6256201267242432,1.3011665344238281,0.46597146987915039,1.4200656414031982,1.4714598655700684,2.6610395908355713,1.9207782745361328,0.43086099624633789,3.1833300590515137,2.4598643779754639,1.9473115205764771,3.065382719039917,0.92172551155090332,3.9019961357116699,3.3373584747314453,1.9153118133544922,1.5452141761779785,1.9324948787689209,1.8805195093154907,0.20594920217990875,4.4809236526489258,1.4993938207626343,2.6425304412841797,2.1370799541473389,1.2190821170806885,2.3126020431518555,2.9257652759552002,0.56550216674804688]}},{"key":"vec-3","data":{"float32":[2.0949294567108154,4.4470553398132324,3.0268621444702148,2.4736731052398682,3.5344963073730469,2.9855494499206543,1.036859393119812,4.744842529296875,3.4655029773712158,3.3818116188049316,2.4369924068450928,2.0898454189300537,4.1412901878356934,2.871164083480835,3.1775896549224854,1.7848192453384399,2.4551651477813721,3.3592405319213867,3.5387363433837891,1.0721006393432617,3.0477993488311768,3.8052654266357422,2.3287665843963623,4.0913147926330566,3.5997383594512939,3.6121718883514404,1.9306149482727051,4.0231342315673828,4.1407999992370605,2.3623261451721191,1.8169181346893311,3.3394622802734375,3.7058489322662354,5.1794734001159668,3.4839634895324707,1.3833014965057373,2.0712647438049316,4.3737325668334961,1.6606740951538086,3.0497159957885742,2.1928889751434326,2.1324312686920166,2.8551201820373535,3.3062455654144287,3.8688869476318359,2.6974704265594482,2.6050012111663818,2.8306527137756348,4.5563230514526367,2.9737112522125244,0.96276652812957764,2.729079008102417,1.787788987159729,4.1758475303649902,3.1949682235717773,1.7400206327438354,3.7980852127075195,2.4445197582244873,3.7079343795776367,2.7067773342132568,2.1448953151702881,3.9427485466003418,3.810157299041748,2.7014670372009277,3.1926796436309814,4.4600419998168945,3.3300976753234863,3.4809033870697021,2.0554790496826172,3.0636732578277588,1.833332896232605,3.6951260566711426,2.1989381313323975,3.2794461250305176,4.1868381500244141,4.6422305107116699,2.7427752017974854,2.8279018402099609,2.0788054466247559,2.9891128540039062,3.2076354026794434,1.3164454698562622,3.34918212890625,1.62732994556427,3.9637413024902344,4.2766904830932617,5.3926310539245605,2.7348854541778564,2.5142409801483154,2.2015435695648193,3.5461745262145996,1.9444279670715332,2.7954063415527344,3.7451794147491455,1.9419890642166138,3.0292470455169678,2.2632544040679932,2.6820051670074463,2.4485893249511719,1.0774942636489868,4.344489574432373,3.6457176208496094,3.0224008560180664,3.3811390399932861,3.1449093818664551,1.8289568424224854,3.3450984954833984,3.3985474109649658,2.476266622543335,1.9573684930801392,1.9032511711120605,2.0621469020843506,1.8886406421661377,2.8337600231170654,3.1207613945007324,1.8730530738830566,0.84500235319137573,1.3852146863937378,3.0250949859619141,2.0164129734039307,1.4502168893814087,4.4129576683044434,1.1467500925064087,4.9639129638671875,4.2976655960083008,1.7880715131759644,2.8447587490081787,4.1379141807556152]}},{"key":"vec-4","data":{"float32":[2.0327627658843994,3.9691755771636963,4.7303309440612793,2.4486451148986816,4.0035524368286133,4.115506649017334,3.208432674407959,4.0144338607788086,2.8058083057403564,3.7831830978393555,4.6720809936523438,3.1440558433532715,1.9840492010116577,4.0365767478942871,3.2772657871246338,3.8606295585632324,4.0399103164672852,3.4673871994018555,5.7448649406433105,4.436429500579834,4.7596807479858398,1.878244161605835,2.4179577827453613,5.9421768188476562,3.3469619750976562,4.4626851081848145,3.0015733242034912,4.345787525177002,5.6555876731872559,2.831568717956543,3.808307409286499,5.4855308532714844,3.0516970157623291,4.0208010673522949,4.5626769065856934,4.1908426284790039,5.0440402030944824,4.5438032150268555,4.3235573768615723,4.55914306640625,3.499326229095459,3.5689749717712402,3.6303555965423584,5.7137703895568848,4.439640998840332,3.3921079635620117,2.2584280967712402,3.469994068145752,2.6595804691314697,2.8201112747192383,4.4530773162841797,4.8577532768249512,4.1214356422424316,4.7529191970825195,4.1661458015441895,3.9581413269042969,4.905606746673584,5.4876303672790527,1.9964791536331177,3.9582664966583252,4.6479573249816895,4.4823641777038574,3.4475605487823486,5.0648493766784668,3.8043143749237061,3.440025806427002,2.23201584815979,3.3391356468200684,2.9905133247375488,4.2476954460144043,4.7765145301818848,5.0880923271179199,4.2649307250976562,3.5765120983123779,5.1525492668151855,5.8147697448730469,3.3042230606079102,2.8456082344055176,3.9022493362426758,4.9296107292175293,2.9246542453765869,2.2333869934082031,5.6725373268127441,5.6339454650878906,5.6685342788696289,2.4982750415802002,2.8128859996795654,3.0927281379699707,2.5975899696350098,5.1736116409301758,3.9208517074584961,4.2797393798828125,2.9031903743743896,4.637359619140625,4.1855125427246094,3.691321849822998,5.4172568321228027,5.0096588134765625,2.7505474090576172,4.2593107223510742,4.7750182151794434,3.4595191478729248,4.6658334732055664,4.808990478515625,3.0115528106689453,3.1539721488952637,3.9152629375457764,3.4582479000091553,2.1429574489593506,4.2380795478820801,4.6477546691894531,5.6532649993896484,3.644073486328125,2.6897299289703369,4.2612528800964355,4.3407392501831055,2.6942522525787354,3.0764164924621582,2.6368787288665771,5.07281494140625,6.1465883255004883,2.9932677745819092,2.6776108741760254,3.5067574977874756,3.4776389598846436,4.9760489463806152,5.4724245071411133,4.6243147850036621]}},{"key":"vec-5","data":{"float32":[5.2350893020629883,4.9248089790344238,4.8803210258483887,6.1443848609924316,5.6195902824401855,5.6817288398742676,5.8598489761352539,3.6856842041015625,6.3433308601379395,4.8469891548156738,5.4760951995849609,5.8684873580932617,5.0486869812011719,4.7447152137756348,4.704526424407959,3.3157222270965576,4.4556846618652344,4.3273115158081055,3.5224390029907227,3.5692036151885986,5.7598056793212891,3.8995285034179688,6.3188157081604004,4.063478946685791,4.386998176574707,3.8582396507263184,3.8793814182281494,5.4206318855285645,4.4476685523986816,3.0249073505401611,5.697117805480957,5.7961978912353516,4.2416038513183594,4.3055033683776855,4.2887330055236816,6.0759892463684082,5.9690370559692383,5.0712995529174805,5.1035184860229492,6.3630304336547852,4.8307876586914062,3.994140625,6.3172016143798828,5.404545783996582,5.8929171562194824,7.0679669380187988,5.6371712684631348,5.0937871932983398,7.8268599510192871,4.655919075012207,4.6031537055969238,5.5181007385253906,4.4120159149169922,6.7267918586730957,5.5000271797180176,4.840364933013916,5.6987361907958984,5.7463908195495605,5.3486032485961914,6.985985279083252,4.3346700668334961,5.9861550331115723,4.9950881004333496,4.1826076507568359,5.1316452026367188,4.8294129371643066,4.8233480453491211,6.4848446846008301,4.5205669403076172,3.394376277923584,6.4893484115600586,5.9927940368652344,4.3216967582702637,4.3256158828735352,4.4675674438476562,5.000943660736084,4.4180870056152344,5.5108604431152344,3.9593355655670166,5.3155150413513184,3.5380237102508545,4.8018941879272461,4.8845043182373047,4.4181046485900879,5.4846696853637695,4.8684592247009277,5.3822751045227051,3.5026760101318359,5.1088528633117676,5.1582489013671875,6.1518659591674805,6.098142147064209,6.016624927520752,5.2011833190917969,5.5850539207458496,4.1452469825744629,4.9827656745910645,6.2661881446838379,4.236290454864502,5.3271164894104004,3.7077863216400146,4.6137547492980957,4.067866325378418,6.1805248260498047,7.0881643295288086,6.64898681640625,6.6782116889953613,5.1057462692260742,4.8528313636779785,5.4440450668334961,4.5335073471069336,4.494534969329834,4.4758610725402832,5.6014413833618164,4.4914531707763672,3.0056252479553223,2.7979488372802734,4.8403401374816895,5.3371057510375977,4.0769295692443848,2.4669632911682129,3.845954418182373,4.8848247528076172,4.9137482643127441,6.2679934501647949,6.6960053443908691,4.1422853469848633,5.7426018714904785]}},{"key":"vec-6","data":{"float32":[5.4665403366088867,4.839411735534668,6.6096973419189453,5.3203396797180176,8.1518230438232422,6.3437981605529785,6.8281416893005371,5.6324071884155273,5.9685220718383789,4.8637351989746094,6.5186753273010254,5.6726841926574707,4.8397951126098633,6.3962502479553223,7.7255387306213379,5.9585323333740234,6.2268486022949219,5.8147249221801758,5.2403497695922852,5.388916015625,3.9943797588348389,7.4659242630004883,3.1369774341583252,8.214390754699707,7.313776969909668,5.5006332397460938,5.5027594566345215,5.5384159088134766,5.8184137344360352,4.5729026794433594,6.8877601623535156,6.571265697479248,4.5038642883300781,4.7832403182983398,6.3962879180908203,4.394871711730957,6.4762148857116699,4.1462206840515137,6.0228681564331055,4.3626937866210938,5.1417756080627441,5.5294904708862305,4.8462963104248047,5.7424039840698242,5.3118939399719238,5.4378910064697266,5.6167478561401367,7.8372564315795898,5.9645266532897949,4.0575175285339355,4.6984090805053711,5.5481963157653809,5.2824563980102539,7.5216507911682129,5.2795281410217285,5.8344254493713379,5.5434131622314453,6.2365036010742188,6.7448554039001465,4.096860408782959,5.8301496505737305,4.3203787803649902,7.1183524131774902,7.3966665267944336,5.3326020240783691,5.5896768569946289,6.1845102310180664,4.9603128433227539,6.1458311080932617,4.8994812965393066,5.1510887145996094,5.8947429656982422,4.8441982269287109,7.5804486274719238,5.341585636138916,7.3924827575683594,7.2422022819519043,4.9836955070495605,7.5370354652404785,8.2724580764770508,6.5473504066467285,5.6617531776428223,5.3313083648681641,7.6961750984191895,5.8830037117004395,5.5226001739501953,6.105614185333252,5.638432502746582,5.4969043731689453,6.1051840782165527,6.6167721748352051,7.1980524063110352,7.9728307723999023,6.505889892578125,5.3122539520263672,5.2691836357116699,6.4409174919128418,5.8752851486206055,5.8115715980529785,4.6312856674194336,6.8593955039978027,7.4930834770202637,6.3023085594177246,4.3994016647338867,4.4615631103515625,6.8674507141113281,7.4466700553894043,6.2174997329711914,6.1698899269104004,5.9281816482543945,5.7049078941345215,6.6789736747741699,5.7763857841491699,6.0641598701477051,9.0664310455322266,4.3232083320617676,6.0123906135559082,7.0311808586120605,5.6094827651977539,5.4867100715637207,7.3818979263305664,3.8770260810852051,4.1063475608825684,4.638676643371582,6.3586688041687012,6.8568391799926758,5.0987253189086914,5.5257077217102051]}},{"key":"vec-7","data":{"float32":[6.3863921165466309,7.510429859161377,7.7310881614685059,6.6109275817871094,7.0409865379333496,6.1897916793823242,7.5675778388977051,7.0943403244018555,6.2706289291381836,6.9534916877746582,9.6377429962158203,7.188453197479248,7.7435407638549805,6.6161432266235352,7.9539661407470703,7.552734375,5.943448543548584,7.8489103317260742,5.5023837089538574,8.6014060974121094,7.5423297882080078,5.6128411293029785,6.9379239082336426,8.0352344512939453,7.527040958404541,7.2648277282714844,6.4040379524230957,6.2825827598571777,7.2296442985534668,6.4100065231323242,5.0097713470458984,7.8759441375732422,6.7088708877563477,6.6480135917663574,4.8872671127319336,10.238924026489258,8.0790338516235352,5.7693281173706055,7.5385527610778809,5.9227743148803711,7.3029880523681641,8.7689723968505859,6.4814953804016113,6.0070991516113281,5.4719133377075195,9.2889499664306641,5.3240599632263184,8.5834131240844727,8.5988979339599609,8.4108543395996094,7.9119930267333984,4.8269186019897461,7.0413069725036621,6.7626461982727051,8.2827520370483398,6.3039364814758301,7.7233099937438965,6.0222468376159668,6.6485018730163574,6.5471982955932617,6.6073479652404785,7.6823272705078125,7.3856759071350098,8.3645315170288086,7.3519997596740723,7.3800830841064453,6.6280875205993652,6.5247392654418945,8.3295602798461914,9.0938711166381836,7.7377572059631348,6.9101424217224121,8.4001474380493164,6.6066927909851074,6.6909313201904297,9.2635679244995117,7.7334356307983398,7.2502036094665527,7.4625506401062012,8.4199924468994141,8.4119405746459961,7.0230655670166016,6.9789881706237793,7.1797637939453125,8.5017433166503906,7.0893149375915527,7.3152732849121094,7.8821706771850586,7.4935851097106934,7.6160273551940918,6.7093558311462402,6.332705020904541,6.7994370460510254,7.1109871864318848,7.4034233093261719,6.2441306114196777,7.0957431793212891,7.3051028251647949,6.4201064109802246,8.1940279006958008,6.6756701469421387,7.9497289657592773,4.418891429901123,7.1924304962158203,6.6766266822814941,7.1414427757263184,8.3323516845703125,6.0024771690368652,6.4987869262695312,7.3644824028015137,5.6996908187866211,7.7956891059875488,8.2347698211669922,6.8805642127990723,6.2992706298828125,6.5353412628173828,7.3718104362487793,6.9932198524475098,6.2382669448852539,6.4218263626098633,7.966343879699707,6.8423161506652832,8.7602672576904297,7.087287425994873,6.140113353729248,5.7564234733581543,9.0155878067016602,8.0181407928466797]}},{"key":"vec-8","data":{"float32":[8.714447021484375,8.5080404281616211,7.8557300567626953,9.7255210876464844,7.7556614875793457,8.4677753448486328,8.1971321105957031,6.8430118560791016,8.236882209777832,8.9718160629272461,8.9234514236450195,7.592404842376709,7.7512531280517578,6.7865414619445801,8.3133411407470703,7.6187362670898438,8.2099266052246094,9.611393928527832,8.3650608062744141,7.2523036003112793,6.6065683364868164,6.6232080459594727,10.046977043151855,8.01666259765625,6.7394261360168457,8.9673233032226562,9.1390600204467773,8.1593341827392578,9.0501937866210938,9.4370336532592773,9.2708835601806641,7.5847582817077637,6.6988792419433594,10.298867225646973,9.1666965484619141,9.3089914321899414,8.9269418716430664,8.0984573364257812,8.8116579055786133,6.2551708221435547,7.3760690689086914,6.874058723449707,7.5993475914001465,8.6482667922973633,8.3272762298583984,7.5308632850646973,8.5407342910766602,7.9387979507446289,8.4666013717651367,8.7548103332519531,8.6025524139404297,7.132901668548584,7.9703526496887207,7.5212516784667969,7.3669619560241699,8.0660486221313477,10.321146011352539,9.429774284362793,6.9695863723754883,9.4372682571411133,8.0487499237060547,7.8806920051574707,8.9960756301879883,7.3899798393249512,8.5428762435913086,9.794438362121582,8.8036041259765625,7.2753825187683105,7.3331818580627441,7.1335635185241699,7.0309081077575684,7.9969367980957031,8.5532503128051758,8.286412239074707,9.911952018737793,8.7549972534179688,9.3737735748291016,5.887967586517334,8.2097177505493164,8.961461067199707,6.670569896697998,9.5460147857666016,10.184636116027832,8.6762161254882812,5.1777515411376953,7.8323068618774414,7.3827767372131348,6.5321354866027832,6.0226283073425293,7.629145622253418,9.1481246948242188,8.1612367630004883,7.729163646697998,7.6924252510070801,8.4678068161010742,8.3762035369873047,7.2596521377563477,8.3632135391235352,6.3435564041137695,7.7061243057250977,9.6533241271972656,7.1265263557434082,9.2319145202636719,8.3414850234985352,9.0503664016723633,9.5677757263183594,9.3442754745483398,7.3792877197265625,8.3122768402099609,6.9113259315490723,7.9592585563659668,8.38018798828125,7.4513835906982422,6.9130830764770508,9.3964347839355469,7.1397795677185059,9.636378288269043,8.3877735137939453,10.046219825744629,8.5042800903320312,8.4745254516601562,9.1591243743896484,7.9808692932128906,9.6269464492797852,8.5490541458129883,8.4410562515258789,7.0842676162719727,10.420576095581055]}},{"key":"vec-9","data":{"float32":[8.3127841949462891,7.5461239814758301,9.0518198013305664,8.2424659729003906,8.650787353515625,10.042887687683105,8.5397043228149414,9.8260440826416016,8.9525489807128906,7.7836251258850098,8.5370893478393555,9.6115751266479492,8.7462196350097656,10.074551582336426,8.6521310806274414,9.6464414596557617,9.3688240051269531,8.5235824584960938,9.6563167572021484,8.6193046569824219,8.5870046615600586,8.3201770782470703,10.957012176513672,7.4051094055175781,9.4356727600097656,9.8129158020019531,8.8811550140380859,7.5090880393981934,11.357903480529785,9.7315502166748047,10.113945007324219,9.9471311569213867,10.235204696655273,8.7592744827270508,8.9942178726196289,8.9072885513305664,7.917750358581543,8.5500564575195312,10.081280708312988,8.9501094818115234,9.926905632019043,8.34503173828125,8.8143415451049805,8.7262287139892578,8.4231500625610352,8.0677337646484375,11.658047676086426,9.1217937469482422,9.9591093063354492,11.457210540771484,9.5471172332763672,9.9980840682983398,8.5155496597290039,10.646119117736816,10.93685245513916,9.5789813995361328,9.0427017211914062,7.9787793159484863,9.9041404724121094,8.1665153503417969,9.9368104934692383,7.5527963638305664,9.728327751159668,8.2439193725585938,9.3694887161254883,6.9376964569091797,7.9348869323730469,9.1356954574584961,8.7663373947143555,9.4359636306762695,7.8728218078613281,9.1665058135986328,6.5023565292358398,8.3594446182250977,10.36433219909668,7.3353972434997559,9.631962776184082,8.1980819702148438,8.3364076614379883,9.2067356109619141,7.2586097717285156,8.7015037536621094,7.6718649864196777,7.4357552528381348,6.512570858001709,7.0182547569274902,11.22458553314209,6.7530007362365723,8.7124385833740234,9.4903154373168945,7.3545727729797363,8.6353120803833008,10.548154830932617,10.715799331665039,7.8893251419067383,8.1671218872070312,11.779814720153809,8.92840576171875,8.7267484664916992,9.7058982849121094,10.334051132202148,8.8273868560791016,9.1700592041015625,8.126678466796875,8.4436321258544922,8.7473173141479492,8.7551546096801758,8.9492464065551758,8.5906496047973633,10.246882438659668,7.6430845260620117,8.5532979965209961,8.9177637100219727,8.8232002258300781,9.8551521301269531,9.2405796051025391,8.3029260635375977,10.499213218688965,8.3186588287353516,8.9873466491699219,7.3326644897460938,8.3738784790039062,8.4092864990234375,10.665643692016602,8.2678775787353516,8.9228925704956055,8.2033424377441406,8.0336360931396484]}}]}
2026-05-10T18:14:12.005+0000 7f60f61936c0 20 req 489896397665188422 0.008000002s s3:s3vector_put_vectors INFO: s3vector created record batch with 10 rows
2026-05-10T18:14:12.005+0000 7f60f61936c0 10 req 489896397665188422 0.008000002s s3:s3vector_put_vectors INFO: s3vector attempting to upsert 10 vectors with dimension 128

gdb debugging

cd build
gdb -q bin/radosgw -p <PID> \
  -ex "set pagination off" \
  -ex "set confirm off" \
  -ex "set print thread-events off" \
  -ex "set logging file /tmp/rgw-gdb.txt" \
  -ex "set logging overwrite on" \
  -ex "set logging on" \
  -ex "continue"

my feature logs - gdb tail

#38 0x00007f730bfb4b2b in either::{impl#12}::poll<futures_util::future::future::Map<lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}, lance::dataset::write::retry::maybe_timeout::{closure_env#0}<core::result::Result<lance::dataset::write::merge_insert::UncommittedMergeInsert, lance_core::error::Error>, lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}>>, futures_util::future::try_future::MapErr<tokio::time::timeout::Timeout<lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}>, lance::dataset::write::retry::maybe_timeout::{closure_env#1}<core::result::Result<lance::dataset::write::merge_insert::UncommittedMergeInsert, lance_core::error::Error>, lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}>>> (self=..., cx=0x7f71d88135d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/either-1.15.0/src/lib.rs:1153
        inner = core::pin::Pin<&mut futures_util::future::future::Map<lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}, lance::dataset::write::retry::maybe_timeout::{closure_env#0}<core::result::Result<lance::dataset::write::merge_insert::UncommittedMergeInsert, lance_core::error::Error>, lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}>>> {pointer: 0x56169406f5b0}
#39 0x00007f730c189447 in lance::dataset::write::retry::execute_with_retry::{async_fn#0}<lance::dataset::write::merge_insert::MergeInsertJobWithIterator> () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.39.0/src/dataset/write/retry.rs:94
*** stack smashing detected ***: terminated


Fatal signal: Aborted
----- Backtrace -----
Backtrace unavailable
---------------------
A fatal error internal to GDB has been detected, further
debugging is not possible.  GDB will now terminate.

This is a bug, please report it.  For instructions, see:
<https://www.gnu.org/software/gdb/bugs/>.

wip-s3-vector logs (backtrace for put_vector, first test that fails and crashes the rgw)

Thread 161 "io_context_pool" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fbd630f36c0 (LWP 960515)]
0x00007fbe4e796a03 in sqlparser::parser::Parser::parse_prefix (self=0x0) at src/parser/mod.rs:1496
1496	    pub fn parse_prefix(&mut self) -> Result<Expr, ParserError> {
Missing rpms, try: dnf --enablerepo='*debug*' install gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 libicu-debuginfo-67.1-10.el9.x86_64 expat-debuginfo-2.5.0-5.el9.x86_64 lmdb-libs-debuginfo-0.9.29-3.el9.x86_64 lua-libs-debuginfo-5.4.4-4.el9.x86_64 openssl-libs-debuginfo-3.5.1-3.el9.x86_64 librdkafka-debuginfo-1.6.1-102.el9.x86_64 librabbitmq-debuginfo-0.11.0-7.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 glibc-debuginfo-2.39-17.el10.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 systemd-libs-debuginfo-252-55.el9.x86_64 libibverbs-debuginfo-57.0-1.el9.x86_64 librdmacm-debuginfo-57.0-1.el9.x86_64 libcurl-debuginfo-7.76.1-31.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 sqlite-libs-debuginfo-3.34.1-8.el9.x86_64 libstdc++-debuginfo-14.1.1-5.el10.x86_64 libgcc-debuginfo-14.1.1-5.el10.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libzstd-debuginfo-1.5.5-1.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-13.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 libbrotli-debuginfo-1.0.9-7.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 systemd-container-debuginfo-252-55.el9.x86_64
#0  0x00007fbe4e796a03 in sqlparser::parser::Parser::parse_prefix (self=0x0) at src/parser/mod.rs:1496
#1  0x00007fbe4e7aa256 in sqlparser::parser::Parser::parse_subexpr (self=0x7fbd13695130, precedence=0) at src/parser/mod.rs:1233
#2  0x00007fbe4e7765f9 in sqlparser::parser::Parser::parse_expr (self=0x7fbd13695130) at src/parser/mod.rs:1207
#3  0x00007fbe4e7faee8 in sqlparser::parser::Parser::parse_wildcard_expr (self=0x7fbd13695130) at src/parser/mod.rs:1202
#4  0x00007fbe4e7cf1f6 in sqlparser::parser::Parser::parse_select_item (self=0x7fbd13695130) at src/parser/mod.rs:15016
#5  0x00007fbe4e7bea09 in sqlparser::parser::{impl#6}::parse_projection::{closure#0} (p=0x7fbd13695130) at src/parser/mod.rs:4344
#6  0x00007fbe4e881dc1 in sqlparser::parser::Parser::parse_comma_separated_with_trailing_commas<sqlparser::ast::query::SelectItem, sqlparser::parser::{impl#6}::parse_projection::{closure_env#0}, fn(&sqlparser::keywords::Keyword, &mut sqlparser::parser::Parser) -> bool> (self=0x7fbd13695130, f=..., trailing_commas=false, is_reserved_keyword=0x0) at src/parser/mod.rs:4458
#7  0x00007fbe4e7be9c7 in sqlparser::parser::Parser::parse_projection (self=0x7fbd13695130) at src/parser/mod.rs:4343
#8  0x00007fbe4e79c0c4 in sqlparser::parser::Parser::parse_select (self=0x7fbd13695130) at src/parser/mod.rs:11801
#9  0x00007fbe4e7beaa9 in sqlparser::parser::Parser::parse_query_body (self=0x7fbd13695130, precedence=0) at src/parser/mod.rs:11651
#10 0x00007fbe4e789fa7 in sqlparser::parser::Parser::parse_query (self=0x7fbd13695130) at src/parser/mod.rs:11180
#11 0x00007fbe4e7b79c3 in sqlparser::parser::Parser::parse_statement (self=0x7fbd13695130) at src/parser/mod.rs:557
#12 0x00007fbe4ade2003 in lance_datafusion::sql::parse_statement (statement=...) at src/sql.rs:117
#13 0x00007fbe4ade1536 in lance_datafusion::sql::parse_sql_expr (expr=...) at src/sql.rs:68
#14 0x00007fbe4ae0c679 in lance_datafusion::planner::Planner::parse_expr (self=0x7fbd13699bf8, expr=...) at src/planner.rs:864
#15 0x00007fbe4adc77c1 in lance_datafusion::projection::ProjectionBuilder::add_column (self=0x7fbd13699bf8, output_name=..., raw_expr=...) at src/projection.rs:77
#16 0x00007fbe47623b9d in lance_datafusion::projection::ProjectionBuilder::add_columns<&str, alloc::string::String> (self=0x7fbd13699bf8, columns=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-datafusion-0.39.0/src/projection.rs:130
#17 0x00007fbe47622d1d in lance_datafusion::projection::ProjectionPlan::from_expressions<&str, alloc::string::String> (base=..., columns=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-datafusion-0.39.0/src/projection.rs:207
#18 0x00007fbe47aedad6 in lance::dataset::scanner::Scanner::project_with_transform<&str, alloc::string::String> (self=0x56040601e820, columns=...) at src/dataset/scanner.rs:706
#19 0x00007fbe47af9f37 in lance::dataset::scanner::Scanner::project<&alloc::string::String> (self=0x56040601e820, columns=...) at src/dataset/scanner.rs:695
#20 0x00007fbe4729125c in lance::datafusion::dataframe::{impl#6}::scan::{async_block#0} () at src/datafusion/dataframe.rs:121
#21 0x00007fbe4b8576e0 in core::future::future::{impl#1}::poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>, datafusion_common::error::DataFusionError>> + core::marker::Send), alloc::alloc::Global>> (self=..., cx=0x7fbd136af980) at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/future/future.rs:133
#22 0x00007fbe4af661d3 in datafusion::physical_planner::{impl#1}::map_logical_node_to_physical::{async_fn#0} () at src/physical_planner.rs:462
#23 0x00007fbe4af4893e in datafusion::physical_planner::{impl#1}::task_helper::{async_fn#0} () at src/physical_planner.rs:378
#24 0x00007fbe4af0cc80 in futures_util::stream::futures_unordered::{impl#7}::poll_next<datafusion::physical_planner::{impl#1}::task_helper::{async_fn_env#0}> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_unordered/mod.rs:528
#25 0x00007fbe4af17ee1 in futures_util::stream::stream::StreamExt::poll_next_unpin<futures_util::stream::futures_unordered::FuturesUnordered<datafusion::physical_planner::{impl#1}::task_helper::{async_fn_env#0}>> (self=0x560405ecbdd8, cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638
#26 0x00007fbe4afdeeee in futures_util::stream::stream::buffer_unordered::{impl#2}::poll_next<futures_util::stream::iter::Iter<core::iter::adapters::map::Map<alloc::vec::into_iter::IntoIter<usize, alloc::alloc::Global>, datafusion::physical_planner::{impl#1}::create_initial_plan::{async_fn#0}::{closure_env#1}>>> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/buffer_unordered.rs:75
#27 0x00007fbe4afffb8b in futures_core::stream::{impl#4}::try_poll_next<futures_util::stream::stream::buffer_unordered::BufferUnordered<futures_util::stream::iter::Iter<core::iter::adapters::map::Map<alloc::vec::into_iter::IntoIter<usize, alloc::alloc::Global>, datafusion::physical_planner::{impl#1}::create_initial_plan::{async_fn#0}::{closure_env#1}>>>, core::option::Option<alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>>, datafusion_common::error::DataFusionError> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206
#28 0x00007fbe4afc64f7 in futures_util::stream::try_stream::try_collect::{impl#2}::poll<futures_util::stream::stream::buffer_unordered::BufferUnordered<futures_util::stream::iter::Iter<core::iter::adapters::map::Map<alloc::vec::into_iter::IntoIter<usize, alloc::alloc::Global>, datafusion::physical_planner::{impl#1}::create_initial_plan::{async_fn#0}::{closure_env#1}>>>, alloc::vec::Vec<core::option::Option<alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>>, alloc::alloc::Global>> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/try_collect.rs:46
#29 0x00007fbe4af4e346 in datafusion::physical_planner::{impl#1}::create_initial_plan::{async_fn#0} () at src/physical_planner.rs:341
#30 0x00007fbe4af70a72 in datafusion::physical_planner::{impl#8}::create_physical_plan::{async_block#0} () at src/physical_planner.rs:191
#31 0x00007fbe4b8576e0 in core::future::future::{impl#1}::poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>, datafusion_common::error::DataFusionError>> + core::marker::Send), alloc::alloc::Global>> (self=..., cx=0x7fbd136f35d8) at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/future/future.rs:133
#32 0x00007fbe46fd7977 in lance::dataset::write::merge_insert::{impl#3}::create_plan::{async_fn#0} () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.39.0/src/dataset/write/merge_insert.rs:1313
#33 0x00007fbe46fe56f1 in lance::dataset::write::merge_insert::{impl#3}::execute_uncommitted_v2::{async_fn#0} () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.39.0/src/dataset/write/merge_insert.rs:1322
#34 0x00007fbe46fe7947 in lance::dataset::write::merge_insert::{impl#3}::execute_uncommitted_impl::{async_fn#0} () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.39.0/src/dataset/write/merge_insert.rs:1425
#35 0x00007fbe46fbd164 in lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn#0} () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.39.0/src/dataset/write/merge_insert.rs:1775
#36 0x00007fbe470e0b71 in futures_util::future::future::map::{impl#2}::poll<lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}, lance::dataset::write::retry::maybe_timeout::{closure_env#0}<core::result::Result<lance::dataset::write::merge_insert::UncommittedMergeInsert, lance_core::error::Error>, lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}>, core::result::Result<core::result::Result<lance::dataset::write::merge_insert::UncommittedMergeInsert, lance_core::error::Error>, lance_core::error::Error>> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55
#37 0x00007fbe47016153 in futures_util::future::future::{impl#15}::poll<lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}, lance::dataset::write::retry::maybe_timeout::{closure_env#0}<core::result::Result<lance::dataset::write::merge_insert::UncommittedMergeInsert, lance_core::error::Error>, lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}>> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86
#38 0x00007fbe46fb4b2b in either::{impl#12}::poll<futures_util::future::future::Map<lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}, lance::dataset::write::retry::maybe_timeout::{closure_env#0}<core::result::Result<lance::dataset::write::merge_insert::UncommittedMergeInsert, lance_core::error::Error>, lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}>>, futures_util::future::try_future::MapErr<tokio::time::timeout::Timeout<lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}>, lance::dataset::write::retry::maybe_timeout::{closure_env#1}<core::result::Result<lance::dataset::write::merge_insert::UncommittedMergeInsert, lance_core::error::Error>, lance::dataset::write::merge_insert::{impl#4}::execute_impl::{async_fn_env#0}>>> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/either-1.15.0/src/lib.rs:1153
#39 0x00007fbe47189447 in lance::dataset::write::retry::execute_with_retry::{async_fn#0}<lance::dataset::write::merge_insert::MergeInsertJobWithIterator> () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.39.0/src/dataset/write/retry.rs:94
#40 0x00007fbe4708fd4d in core::future::future::{impl#1}::poll<alloc::boxed::Box<lance::dataset::write::retry::execute_with_retry::{async_fn_env#0}<lance::dataset::write::merge_insert::MergeInsertJobWithIterator>, alloc::alloc::Global>> (self=..., cx=0x7fbd136f35d8) at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/future/future.rs:133
#41 0x00007fbe46ff32fc in lance::dataset::write::merge_insert::{impl#3}::execute::{async_fn#0} () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.39.0/src/dataset/write/merge_insert.rs:1247
#42 0x00007fbe46fd812a in lance::dataset::write::merge_insert::{impl#3}::execute_reader::{async_fn#0}<alloc::boxed::Box<(dyn arrow_array::record_batch::RecordBatchReader<Item=core::result::Result<arrow_array::record_batch::RecordBatch, arrow_schema::error::ArrowError>> + core::marker::Send), alloc::alloc::Global>> () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.39.0/src/dataset/write/merge_insert.rs:528
#43 0x00007fbe46fc5f1b in futures_core::future::{impl#2}::try_poll<lance::dataset::write::merge_insert::{impl#3}::execute_reader::{async_fn_env#0}<alloc::boxed::Box<(dyn arrow_array::record_batch::RecordBatchReader<Item=core::result::Result<arrow_array::record_batch::RecordBatch, arrow_schema::error::ArrowError>> + core::marker::Send), alloc::alloc::Global>>, (alloc::sync::Arc<lance::dataset::Dataset, alloc::alloc::Global>, lance::dataset::write::merge_insert::MergeStats), lance_core::error::Error> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/future.rs:92
#44 0x00007fbe470968b3 in futures_util::future::try_future::into_future::{impl#2}::poll<lance::dataset::write::merge_insert::{impl#3}::execute_reader::{async_fn_env#0}<alloc::boxed::Box<(dyn arrow_array::record_batch::RecordBatchReader<Item=core::result::Result<arrow_array::record_batch::RecordBatch, arrow_schema::error::ArrowError>> + core::marker::Send), alloc::alloc::Global>>> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/try_future/into_future.rs:34
#45 0x00007fbe470dfb83 in futures_util::future::future::map::{impl#2}::poll<futures_util::future::try_future::into_future::IntoFuture<lance::dataset::write::merge_insert::{impl#3}::execute_reader::{async_fn_env#0}<alloc::boxed::Box<(dyn arrow_array::record_batch::RecordBatchReader<Item=core::result::Result<arrow_array::record_batch::RecordBatch, arrow_schema::error::ArrowError>> + core::marker::Send), alloc::alloc::Global>>>, futures_util::fns::MapErrFn<lancedb::table::{impl#77}::merge_insert::{async_block#0}::{closure_env#1}>, core::result::Result<(alloc::sync::Arc<lance::dataset::Dataset, alloc::alloc::Global>, lance::dataset::write::merge_insert::MergeStats), lancedb::error::Error>> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55
#46 0x00007fbe47015fc3 in futures_util::future::future::{impl#15}::poll<futures_util::future::try_future::into_future::IntoFuture<lance::dataset::write::merge_insert::{impl#3}::execute_reader::{async_fn_env#0}<alloc::boxed::Box<(dyn arrow_array::record_batch::RecordBatchReader<Item=core::result::Result<arrow_array::record_batch::RecordBatch, arrow_schema::error::ArrowError>> + core::marker::Send), alloc::alloc::Global>>>, futures_util::fns::MapErrFn<lancedb::table::{impl#77}::merge_insert::{async_block#0}::{closure_env#1}>> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86
#47 0x00007fbe46fb7a03 in futures_util::future::try_future::{impl#61}::poll<lance::dataset::write::merge_insert::{impl#3}::execute_reader::{async_fn_env#0}<alloc::boxed::Box<(dyn arrow_array::record_batch::RecordBatchReader<Item=core::result::Result<arrow_array::record_batch::RecordBatch, arrow_schema::error::ArrowError>> + core::marker::Send), alloc::alloc::Global>>, lancedb::table::{impl#77}::merge_insert::{async_block#0}::{closure_env#1}> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86
#48 0x00007fbe47094e0f in futures_util::future::either::{impl#4}::poll<futures_util::future::future::Map<tokio::time::timeout::Timeout<lance::dataset::write::merge_insert::{impl#3}::execute_reader::{async_fn_env#0}<alloc::boxed::Box<(dyn arrow_array::record_batch::RecordBatchReader<Item=core::result::Result<arrow_array::record_batch::RecordBatch, arrow_schema::error::ArrowError>> + core::marker::Send), alloc::alloc::Global>>>, lancedb::table::{impl#77}::merge_insert::{async_block#0}::{closure_env#0}>, futures_util::future::try_future::MapErr<lance::dataset::write::merge_insert::{impl#3}::execute_reader::{async_fn_env#0}<alloc::boxed::Box<(dyn arrow_array::record_batch::RecordBatchReader<Item=core::result::Result<arrow_array::record_batch::RecordBatch, arrow_schema::error::ArrowError>> + core::marker::Send), alloc::alloc::Global>>, lancedb::table::{impl#77}::merge_insert::{async_block#0}::{closure_env#1}>> (self=..., cx=0x7fbd136f35d8) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/either.rs:109
#49 0x00007fbe470b747d in lancedb::table::{impl#77}::merge_insert::{async_block#0} () at src/table.rs:2480
#50 0x00007fbe46d8810f in core::future::future::{impl#1}::poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<lancedb::table::MergeResult, lancedb::error::Error>> + core::marker::Send), alloc::alloc::Global>> (self=..., cx=0x7fbd136f35d8) at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/future/future.rs:133
#51 0x00007fbe46d97b15 in lancedb::table::merge::{impl#0}::execute::{async_fn#0} () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lancedb-0.22.3/src/table/merge.rs:124
#52 0x00007fbe46d9e735 in lancedb::table::lancedb_table_merge_insert::{async_block#0} () at src/table.rs:159
#53 0x00007fbe46dfc0db in tokio::runtime::park::{impl#4}::block_on::{closure#0}<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}> () at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/park.rs:285
#54 0x00007fbe46df9739 in tokio::task::coop::with_budget<core::task::poll::Poll<core::result::Result<lancedb::table::MergeResult, lancedb::error::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}>> (budget=..., f=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167
#55 tokio::task::coop::budget<core::task::poll::Poll<core::result::Result<lancedb::table::MergeResult, lancedb::error::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}>> (f=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133
#56 tokio::runtime::park::CachedParkThread::block_on<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}> (self=0x7fbd136f391f, f=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/park.rs:285
#57 0x00007fbe46d7f4d4 in tokio::runtime::context::blocking::BlockingRegionGuard::block_on<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}> (self=0x7fbd136f3db0, f=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/blocking.rs:66
#58 0x00007fbe46d7ce71 in tokio::runtime::scheduler::multi_thread::{impl#0}::block_on::{closure#0}<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}> (blocking=0x7fbd136f3db0) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/mod.rs:87
#59 0x00007fbe46ccac4b in tokio::runtime::context::runtime::enter_runtime<tokio::runtime::scheduler::multi_thread::{impl#0}::block_on::{closure_env#0}<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}>, core::result::Result<lancedb::table::MergeResult, lancedb::error::Error>> (handle=0x7fbe50e6aad8 <lancedb::connection::RUNTIME+48>, allow_block_in_place=true, f=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65
#60 0x00007fbe46d7b3a2 in tokio::runtime::scheduler::multi_thread::MultiThread::block_on<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}> (self=0x7fbe50e6aab0 <lancedb::connection::RUNTIME+8>, handle=0x7fbe50e6aad8 <lancedb::connection::RUNTIME+48>, future=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/mod.rs:86
#61 0x00007fbe46da5402 in tokio::runtime::runtime::Runtime::block_on_inner<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}> (self=0x7fbe50e6aaa8 <lancedb::connection::RUNTIME>, future=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:358
#62 0x00007fbe46da78ca in tokio::runtime::runtime::Runtime::block_on<lancedb::table::lancedb_table_merge_insert::{async_block_env#0}> (self=0x7fbe50e6aaa8 <lancedb::connection::RUNTIME>, future=...) at /home/adarsh9401/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330
#63 0x00007fbe46ce48f7 in lancedb::table::lancedb_table_merge_insert (table=0x560405faf6b0, data=0x5604036369b0, on_columns=0x5603fda714b0 <rgw::s3vector::key_columns>, num_columns=1, config=0x7fbd136f4e00, error_message=0x7fbd136f4dc0) at src/table.rs:141
#64 0x00005603fb65bfb3 in rgw::s3vector::put_vectors (configuration=..., dpp=<optimized out>, dpp@entry=0x560405f40420, y=..., y@entry=...) at /home/adarsh9401/ceph-ws/my-fork/ceph/src/rgw/rgw_s3vector.cc:1066
#65 0x00005603fb64a171 in (anonymous namespace)::RGWS3VectorPutVectors::execute (this=0x560405f40420, y=...) at /home/adarsh9401/ceph-ws/my-fork/ceph/src/rgw/rgw_rest_s3vector.cc:402
#66 0x00005603fb2cbcff in rgw_process_authenticated (handler=handler@entry=0x560405f02d40, op=@0x7fbd136f5540: 0x560405f40420, req=req@entry=0x7fbd136f65d0, s=0x7fbd136f5640, y=y@entry=..., driver=driver@entry=0x560403575fb0, skip_retarget=false) at /home/adarsh9401/ceph-ws/my-fork/ceph/src/rgw/rgw_process.cc:296
#67 0x00005603fb2cd9af in process_request (penv=..., req=req@entry=0x7fbd136f65d0, frontend_prefix="", client_io=client_io@entry=0x7fbd136f6660, yield=..., yield@entry=..., scheduler=scheduler@entry=0x560403c05928, user=0x7fbd136f67c0, latency=0x7fbd136f65a8, http_ret=0x7fbd136f65a4) at /home/adarsh9401/ceph-ws/my-fork/ceph/src/rgw/rgw_process.cc:450
#68 0x00005603fb21e5b2 in (anonymous namespace)::handle_connection<boost::asio::basic_stream_socket<boost::asio::ip::tcp> > (context=..., env=..., stream=..., timeout=..., header_limit=16384, buffer=..., is_ssl=false, pause_mutex=..., scheduler=0x560403c05928, uri_prefix="", ec=..., yield=...) at /home/adarsh9401/ceph-ws/my-fork/ceph/src/rgw/rgw_asio_frontend.cc:355
#69 0x00005603fb21f1ef in operator() (__closure=__closure@entry=0x7fbd136f7dc8, yield=...) at /home/adarsh9401/ceph-ws/my-fork/ceph/src/rgw/rgw_asio_frontend.cc:1236
#70 0x00005603fb21f374 in boost::asio::detail::spawn_entry_point<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0> >, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(boost::asio::yield_context)>, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(std::__exception_ptr::exception_ptr)> >::call(const boost::asio::basic_yield_context<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0> > > &, boost::asio::void_type<void>) (this=this@entry=0x7fbd136f7db0, yield=...) at /home/adarsh9401/ceph-ws/my-fork/ceph/build/boost/include/boost/asio/impl/spawn.hpp:748
#71 0x00005603fb21f59c in boost::asio::detail::spawn_entry_point<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0> >, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(boost::asio::yield_context)>, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(std::__exception_ptr::exception_ptr)> >::operator()(boost::asio::detail::spawned_thread_base *) (this=this@entry=0x7fbd136f7db0, spawned_thread=spawned_thread@entry=0x7fbd136f7d60) at /home/adarsh9401/ceph-ws/my-fork/ceph/build/boost/include/boost/asio/impl/spawn.hpp:737
#72 0x00005603fb21f6e1 in boost::asio::detail::spawned_fiber_thread::entry_point<boost::asio::detail::spawn_entry_point<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0> >, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(boost::asio::yield_context)>, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(std::__exception_ptr::exception_ptr)> > >::operator()(boost::asio::detail::spawned_fiber_thread::fiber_type &&) (this=this@entry=0x7fbd136f7f18, caller=...) at /home/adarsh9401/ceph-ws/my-fork/ceph/build/boost/include/boost/asio/impl/spawn.hpp:152
#73 0x00005603fb21f857 in std::__invoke_impl<boost::context::fiber, boost::asio::detail::spawned_fiber_thread::entry_point<boost::asio::detail::spawn_entry_point<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0> >, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(boost::asio::yield_context)>, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(std::__exception_ptr::exception_ptr)> > >&, boost::context::fiber> (__f=...) at /usr/include/c++/14/bits/invoke.h:61
#74 std::__invoke<boost::asio::detail::spawned_fiber_thread::entry_point<boost::asio::detail::spawn_entry_point<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0> >, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(boost::asio::yield_context)>, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(std::__exception_ptr::exception_ptr)> > >&, boost::context::fiber> (__fn=...) at /usr/include/c++/14/bits/invoke.h:97
#75 std::invoke<boost::asio::detail::spawned_fiber_thread::entry_point<boost::asio::detail::spawn_entry_point<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0> >, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(boost::asio::yield_context)>, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(std::__exception_ptr::exception_ptr)> > >&, boost::context::fiber> (__fn=...) at /usr/include/c++/14/functional:121
#76 boost::context::detail::fiber_record<boost::context::fiber, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>, boost::asio::detail::spawned_fiber_thread::entry_point<boost::asio::detail::spawn_entry_point<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0> >, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(boost::asio::yield_context)>, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(std::__exception_ptr::exception_ptr)> > > >::run(boost::context::detail::fcontext_t) (this=this@entry=0x7fbd136f7f00, fctx=<optimized out>) at /home/adarsh9401/ceph-ws/my-fork/ceph/build/boost/include/boost/context/fiber_fcontext.hpp:157
#77 0x00005603fb21f928 in boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>, boost::asio::detail::spawned_fiber_thread::entry_point<boost::asio::detail::spawn_entry_point<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0> >, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(boost::asio::yield_context)>, (anonymous namespace)::AsioFrontend::on_accept(Listener&, boost::asio::ip::tcp::socket)::<lambda(std::__exception_ptr::exception_ptr)> > > > >(boost::context::detail::transfer_t) (t=...) at /home/adarsh9401/ceph-ws/my-fork/ceph/build/boost/include/boost/context/fiber_fcontext.hpp:97
#78 0x00005603fcae25ff in make_fcontext ()
#79 0x0000000000000000 in ?? ()

root cause

for sql parsing in lancedb - when we use coroutines (boost asio, we set an upper bound for stack size). the existing bound is not enough (512 KB) -> we set it to a 1 MB instead.

the fix is to change line 70 on rgw_asio_frontend.cc to 1024*1024.

after fix, pytest summary on my feature

============================= test session starts ==============================
platform linux -- Python 3.9.23, pytest-8.4.1, pluggy-1.5.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/adarsh9401/ceph-ws/my-fork/ceph/src/test/rgw/s3vectors
configfile: pytest.ini
plugins: benchmark-4.0.0
collected 25 items

s3vector_test.py .s.......................                               [100%]

=============================== warnings summary ===============================
s3vector_test.py: 24 warnings
  /home/adarsh9401/.local/lib/python3.9/site-packages/boto3/compat.py:89: PythonDeprecationWarning: Boto3 will no longer support Python 3.9 starting April 29, 2026. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.10 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/
    warnings.warn(warning, PythonDeprecationWarning)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================= 24 passed, 1 skipped, 24 warnings in 13.71s ==================

@9401adarsh could you please run rgw under gdb, to get the backtrace of the crash?
to do that, you run vstart, then run pgrep -a radosgw to get the process id of the rgw you want to track.
then from inside the build directory, run:
gdb bin/radosgw -p <proc id>