cuML Python tests compute sanitizer output

full_run.txt

compute-sanitizer pytest python/cuml/test

========= COMPUTE-SANITIZER
============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rapids_pytest_benchmark: 0.0.13
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/whicks/proj_cuml/cuml/python, configfile: pytest.ini
plugins: hypothesis-6.1.1, cov-2.11.1, rapids-pytest-benchmark-0.0.13, benchmark-3.2.3, timeout-1.4.2, repeat-0.8.0, forked-1.3.0, asyncio-0.12.0, xdist-2.2.0
collected 27716 items

python/cuml/test/test_adapters.py ...                                    [  0%]
python/cuml/test/test_allocator.py ...                                   [  0%]
python/cuml/test/test_api.py ........................................... [  0%]
..................................x...........x.....x.........xxx.       [  0%]
python/cuml/test/test_arima.py ........s........s........s........s..... [  0%]
...s........s........s........s........s........s........s               [  0%]
python/cuml/test/test_array.py ...s..................................s.. [  0%]
..s....s....s....s..................................s....s....s....s.... [  1%]
s..................................s....s....s....s....s....s....s....s. [  1%]
...s....s....s....s....s....s....s....s....s....s....s....s....s....s... [  1%]
.s....s....s....s....s..................................s....s....s....s [  1%]
....s..................................s....s....s....s....s............ [  2%]
......................s....s....s....s....s....s....s....s....s....s.... [  2%]
s....s....s....s....s....s....s....s....s....s....s....s....s....s....s. [  2%]
...s.s.................................................................. [  2%]
........................................................................ [  3%]
........................................................................ [  3%]
........................................................................ [  3%]
........................................................................ [  4%]
........................................................................ [  4%]
........................................................................ [  4%]
........................................................................ [  4%]
........................................................................ [  5%]
........................................................................ [  5%]
........................................................................ [  5%]
........................................................................ [  5%]
...............................sssss.s.................................. [  6%]
........................................................................ [  6%]
........................................................................ [  6%]
........................................................................ [  6%]
........................................................................ [  7%]
........................................................................ [  7%]
........................................................................ [  7%]
........................................................................ [  7%]
........................................................................ [  8%]
........................................................................ [  8%]
........................................................................ [  8%]
........................................................................ [  8%]
........................................................................ [  9%]
........................................................................ [  9%]
........................................................................ [  9%]
........................................................................ [ 10%]
........................................................................ [ 10%]
........................................................................ [ 10%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 11%]
........................................................................ [ 11%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 12%]
........................................................................ [ 12%]
........................................................................ [ 12%]
........................................................................ [ 13%]
........................................................................ [ 13%]
........................................................................ [ 13%]
........................................................................ [ 13%]
........................................................................ [ 14%]
........................................................................ [ 14%]
........................................................................ [ 14%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 15%]
........................................................................ [ 15%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 16%]
........................................................................ [ 16%]
........................................................................ [ 17%]
........................................................................ [ 17%]
........................................................................ [ 17%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 18%]
........................................................................ [ 18%]
........................................................................ [ 18%]
........................................................................ [ 19%]
........................................................................ [ 19%]
........................................................................ [ 19%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 20%]
........................................................................ [ 20%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 21%]
........................................................................ [ 21%]
........................................................................ [ 21%]
........................................................................ [ 22%]
........................................................................ [ 22%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 23%]
........................................................................ [ 23%]
........................................................................ [ 23%]
........................................................................ [ 24%]
........................................................................ [ 24%]
........................................................................ [ 24%]
........................................................................ [ 24%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 26%]
........................................................................ [ 26%]
........................................................................ [ 26%]
........................................................................ [ 27%]
........................................................................ [ 27%]
........................................................................ [ 27%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 28%]
........................................................................ [ 28%]
........................................................................ [ 28%]
........................................................................ [ 29%]
........................................................................ [ 29%]
........................................................................ [ 29%]
........................................................................ [ 30%]
........................................................................ [ 30%]
........................................................................ [ 30%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 31%]
........................................................................ [ 31%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 32%]
........................................................................ [ 32%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 33%]
........................................................................ [ 33%]
........................................................................ [ 33%]
........................................................................ [ 34%]
........................................................................ [ 34%]
........................................................................ [ 34%]
........................................................................ [ 34%]
........................................................................ [ 35%]
........................................................................ [ 35%]
........................................................................ [ 35%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 36%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 37%]
........................................................................ [ 37%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................                                                 [ 38%]
python/cuml/test/test_array_sparse.py .................................. [ 38%]
...................                                                      [ 38%]
python/cuml/test/test_auto_arima.py .................................... [ 38%]
.............................................                            [ 38%]
python/cuml/test/test_base.py .......................................... [ 38%]
................s..............sss..................sss                  [ 38%]
python/cuml/test/test_batched_lbfgs.py .                                 [ 38%]
python/cuml/test/test_benchmark.py ..........................            [ 39%]
python/cuml/test/test_class_enumerator.py ....                           [ 39%]
python/cuml/test/test_coordinate_descent.py ........ssssssssssssssssssss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssss..ssssssssssssssss........ss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss..ssssssss [ 39%]
ssssssss........                                                         [ 39%]
python/cuml/test/test_cuml_descr_decor.py .s............................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 40%]
........................x...x.xxxx..x...x...x.xxxx..x...x...x.xxxx..x... [ 40%]
x...x.xxxx..x.                                                           [ 40%]
python/cuml/test/test_dataset_generator_types.py ....................    [ 40%]
python/cuml/test/test_dbscan.py ........ssssssssssssssssssssssssssssssss [ 40%]
ssssssssssssssssssssssssssssssss........ssssssssssssssssssssssssssssssss [ 41%]
ssssssssssssssssssssssssssssssss........ssssssssssssssssssssssssssssssss [ 41%]
ssssssssssssssssssssssssssssssss........ssssssssssssssssssssssssssssssss [ 41%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 41%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 42%]
ssssssssssssssssssssssssssssssss...ssssss...ssssss...ssssss...x......... [ 42%]
...........                                                              [ 42%]
python/cuml/test/test_fil.py .ssssssss.ssssssssssssssssssssssssss.ssssss [ 42%]
ss.ssssssssssssssssssssssssss.ssssssss.ssssssssssssssssssssssssss.ssssss [ 42%]
ssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssss [ 43%]
ss...................................................................... [ 43%]
.................                                                        [ 43%]
python/cuml/test/test_holtwinters.py .x.x.x.x........................... [ 43%]
...                                                                      [ 43%]
python/cuml/test/test_incremental_pca.py .ss............                 [ 43%]
python/cuml/test/test_input_utils.py ................................... [ 43%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 45%]
........................................................................ [ 45%]
........................................................................ [ 45%]
......................................................xxxxxxxxxxxx...... [ 45%]
....x..........x..........x..........x..........x..........x..........x. [ 46%]
.........x..........x..........xxxxxxxxxxxx..........x..........x....... [ 46%]
...x..........x..........x..........x..........x..........x..........x.. [ 46%]
........xxxxxxxxxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xx [ 46%]
xxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxxxxxxxxxxxxx... [ 47%]
...xxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xxx [ 47%]
xx......xxxxx......xxxxx......xxxxxxxxxxxxxxxx..........x..........x.... [ 47%]
......x..........x..........x..........x..........x..........x.......... [ 48%]
x....................................................................... [ 48%]
............................................................xxxxxxxxxxxx [ 48%]
..........x..........x..........x..........x..........x..........x...... [ 48%]
....x..........x..........x..........xxxxxxxxxxxx..........x..........x. [ 49%]
.........x..........x..........x..........x..........x..........x....... [ 49%]
...x..........xxxxxxxxxxxx......xxxxx......xxxxx......xxxxx......xxxxx.. [ 49%]
....xxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxxxxxxxxxx [ 49%]
xxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxx... [ 50%]
...xxxxx......xxxxx......xxxxx......xxxxxxxxxxxxxxxx..........x......... [ 50%]
.x..........x..........x..........x..........x..........x..........x.... [ 50%]
......x................................................................. [ 50%]
.................................xxxxxxxx............................... [ 51%]
.................................................xxxxxxxx............... [ 51%]
.................................................................xxxxxxx [ 51%]
x................................................xxxxxxxxxxxxxxxxxxxxxxx [ 51%]
xxxxxxxxxxxxxxxxx....................................................... [ 52%]
.........................xxxxxxxx....................................... [ 52%]
........................................................................ [ 52%]
.........................................................xxxxxxxx....... [ 52%]
........................................................................ [ 53%]
.xxxxxxxx............................................................... [ 53%]
.................xxxxxxxx............................................... [ 53%]
.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx............................... [ 53%]
.................................................xxxxxxxx............... [ 54%]
........................................................................ [ 54%]
..........................                                               [ 54%]
python/cuml/test/test_internals_api.py ...                               [ 54%]
python/cuml/test/test_kmeans.py x.........................sssss.....ssss [ 54%]
ssssss..........sssss.ss.ssssssssssssss                                  [ 54%]
python/cuml/test/test_kneighbors_classifier.py ......................... [ 54%]
........................................................................ [ 55%]
........................................................................ [ 55%]
........................................................................ [ 55%]
..............                                                           [ 55%]
python/cuml/test/test_kneighbors_regressor.py .......................... [ 55%]
...........................................................              [ 56%]
python/cuml/test/test_label_binarizer.py ........                        [ 56%]
python/cuml/test/test_label_encoder.py ............................      [ 56%]
python/cuml/test/test_lars.py ..ssss..ssssssssssssssss..ssss..ssssssssss [ 56%]
ssssss..ssss..ssssssssssssssss..ssss..ssssssssssssssss..ssss..ssssssssss [ 56%]
ssssss..ssss..ssssssssssssssss..ssssssssssssssss..ssssssssssssssss...... [ 56%]
......                                                                   [ 56%]
python/cuml/test/test_linear_model.py ....ssssssssssssssssssssssssssssss [ 57%]
ss.........ssssssssssssssssssssssssssssssss............................. [ 57%]
...........                                                              [ 57%]
python/cuml/test/test_logger.py ...                                      [ 57%]
python/cuml/test/test_make_arima.py .................................... [ 57%]
............                                                             [ 57%]
python/cuml/test/test_make_blobs.py .................................... [ 57%]
........................................................................ [ 57%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 59%]
........................................................................ [ 59%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 60%]
........................................................................ [ 60%]
........................................................................ [ 60%]
........................................................................ [ 61%]
........................................................................ [ 61%]
........................................................................ [ 61%]
....................................                                     [ 61%]
python/cuml/test/test_make_classification.py ........................... [ 61%]
........................................................................ [ 62%]
........................................................................ [ 62%]
........................................................................ [ 62%]
..............                                                           [ 62%]
python/cuml/test/test_make_regression.py ............................... [ 62%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 63%]
.........                                                                [ 63%]
python/cuml/test/test_mbsgd_classifier.py ..................ssssssssssss [ 63%]
ssssssssssssssssssssssss                                                 [ 63%]
python/cuml/test/test_mbsgd_regressor.py ..................sssssssssssss [ 63%]
sssssssssssssssssssssss                                                  [ 63%]
python/cuml/test/test_metrics.py ......ssssssssssssssssssssssssss...XXss [ 64%]
ssssssss................................................................ [ 64%]
..................................................................s.s.s. [ 64%]
s.s.s.s.s.s.s.s.s...............s.s.s.s.s.s.s.s....s..s..s..s..........s [ 64%]
..s..s..s............................................................... [ 65%]
..............................ssssssssssssss............................ [ 65%]
.ss.ss========= Invalid __global__ read of size 8 bytes
=========     at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
=========     by thread (0,28,0) in block (0,0,0)
=========     Address 0x7f18f78f3ae0 is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:cuLaunchKernel [0x7f1e83504dd8]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x7f1e1fa1262b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Invalid __global__ read of size 8 bytes
=========     at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
=========     by thread (0,29,0) in block (0,0,0)
=========     Address 0x7f18f78f3ae8 is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:cuLaunchKernel [0x7f1e83504dd8]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x7f1e1fa1262b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Invalid __global__ read of size 8 bytes
=========     at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
=========     by thread (0,30,0) in block (0,0,0)
=========     Address 0x7f18f78f3af0 is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:cuLaunchKernel [0x7f1e83504dd8]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x7f1e1fa1262b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Invalid __global__ read of size 8 bytes
=========     at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
=========     by thread (0,31,0) in block (0,0,0)
=========     Address 0x7f18f78f3af8 is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:cuLaunchKernel [0x7f1e83504dd8]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x7f1e1fa1262b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Invalid __global__ read of size 8 bytes
=========     at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
=========     by thread (0,24,0) in block (0,0,0)
=========     Address 0x7f18f78f3ac0 is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:cuLaunchKernel [0x7f1e83504dd8]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x7f1e1fa1262b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Invalid __global__ read of size 8 bytes
=========     at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
=========     by thread (0,25,0) in block (0,0,0)
=========     Address 0x7f18f78f3ac8 is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:cuLaunchKernel [0x7f1e83504dd8]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x7f1e1fa1262b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Invalid __global__ read of size 8 bytes
=========     at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
=========     by thread (0,26,0) in block (0,0,0)
=========     Address 0x7f18f78f3ad0 is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:cuLaunchKernel [0x7f1e83504dd8]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x7f1e1fa1262b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Invalid __global__ read of size 8 bytes
=========     at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
=========     by thread (0,27,0) in block (0,0,0)
=========     Address 0x7f18f78f3ad8 is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:cuLaunchKernel [0x7f1e83504dd8]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x7f1e1fa1262b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaMemcpyAsync.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaMemcpyAsync [0x7f1e1fa4fdc9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void raft::copy<double>(double*, raft::copy<double> const *, unsigned long, CUstream_st*) [0x7f1d1ecbe7e5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee761cb]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:void raft::copy<double>(double*, raft::copy<double> const *, unsigned long, CUstream_st*) [0x7f1d1ecbe81d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee761cb]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:rmm::device_buffer::deallocate(void) [0x7f1d1ee3bd18]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c48e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame:rmm::device_buffer::deallocate(void) [0x7f1d1ee3bd18]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8cd0f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
=========     Host Frame: [0x7f1cf4d019c3]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1dbb6df451]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/rmm/_lib/device_buffer.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d79136]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6bfc]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6bfc]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1de3d79136]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf4cff720]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1dbb6df451]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/rmm/_lib/device_buffer.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d79136]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6bfc]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6bfc]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1de3d79136]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf4cff5db]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1cf47bcaee]
=========                in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0c9d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaFree [0x7f1e1fa41150]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9fa4b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8f35b6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyEval_EvalCode [0x563deb921a3b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb921ad3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb9397a3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_RunMain [0x563deb93ee1d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_BytesMain [0x563deb93efd9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:__libc_start_main [0x7f1e84a070b3]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x563deb8b2b69]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= 
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x7f1e8365c453]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
=========     Host Frame: [0x7f1e1fc9c7d0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1fc9fa9f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7f62d]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1de3d7933e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x7f1e1feaeab6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7c6be3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8566b5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb90a03f]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f435]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb85f415]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c4ebf]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7c6a1b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x7f1d2e8e7794]
=========                in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
=========     Host Frame: [0x563deb7ee874]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8cab89]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_GC_New [0x563deb7ffc64]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyList_New [0x563deb7ffef4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b80]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb86aca0]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91bdce]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb91c0e4]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb8d3b99]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyAST_mod2obj [0x563deb91e532]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:Py_CompileStringObject [0x563deb93d9a5]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb93dc11]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb7f3a52]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87bbf7]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb87ba76]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame: [0x563deb83f6f9]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:PyObject_Call [0x563deb7f0aae]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
=========                in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========     Host Frame:_PyEval_EvalCodeWithName [0x563d

quick_run.txt