tensorflow error

checkpoint loaded
WARNING:tensorflow:From /opt/chatbot/main.py:24: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

[2021-04-27 07:27:48,434] DEBUG in main: Received request with text: Hey
2021-04-27 07:28:59.898336: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
2021-04-27 07:28:59.898614: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
172.17.0.1 - - [27/Apr/2021 07:28:59] "POST /generate HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Blas xGEMMBatched launch failed : a.shape=[16,10,64], b.shape=[16,10,64], m=10, n=10, k=64, batch_size=16
	 [[{{node sample_sequence/model/h0/attn/MatMul}}]]
	 [[strided_slice/_691]]
  (1) Internal: Blas xGEMMBatched launch failed : a.shape=[16,10,64], b.shape=[16,10,64], m=10, n=10, k=64, batch_size=16
	 [[{{node sample_sequence/model/h0/attn/MatMul}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/flask_cors/decorator.py", line 128, in wrapped_function
    resp = make_response(f(*args, **kwargs))
  File "/opt/chatbot/main.py", line 57, in get_gen
    response, _ = getResponse(data['text'], data['prefixHistory'])
  File "/opt/chatbot/main.py", line 32, in getResponse
    text = gpt2.generate(sess, length=200,temperature=0.6,prefix=prefixHistory, return_as_list=True)[0]
  File "/opt/chatbot/.venv/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 471, in generate
    context: batch_size * [context_tokens]
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Blas xGEMMBatched launch failed : a.shape=[16,10,64], b.shape=[16,10,64], m=10, n=10, k=64, batch_size=16
	 [[node sample_sequence/model/h0/attn/MatMul (defined at usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
	 [[strided_slice/_691]]
  (1) Internal: Blas xGEMMBatched launch failed : a.shape=[16,10,64], b.shape=[16,10,64], m=10, n=10, k=64, batch_size=16
	 [[node sample_sequence/model/h0/attn/MatMul (defined at usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'sample_sequence/model/h0/attn/MatMul':
  File "usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "usr/lib/python3.6/socketserver.py", line 654, in process_request_thread
    self.finish_request(request, client_address)
  File "usr/lib/python3.6/socketserver.py", line 364, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "usr/lib/python3.6/socketserver.py", line 724, in __init__
    self.handle()
  File "usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 329, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "usr/lib/python3.6/http/server.py", line 418, in handle
    self.handle_one_request()
  File "usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 364, in handle_one_request
    return self.run_wsgi()
  File "usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 306, in run_wsgi
    execute(self.server.app)
  File "usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 296, in execute
    for data in application_iter:
  File "usr/local/lib/python3.6/dist-packages/werkzeug/debug/__init__.py", line 330, in debug_application
    app_iter = self.app(environ, start_response)
  File "opt/chatbot/.venv/lib/python3.6/site-packages/flask/cli.py", line 337, in __call__
    return self._app(environ, start_response)
  File "opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "opt/chatbot/.venv/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "opt/chatbot/.venv/lib/python3.6/site-packages/flask_cors/decorator.py", line 128, in wrapped_function
    resp = make_response(f(*args, **kwargs))
  File "opt/chatbot/main.py", line 57, in get_gen
    response, _ = getResponse(data['text'], data['prefixHistory'])
  File "opt/chatbot/main.py", line 32, in getResponse
    text = gpt2.generate(sess, length=200,temperature=0.6,prefix=prefixHistory, return_as_list=True)[0]
  File "opt/chatbot/.venv/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 459, in generate
    temperature=temperature, top_k=top_k, top_p=top_p
  File "opt/chatbot/.venv/lib/python3.6/site-packages/gpt_2_simple/src/sample.py", line 67, in sample_sequence
    context_output = step(hparams, context[:, :-1])
  File "opt/chatbot/.venv/lib/python3.6/site-packages/gpt_2_simple/src/sample.py", line 52, in step
    past=past, reuse=tf.compat.v1.AUTO_REUSE)
  File "opt/chatbot/.venv/lib/python3.6/site-packages/gpt_2_simple/src/model.py", line 197, in model
    h, present = block(h, 'h%d' % layer, past=past, hparams=hparams)
  File "opt/chatbot/.venv/lib/python3.6/site-packages/gpt_2_simple/src/model.py", line 156, in block
    a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
  File "opt/chatbot/.venv/lib/python3.6/site-packages/gpt_2_simple/src/model.py", line 139, in attn
    a = multihead_attn(q, k, v)
  File "opt/chatbot/.venv/lib/python3.6/site-packages/gpt_2_simple/src/model.py", line 123, in multihead_attn
    w = tf.matmul(q, k, transpose_b=True)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_ops.py", line 2716, in matmul
    return batch_mat_mul_fn(a, b, adj_x=adjoint_a, adj_y=adjoint_b, name=name)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 1712, in batch_mat_mul_v2
    "BatchMatMulV2", x=x, y=y, adj_x=adj_x, adj_y=adj_y, name=name)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()