mwtian

Debugging Sui node memory growth

When there is memory usage growth in the sui-node process (validator or fullnode), usually this is hard to reproduce and not unobserved on many other instances (otherwise the issue would have been fixed before the release). More infromation are usually needed to help us debug the issue.

Metrics

First, a few metrics can be checked to see if a particular stream is very large and keeps growing.

rocksdb_estimate_table_readers_mem
rocksdb_block_cache_usage

	import datetime
	import random
	import time
	import ray
	from ray import ray_constants
	from ray import cloudpickle
	from collections import defaultdict

	ray.init(address="auto")

	import datetime
	import time
	import ray
	from ray import ray_constants
	from ray import cloudpickle


	OBJ_REFS = 500 # number of put_object tasks to run, and number of top-level object refs input to ray.get()
	EMBEDDED_OBJ_REFS = 2400 # number of embedded object refs per input object
	REF_HIDING = False # whether to hide embedded object refs via cloudpickle.dumps(ray.put(input_object)))