tvmcrpc.md

How to Setup TVMC over RPC

The official guide talks a little about how to setup TVMC (a simple interface) over RPC, there are some pitfalls if you are not careful.

Install TVM

You need to install the full tvm on the host machine by make -j8. On the target machine, you need to install the runtime by make runtime.

Then, setup the python environment properly.

Setup the servers

Your host machine needs a RPC tracker, boot it up by

python -m tvm.exec.rpc_tracker --host=0.0.0.0 --port=9190

Notice, you cannot use port forwarding if your machine is in a virtual environment, which won't get detected by TVM (so far). For example, if your host machine is a WSL on Windows, try to add a virtual network card by Hyper-V and connect the virtual card to your WSL system, which will create a real IP in the local network to get discovered by other machines. Get the host machine IP address: [IP].

Then, setup the RPC server on the target machine by

python -m tvm.exec.rpc_server --tracker=[IP]:9190 --key=rasp4b --port=9090

Here the key is used to identify the target.

You could test the connection on the host machine by

python -m tvm.exec.query_rpc_tracker --host=0.0.0.0 --port=9190

If your target machine is on the list, then you are ready to go.

Compile, Tune and Run

Here is the code:

# RPC key
rpc_key = "rasp4b"
# hostname of the RPC tracker
hostname = "0.0.0.0"
# port of the RPC tracker
port = 9190
# compilation target
target = "llvm -device=arm_cpu -model=bcm2835 -mtriple=aarch64-linux-gnu -mattr=+neon"

# load ONNX model
model = tvmc.load("*.onnx")

# Tune by AutoTVM
# tvmc.tune(model, target=target, rpc_key=rpc_key, hostname=hostname, port=port)
# Tune by Auto Scheduler
# tvmc.tune(model, target=target, enable_autoscheduler=True, rpc_key=rpc_key, hostname=hostname, port=port)

# Compile the package
#   The argument `output_format` is crucial, which is even not on the official documentation!
#   Otherwise, the next step will get failed.
package = tvmc.compile(model, target=target, package_path=path, output_format="tar")

# Run the model remotely.
result = tvmc.run(package, device="cpu", inputs=testdata, repeat=10000, benchmark=True, number=1, rpc_key=rpc_key, hostname=hostname, port=port)

print(result)

Some errors like file format is not correct is because you didn't set output_format="tar" in the compilation step.