Setup a blender rendernode on a headless server with crowdrender addon

blender_cr_headless.md

GUIDE: How To create a headless blender renderserver with CrowdRender addon

Requirements

• any headless Linux server
• Blender v2.79 or v2.80 - v2.93 (from package manager or blender.org)
• latest version of CrowdRender addon
• optional: GPU drivers for GPU-rendering

install and activate addon

for blender <= 2.79

$ blender -b --python-console
(InteractiveConsole)
>>> import bpy
>>> bpy.ops.wm.addon_install(overwrite=True, filepath="/path/to/folder/addon.zip")
>>> bpy.ops.wm.addon_enable(module='crowdrender')
>>> bpy.ops.wm.save_userpref() 
>>> exit()

for blender >= 2.80 (thanks to @Marcus-Zhu)

$ blender -b --python-console
(InteractiveConsole)
>>> import bpy
>>> bpy.ops.preferences.addon_install(overwrite=True,filepath='/path/to/addon.zip') 
>>> bpy.ops.preferences.addon_enable(module='crowdrender')
>>> bpy.ops.wm.save_userpref()
>>> exit()

start server

for Crowdrender <= 0.1.6

$ blender -noaudio -b --python ~/.config/blender/2.79/scripts/addons/crowdrender/src/bl_2_79/serv_int_start.py -- -t "server_int_proc"
Crowd Render Server Interface Process ready

for blender v2.80 you will need to change the path to serv_int_start.py accordingly

for Crowdrender >= 0.1.7 / Blender <= 2.79

$ blender -noaudio -b --python ~/.config/blender/2.79/scripts/addons/crowdrender/src/py_3_5_3/serv_int_start.py -- -t "server_int_proc"

for Crowdrender >= 0.1.7 / Blender >= 2.80

$ blender -noaudio -b --python ~/.config/blender/2.80/scripts/addons/crowdrender/src/py_3_7/serv_int_start.py -- -t "server_int_proc"

for Crowdrender >= 0.3.0 / Blender >= 2.80

$ blender -noaudio -b --python ~/.config/blender/2.80/scripts/addons/crowdrender/src/cr/serv_int_start.py -- -t "server_int_proc"

now it should show the IP address of your server and you should be able to connect to the server via any client running in the same network

additional CLI arguments

• -p false - if you're using a virtual machine from a cloud provider, you may not want to permanently store it in your crowdrender account as if you do, each time you use a vm from a cloud provider, you'll get a different vm and they'll start to pile up in your account.
• -ak - called the access key, if you setup multiple vm's on your own servers, best give them each a unique access key. This can be literally anything, but its important if you're using containerisation or vms on your own hardware since if you do, chances are the hardware id of those vm's containers could be the same, which totally prevents you from connecting to more than on of them at once, defeating the purpose of using our software.
• -ct - authorisation token of your Crowdrender cloud account. Using this allows the vm's / containers to post to your cloud account on our servers when they are alive. This means you'll actually be able to see which of them are alive and able to be used and which are offline (or just not posting cause they're not logged in).

you can run the process in the background by using screen and detatch it with Ctrl + a + d

$ screen blender -noaudio -b [...]

create systemd service for automatic startup and restart (thanks to @jan-thoma)

(Blender 2.80)

You create a unit /etc/systemd/system/crowdrender.service

[Unit]
Description=CrowdRender service
After=network.target

[Service]
Type=simple
ExecStart=/path/to/blender -noaudio -b --python /home/example/.config/blender/2.80/scripts/addons/crowdrender/src/py_3_7/serv_int_start.py -- -t "server_int_proc"
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

for Blender 2.79/2.81 etc. you need to change the path to the addon in ExecStart

To start the demon you run:

systemctl start crowdrender.service

To start at boot you enable it:

systemctl enable crowdrender.service

@valerit1 ok, so there's no documentation we've written for Crowdrender that suggests this should happen afaik. So I am puzzled why you think the server should print the ip address after it starts. If you've read this on the web somewhere, be great to know because its wrong, we've never had the intention of printing the IP address of the server once it starts.

However, if you really just need the ip address of the server, then you can easily discover what IP addresses (the server will bind to all ipV4 interfaces) your server has assigned to it using the appropriate commands for your OS type.

Well, actually I just need to run blender without a display. I do not need the IP address. So - what should I do after I see what is on my screenshot? When it just stays like this indefinitely, unless I terminate it with double Cntl-C. How do I run my Python script from this point? Thanks!

Well, actually I just need to run blender without a display

Yeah, running Blender without a display doesn't actually require Crowdrender. Why not simply do

./blender -b -noaudio --python-console

--python-console

Well, actually I just need to run blender without a display

Yeah, running Blender without a display doesn't actually require Crowdrender. Why not simply do

./blender -b -noaudio --python-console

Well, actually I need to run blender as a background process passing python script to it in the command line, without interactions,
possibly even from another process, like this:

blender -noaudio -b some_pytron_script.py

NOT as an interactive process --python-console option creates.

How can I do it without a display? I was hoping the add-on enables me to do so. Thanks!

@valerit1 you can just use this command:
blender -b -noaudio --python something.py

@valerit1 you can just use this command:
blender -b -noaudio --python something.py

With this:

blender -b -noaudio --python something.py

I get this:

Unable to open a display
Aborted (core dumped)

Are you using Eevee or Grease Pencil in your scene?

If yes, then it will not work on headless server.
If no, then it is hard to say, where the problem is, because I am using this exact command for all my scripting on a headless server without issues.

@valerit1 ok, so there's no documentation we've written for Crowdrender that suggests this should happen afaik. So I am puzzled why you think the server should print the ip address after it starts. If you've read this on the web somewhere, be great to know because its wrong, we've never had the intention of printing the IP address of the server once it starts.

However, if you really just need the ip address of the server, then you can easily discover what IP addresses (the server will bind to all ipV4 interfaces) your server has assigned to it using the appropriate commands for your OS type.

Are you using Eevee or Grease Pencil in your scene?

If yes, then it will not work on headless server.
If no, then it is hard to say, where the problem is, because I am using this exact command for all my scripting on a headless server without issues.

I am not a big expert in Blender use - how do I find out which one from my script? And how do I switch between these engines? Thanks a lot!

I am not a big expert in Blender use - how do I find out which one from my script? And how do I switch between these engines? Thanks a lot!

you can get and set it with bpy.context.scene.render.engine

EDIT: in scripting it might be better to call from the scene directly instead of using the context:
bpy.data.scenes['Scene'].render.engine

I am not a big expert in Blender use - how do I find out which one from my script? And how do I switch between these engines? Thanks a lot!

you can get and set it with bpy.context.scene.render.engine

EDIT: in scripting it might be better to call from the scene directly instead of using the context:
bpy.data.scenes['Scene'].render.engine

Excellent - thank you! I have Blender 2.92. Which engine should I set to be able to run it in the background without a display? Thanks!

You have to use the CYCLES engine, that one does work in headless mode (with some exceptions like Grease Pencil and some others I cannot recall from the top of my head right now.

That being said, I am kinda surprised that you try to run Blender using a script on a headless machine, which is more of an advanced task, while not knowing which render engines are available.....do you even know what the scripts you try to run are supposed to do?

I am saying this, because switching engines can result in broken or weird looking render results, because there are shaders and settings that only work for one and not the other.

You have to use the CYCLES engine, that one does work in headless mode (with some exceptions like Grease Pencil and some others I cannot recall from the top of my head right now.

That being said, I am kinda surprised that you try to run Blender using a script on a headless machine, which is more of an advanced task, while not knowing which render engines are available.....do you even know what the scripts you try to run are supposed to do?

I am saying this, because switching engines can result in broken or weird looking render results, because there are shaders and settings that only work for one and not the other.

Thank you - it works. We need some rendering for a scientific model, so I am no expert in Blender. Looks ok so far. Appreciate your help!

nice glad it works for you 👍

I am attempting to run this headless on ubuntu server 21.04 and I get the error

Blender 2.80 (sub 75) (hash f6cb5f54494e built 2019-07-29 17:17:04)
/run/user/1000/gvfs/ non-existent directory
Read prefs: /home/jacob/.config/blender/2.80/config/userpref.blend
found bundled python: /home/jacob/blender-2.80-linux-glibc217-x86_64/2.80/python
Crowd Render Server Interface Process ready
Traceback (most recent call last):
  File "/home/jacob/.config/blender/2.80/scripts/addons/crowdrender/src/py_3_7/serv_int_start.py", line 72, in <module>
    server_interface = crowdrender.src.py_3_7.server_interface
AttributeError: module 'crowdrender.src' has no attribute 'py_3_7'

Blender quit

I tried with blender 2.80, 2.82, and 2.83 and get the same error on all of them no matter what version of python I choose (in the code above I used py_3_7, but I also used 8 and 9). This is with crowdrender V0.3.1. The system python version is 3.9.5 although I don't think that matters.
Any suggestions?

Hi, can you share the command you used to start the server with? This helps a lot in attempting to reproduce the problem you've got.

@Jeducious Sorry about that, the command I used was:
./blender -noaudio -b --python ~/.config/blender/2.80/scripts/addons/crowdrender/src/py_3_7/serv_int_start.py -- -t "server_int_proc"
in the directory of blender

No problem man :) ok, so that command is probably outdated since we restructured the addon to include open source code. Full disclosure, we used to pack only compiled binaries, and someone pulled us up and told me this wasn't cool with the GPL license of Blender unless we gave the source code. I looked into this after being a bit skeptical at first, but turns out they were right.

This change to the packaging of the addon caused the command to change slightly to

./blender -noaudio -b --python ~/.config/blender/2.80/scripts/addons/crowdrender/src/cr/serv_int_start.py -- -t "server_int_proc"

also consider your circumstances for the other options
-p false - if you're using a virtual machine from a cloud provider, you may not want to permanently store it in your crowdrender account as if you do, each time you use a vm from a cloud provider, you'll get a different vm and they'll start to pile up in your account.
-ak - called the access key, if you setup multiple vm's on your own servers, best give them each a unique access key. This can be literally anything, but its important if you're using containerisation or vms on your own hardware since if you do, chances are the hardware id of those vm's containers could be the same, which totally prevents you from connecting to more than on of them at once, defeating the purpose of using our software.
-ct - authorisation token of your Crowdrender cloud account. Using this allows the vm's / containers to post to your cloud account on our servers when they are alive. This means you'll actually be able to see which of them are alive and able to be used and which are offline (or just not posting cause they're not logged in).

I think that's about everything, good luck!!

@Jeducious That works perfectly, thank you so much! You would not believe the amount of time I spent trying to figure this out on my own lol

You're welcome!

@Jacob439 thank you for bringing this up, I sadly forgot to update the instructions to reflect the new packaging. I updated it now

FYI @zocker-160, we've also changed the structure of the addon for V0.4.1, which will be the final one for this addon before CRESS is used to replace it.

The new command for V0.4.1 will be

./blender -noaudio -b --python ~/.config/blender/2.XX/scripts/addons/crowdrender/cr_source/core/serv_int_start.py -- -t "server_int_proc"

No problem man :) ok, so that command is probably outdated since we restructured the addon to include open source code. Full disclosure, we used to pack only compiled binaries, and someone pulled us up and told me this wasn't cool with the GPL license of Blender unless we gave the source code. I looked into this after being a bit skeptical at first, but turns out they were right.

This change to the packaging of the addon caused the command to change slightly to

./blender -noaudio -b --python ~/.config/blender/2.80/scripts/addons/crowdrender/src/cr/serv_int_start.py -- -t "server_int_proc"

also consider your circumstances for the other options -p false - if you're using a virtual machine from a cloud provider, you may not want to permanently store it in your crowdrender account as if you do, each time you use a vm from a cloud provider, you'll get a different vm and they'll start to pile up in your account. -ak - called the access key, if you setup multiple vm's on your own servers, best give them each a unique access key. This can be literally anything, but its important if you're using containerisation or vms on your own hardware since if you do, chances are the hardware id of those vm's containers could be the same, which totally prevents you from connecting to more than on of them at once, defeating the purpose of using our software. -ct - authorisation token of your Crowdrender cloud account. Using this allows the vm's / containers to post to your cloud account on our servers when they are alive. This means you'll actually be able to see which of them are alive and able to be used and which are offline (or just not posting cause they're not logged in).

I think that's about everything, good luck!!

I am attempting to add a second node but I am not having much luck. The two nodes are in vms running ubuntu on two separate servers. I can get them both to work individually, but I cannot get them to work at the same time because I am not using that access key. I added -ak brug to the end of one of the run commands (same as quoted) and -ak bruh to the end of the other. Blender starts and runs like usual on the servers, saying "crowd render server interface process ready". I now open up the blender client and try to connect to these servers with crowd render, but I get the error (on client) "; WARNING ; CRMain.connect_node_failed;couldn't connect to :r710 at endpoint 192.168.0.214". I see no place to enter an access key, am I missing something?

edit:
did some testing and found that if I add -ak "$uuidgen" I get the same result as I do when there is no -ak argument rather than the client not being able to connect to the nodes at all.

Ok, if you're going to use the -ak option, you should be using the -ct option as well and putting in your CR token, which you can get from your cloud account profile page here -> discovery.crowd-render.com/profile.

All you need to do is login to your profile and then you'll be able to copy your auth token and use it to fulfil the -ct option (ct = crowdrender token).

This automatically populates the vm's to your cloud account, so you should then see them appear in your list of nodes on your master machine (your master machine needs to also be logged in for this to work as the master will query your account to get the list of nodes that are associated with your account).

Doing it this way will automatically populate the access keys for the nodes on your master machine, meaning your master will be able to connect since it has the right key for each machine.

You may also want to consider if you want the VMs to persist in your cloud account. If you do then no further config is necessary, If you'd rather they didn't get saved to your account, then you can use -p false which stops a node from persisting in your account, it will be forgotten about. Persistent nodes will always be shown in your list, even if they are off-line.

Thanks for the help, this did answer my question but now I am just running into performance issues: When I add another node there is a very noticeable latency increase when navigating blender, and when I try to open the project I am working on (probably a worst case scenario: 400mb .obj file) blender crashes. I am watching my resources and I should have plenty of RAM to spare, although maybe there is an error with my system monitor. I guess I can use multiple nodes when working on smaller projects, but I guess for this project I am stuck with only one node. One last thing, is there documentation somewhere with this information? I feel like the questions I am asking should be somewhere, but I have done a lot of googling and found nothing. Thanks again for your help!

Hi Jacob, what are your hardware specs for the machine you are using as the client/master?

Specs are listed below, but before I go into those I do have a bit more information now. I am able to render easily with multiple nodes and I found that the automatic load balancer works pretty well (Nodes finish within about 5% of each other at the end of a render). This was with a much smaller .obj file than the excessively large 400mb obj file, but I still found it weird that crowdrender was what crashed blender (I was able to open the file after removing CR). Maybe it was a case of the straw that broke the camel's back, but it would be nice if there was a failsafe where CR just stops working instead of crashing blender altogether but I don't know the feasibility of that. And now here are the specs of the hardware that I love to play around with too much:

The master:
OS: Manjaro kde kernel: 5.14.1
cpu: i5-7500 @3.8ghz
ram: 16G ddr4

The client(server1):
OS host: Proxmox
OS VM: Ubuntu 21.04
CPU: 2x Xeon X5672 (VM has access to full cpu)
RAM(VM): 16G ddr3

Client(server2):
OS host: Proxmox
OS VM: Ubuntu 21.04
CPU : Xeon X5687 (VM only has access to 6 of the 8 threads)
RAM(VM): 4G

If you don't know much about proxmox, I tend to get less than 5% performance loss in a vm

Great, thanks for the data! If blender crashes due to CR, there might be some information either in the terminal you launch Blender from, or from logging files that CR writes to the machines disk. It would be great if you could take a look and maybe share that here.

Or, if you prefer not to share data from your machine with the whole internet, then you can also submit them in a support request form here

Will take a look through these and see if anything jumps out as the cause.

I know there was nothing unique from the terminal I launched blender (except for killed or something like that) or the client terminals, but next time this happens I will submit a support request with the data from the logs. Thanks for the help!