Skip to content

Instantly share code, notes, and snippets.

@adamruzicka

adamruzicka/scaling.md

Last active October 30, 2020 14:36
Show Gist options
  • Save adamruzicka/1991892ce22b18e030f9a4db95406319 to your computer and use it in GitHub Desktop.
Save adamruzicka/1991892ce22b18e030f9a4db95406319 to your computer and use it in GitHub Desktop.
Download ZIP
Dynflow workers scaling
Raw
scaling.md

Intro

Out of the box, foreman ships with orchestrator and a single worker If you have Katello, you will get an additional worker for processing of the host queue

root@modest-gator:~# ls -l /etc/foreman/dynflow/
total 1
-rw-r--r--. 1 root foreman 51 May 14 07:35 orchestrator.yml
-rw-r--r--. 1 root foreman 59 May 14 07:35 worker.yml

The orchestrator consumes items only from the dynflow_orchestrator queue and has only one thread. This is by design. Do not change this

root@modest-gator:~# cat /etc/foreman/dynflow/orchestrator.yml 
:concurrency: 1
:queues:
  - dynflow_orchestrator

The regular worker has 5 threads and consumes items from default and remote_execution queues. It is not recommended to change the configuration for this process either, because the changes would be overwritten by the installer, if run again.

root@modest-gator:~# cat /etc/foreman/dynflow/worker.yml 
:concurrency: 5
:queues:
  - default
  - remote_execution

Scaling up

Scaling up is pretty straightforward, especially if you want to only scale up what you have Here we use symbolic links to "share" the actual configuration among worker, worker-1 and worker-2

root@modest-gator:/etc/foreman/dynflow# ln -s worker.yml worker-1.yml
root@modest-gator:/etc/foreman/dynflow# ln -s worker.yml worker-2.yml

Check that the symbolic links are poining to the right files

root@modest-gator:/etc/foreman/dynflow# ls -l /etc/foreman/dynflow/
total 2
-rw-r--r--. 1 root foreman 51 May 14 07:35 orchestrator.yml
lrwxrwxrwx. 1 root root    10 May 14 11:24 worker-1.yml -> worker.yml
lrwxrwxrwx. 1 root root    10 May 14 11:24 worker-2.yml -> worker.yml
-rw-r--r--. 1 root foreman 59 May 14 07:35 worker.yml

Now start the newly configured services and check their status

root@modest-gator:/etc/foreman/dynflow# systemctl enable --now dynflow-sidekiq@worker-{1,2}
Created symlink /etc/systemd/system/multi-user.target.wants/[email protected] → /lib/systemd/system/[email protected].
Created symlink /etc/systemd/system/multi-user.target.wants/[email protected] → /lib/systemd/system/[email protected].

root@modest-gator:/etc/foreman/dynflow# systemctl status dynflow-sidekiq@worker-{1,2}
● [email protected] - Foreman jobs daemon - worker-1 on sidekiq
   Loaded: loaded (/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-14 11:24:19 UTC; 6s ago
     Docs: https://theforeman.org
 Main PID: 1422 (ruby2.5)
    Tasks: 5 (limit: 38437)
   Memory: 175.9M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/[email protected]
           └─1422 ruby2.5 /usr/share/foreman/vendor/ruby/2.5.0/bin/sidekiq -e production -r /usr/share/foreman/extras/dynflow-sidekiq.rb -C /etc/foreman/dynflow/worker-1.yml

May 14 11:24:19 modest-gator systemd[1]: Started Foreman jobs daemon - worker-1 on sidekiq.
May 14 11:24:21 modest-gator dynflow-sidekiq@worker-1[1422]: 2020-05-14T11:24:21.442Z 1422 TID-gpznscy7m INFO: GitLab reliable fetch activated!
May 14 11:24:21 modest-gator dynflow-sidekiq@worker-1[1422]: 2020-05-14T11:24:21.443Z 1422 TID-gpzo69up6 INFO: Booting Sidekiq 5.2.8 with redis options {:id=>"Sidekiq-server-PID-1422", :url=>"redis://localhost:6379/0"}

● [email protected] - Foreman jobs daemon - worker-2 on sidekiq
   Loaded: loaded (/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-14 11:24:19 UTC; 6s ago
     Docs: https://theforeman.org
 Main PID: 1423 (ruby2.5)
    Tasks: 5 (limit: 38437)
   Memory: 178.4M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/[email protected]
           └─1423 ruby2.5 /usr/share/foreman/vendor/ruby/2.5.0/bin/sidekiq -e production -r /usr/share/foreman/extras/dynflow-sidekiq.rb -C /etc/foreman/dynflow/worker-2.yml

May 14 11:24:19 modest-gator systemd[1]: Started Foreman jobs daemon - worker-2 on sidekiq.
May 14 11:24:21 modest-gator dynflow-sidekiq@worker-2[1423]: 2020-05-14T11:24:21.448Z 1423 TID-gpcxyxoi7 INFO: GitLab reliable fetch activated!
May 14 11:24:21 modest-gator dynflow-sidekiq@worker-2[1423]: 2020-05-14T11:24:21.449Z 1423 TID-gpcyculz7 INFO: Booting Sidekiq 5.2.8 with redis options {:id=>"Sidekiq-server-PID-1423", :url=>"redis://localhost:6379/0"}

Scaling down

Scaling down is just reverse process, let's say having worker, worker-1 and worker-2 was an overkill and we think we would be fine with just worker and worker-1.

root@modest-gator:/etc/foreman/dynflow# systemctl disable --now dynflow-sidekiq@worker-2
Removed /etc/systemd/system/multi-user.target.wants/[email protected].

root@modest-gator:/etc/foreman/dynflow# rm /etc/foreman/dynflow/worker-2.yml

Dedicating a worker to a queue

For this we need to create a new configuration for a worker (or just copy an existing one and change the relevant lines)

Create the configuration file, here we create a worker called rex having 5 threads and consuming items from the remote_execution queue

root@modest-gator:/etc/foreman/dynflow# cat <<EOF >/etc/foreman/dynflow/rex.yml
:concurrency: 5
:queues:
  - remote_execution
EOF

Now let's start the service and check its status

root@modest-gator:/etc/foreman/dynflow# systemctl start dynflow-sidekiq@rex

root@modest-gator:/etc/foreman/dynflow# systemctl status dynflow-sidekiq@rex
● [email protected] - Foreman jobs daemon - rex on sidekiq
   Loaded: loaded (/lib/systemd/system/[email protected]; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-14 11:32:01 UTC; 3s ago
     Docs: https://theforeman.org
 Main PID: 1785 (ruby2.5)
    Tasks: 5 (limit: 38437)
   Memory: 115.8M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/[email protected]
           └─1785 ruby2.5 /usr/share/foreman/vendor/ruby/2.5.0/bin/sidekiq -e production -r /usr/share/foreman/extras/dynflow-sidekiq.rb -C /etc/foreman/dynflow/rex.yml

May 14 11:32:01 modest-gator systemd[1]: Started Foreman jobs daemon - rex on sidekiq.
May 14 11:32:02 modest-gator dynflow-sidekiq@rex[1785]: 2020-05-14T11:32:02.789Z 1785 TID-gpaz8n321 INFO: GitLab reliable fetch activated!
May 14 11:32:02 modest-gator dynflow-sidekiq@rex[1785]: 2020-05-14T11:32:02.790Z 1785 TID-gpazi1pg1 INFO: Booting Sidekiq 5.2.8 with redis options {:id=>"Sidekiq-server-PID-1785", :url=>"redis://localhost:6379/0"}

Tuning number of threads per worker

The only thing that needs to be done is changing the number at the :concurrency: line and restarting the service. Either open it up in your favourite $EDITOR or do it with sed.

Here we can see the worker from previous example has 5 threads.

# 1785 is the process id taken from previous invocation of systemctl status
root@modest-gator:/etc/foreman/dynflow# ps -fp 1785
foreman     1785       1  0 11:32 ?        00:00:14 sidekiq 5.2.8  [0 of 5 busy]

Next we change it to 15

root@modest-gator:/etc/foreman/dynflow# sed -i 's/:concurrency: .*/:concurrency: 15/' /etc/foreman/dynflow/rex.yml

Restart the service and check its status

root@modest-gator:/etc/foreman/dynflow# systemctl restart dynflow-sidekiq@rex

root@modest-gator:/etc/foreman/dynflow# systemctl status dynflow-sidekiq@rex
● [email protected] - Foreman jobs daemon - rex on sidekiq
   Loaded: loaded (/lib/systemd/system/[email protected]; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-14 11:34:20 UTC; 2s ago
     Docs: https://theforeman.org
 Main PID: 1903 (ruby2.5)
    Tasks: 4 (limit: 38437)
   Memory: 94.4M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/[email protected]
           └─1903 ruby2.5 /usr/share/foreman/vendor/ruby/2.5.0/bin/sidekiq -e production -r /usr/share/foreman/extras/dynflow-sidekiq.rb -C /etc/foreman/dynflow/rex.yml

May 14 11:34:20 modest-gator systemd[1]: Started Foreman jobs daemon - rex on sidekiq.
May 14 11:34:21 modest-gator dynflow-sidekiq@rex[1903]: 2020-05-14T11:34:21.852Z 1903 TID-gn2jupy2z INFO: GitLab reliable fetch activated!
May 14 11:34:21 modest-gator dynflow-sidekiq@rex[1903]: 2020-05-14T11:34:21.852Z 1903 TID-gn2k8l9cr INFO: Booting Sidekiq 5.2.8 with redis options {:id=>"Sidekiq-server-PID-1903", :url=>"redis://localhost:6379/0"}

And finally we can check it has more threads.

root@modest-gator:/etc/foreman/dynflow# ps -fp 1903
UID          PID    PPID  C STIME TTY          TIME CMD
foreman     1903       1  0 11:34 ?        00:00:14 sidekiq 5.2.8  [0 of 15 busy]
@ares
Copy link

ares commented May 20, 2020

few more comments

Q: what's the difference between adding more threads to a worker versus running a new worker?
A: workers are independent process, not limited by Ruby thread model, where some code can't run in parallel, e.g. having 2 workers each with 5 thread will probably scale better than single worker with 10 threads, however it will also consume more resources (e.g. RAM)

Q: the example mentions "remote_execution" queue, what all queues are available?
A: this can change with every release, as of now, these are available: default, remote_execution, hosts

Q: will the installer override custom changes to these config files?
A: only if you change one of the config file that is present by default, e.g. if you modified number of threads for the default worker. If you scale up by adding more workers, installer does not touch that.

@adamruzicka
Copy link
Author

ad1) We will need to find a balance between number of threads and number of processes. A separate process can possibly incur higher management cost (at least it would open another connection to redis) so more processes isn't always better even if you have enough ram and spare cpus. I'd say 10 workers with 10 threads each would be a better choice than a single worker with 100 threads or 100 workers with 1 thread each.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment