Skip to content

Instantly share code, notes, and snippets.

@mccun934

mccun934/scaling.md

Forked from adamruzicka/scaling.md
Created October 30, 2020 14:36
Show Gist options
  • Save mccun934/8e476f851bdfa21465946a316b6f96e5 to your computer and use it in GitHub Desktop.
Save mccun934/8e476f851bdfa21465946a316b6f96e5 to your computer and use it in GitHub Desktop.
Download ZIP
Dynflow workers scaling
Raw
scaling.md

Intro

Out of the box, foreman ships with orchestrator and a single worker If you have Katello, you will get an additional worker for processing of the host queue

root@modest-gator:~# ls -l /etc/foreman/dynflow/
total 1
-rw-r--r--. 1 root foreman 51 May 14 07:35 orchestrator.yml
-rw-r--r--. 1 root foreman 59 May 14 07:35 worker.yml

The orchestrator consumes items only from the dynflow_orchestrator queue and has only one thread. This is by design. Do not change this

root@modest-gator:~# cat /etc/foreman/dynflow/orchestrator.yml 
:concurrency: 1
:queues:
  - dynflow_orchestrator

The regular worker has 5 threads and consumes items from default and remote_execution queues. It is not recommended to change the configuration for this process either, because the changes would be overwritten by the installer, if run again.

root@modest-gator:~# cat /etc/foreman/dynflow/worker.yml 
:concurrency: 5
:queues:
  - default
  - remote_execution

Scaling up

Scaling up is pretty straightforward, especially if you want to only scale up what you have Here we use symbolic links to "share" the actual configuration among worker, worker-1 and worker-2

root@modest-gator:/etc/foreman/dynflow# ln -s worker.yml worker-1.yml
root@modest-gator:/etc/foreman/dynflow# ln -s worker.yml worker-2.yml

Check that the symbolic links are poining to the right files

root@modest-gator:/etc/foreman/dynflow# ls -l /etc/foreman/dynflow/
total 2
-rw-r--r--. 1 root foreman 51 May 14 07:35 orchestrator.yml
lrwxrwxrwx. 1 root root    10 May 14 11:24 worker-1.yml -> worker.yml
lrwxrwxrwx. 1 root root    10 May 14 11:24 worker-2.yml -> worker.yml
-rw-r--r--. 1 root foreman 59 May 14 07:35 worker.yml

Now start the newly configured services and check their status

root@modest-gator:/etc/foreman/dynflow# systemctl enable --now dynflow-sidekiq@worker-{1,2}
Created symlink /etc/systemd/system/multi-user.target.wants/[email protected] → /lib/systemd/system/[email protected].
Created symlink /etc/systemd/system/multi-user.target.wants/[email protected] → /lib/systemd/system/[email protected].

root@modest-gator:/etc/foreman/dynflow# systemctl status dynflow-sidekiq@worker-{1,2}
● [email protected] - Foreman jobs daemon - worker-1 on sidekiq
   Loaded: loaded (/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-14 11:24:19 UTC; 6s ago
     Docs: https://theforeman.org
 Main PID: 1422 (ruby2.5)
    Tasks: 5 (limit: 38437)
   Memory: 175.9M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/[email protected]
           └─1422 ruby2.5 /usr/share/foreman/vendor/ruby/2.5.0/bin/sidekiq -e production -r /usr/share/foreman/extras/dynflow-sidekiq.rb -C /etc/foreman/dynflow/worker-1.yml

May 14 11:24:19 modest-gator systemd[1]: Started Foreman jobs daemon - worker-1 on sidekiq.
May 14 11:24:21 modest-gator dynflow-sidekiq@worker-1[1422]: 2020-05-14T11:24:21.442Z 1422 TID-gpznscy7m INFO: GitLab reliable fetch activated!
May 14 11:24:21 modest-gator dynflow-sidekiq@worker-1[1422]: 2020-05-14T11:24:21.443Z 1422 TID-gpzo69up6 INFO: Booting Sidekiq 5.2.8 with redis options {:id=>"Sidekiq-server-PID-1422", :url=>"redis://localhost:6379/0"}

● [email protected] - Foreman jobs daemon - worker-2 on sidekiq
   Loaded: loaded (/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-14 11:24:19 UTC; 6s ago
     Docs: https://theforeman.org
 Main PID: 1423 (ruby2.5)
    Tasks: 5 (limit: 38437)
   Memory: 178.4M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/[email protected]
           └─1423 ruby2.5 /usr/share/foreman/vendor/ruby/2.5.0/bin/sidekiq -e production -r /usr/share/foreman/extras/dynflow-sidekiq.rb -C /etc/foreman/dynflow/worker-2.yml

May 14 11:24:19 modest-gator systemd[1]: Started Foreman jobs daemon - worker-2 on sidekiq.
May 14 11:24:21 modest-gator dynflow-sidekiq@worker-2[1423]: 2020-05-14T11:24:21.448Z 1423 TID-gpcxyxoi7 INFO: GitLab reliable fetch activated!
May 14 11:24:21 modest-gator dynflow-sidekiq@worker-2[1423]: 2020-05-14T11:24:21.449Z 1423 TID-gpcyculz7 INFO: Booting Sidekiq 5.2.8 with redis options {:id=>"Sidekiq-server-PID-1423", :url=>"redis://localhost:6379/0"}

Scaling down

Scaling down is just reverse process, let's say having worker, worker-1 and worker-2 was an overkill and we think we would be fine with just worker and worker-1.

root@modest-gator:/etc/foreman/dynflow# systemctl disable --now dynflow-sidekiq@worker-2
Removed /etc/systemd/system/multi-user.target.wants/[email protected].

root@modest-gator:/etc/foreman/dynflow# rm /etc/foreman/dynflow/worker-2.yml

Dedicating a worker to a queue

For this we need to create a new configuration for a worker (or just copy an existing one and change the relevant lines)

Create the configuration file, here we create a worker called rex having 5 threads and consuming items from the remote_execution queue

root@modest-gator:/etc/foreman/dynflow# cat <<EOF >/etc/foreman/dynflow/rex.yml
:concurrency: 5
:queues:
  - remote_execution
EOF

Now let's start the service and check its status

root@modest-gator:/etc/foreman/dynflow# systemctl start dynflow-sidekiq@rex

root@modest-gator:/etc/foreman/dynflow# systemctl status dynflow-sidekiq@rex
● [email protected] - Foreman jobs daemon - rex on sidekiq
   Loaded: loaded (/lib/systemd/system/[email protected]; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-14 11:32:01 UTC; 3s ago
     Docs: https://theforeman.org
 Main PID: 1785 (ruby2.5)
    Tasks: 5 (limit: 38437)
   Memory: 115.8M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/[email protected]
           └─1785 ruby2.5 /usr/share/foreman/vendor/ruby/2.5.0/bin/sidekiq -e production -r /usr/share/foreman/extras/dynflow-sidekiq.rb -C /etc/foreman/dynflow/rex.yml

May 14 11:32:01 modest-gator systemd[1]: Started Foreman jobs daemon - rex on sidekiq.
May 14 11:32:02 modest-gator dynflow-sidekiq@rex[1785]: 2020-05-14T11:32:02.789Z 1785 TID-gpaz8n321 INFO: GitLab reliable fetch activated!
May 14 11:32:02 modest-gator dynflow-sidekiq@rex[1785]: 2020-05-14T11:32:02.790Z 1785 TID-gpazi1pg1 INFO: Booting Sidekiq 5.2.8 with redis options {:id=>"Sidekiq-server-PID-1785", :url=>"redis://localhost:6379/0"}

Tuning number of threads per worker

The only thing that needs to be done is changing the number at the :concurrency: line and restarting the service. Either open it up in your favourite $EDITOR or do it with sed.

Here we can see the worker from previous example has 5 threads.

# 1785 is the process id taken from previous invocation of systemctl status
root@modest-gator:/etc/foreman/dynflow# ps -fp 1785
foreman     1785       1  0 11:32 ?        00:00:14 sidekiq 5.2.8  [0 of 5 busy]

Next we change it to 15

root@modest-gator:/etc/foreman/dynflow# sed -i 's/:concurrency: .*/:concurrency: 15/' /etc/foreman/dynflow/rex.yml

Restart the service and check its status

root@modest-gator:/etc/foreman/dynflow# systemctl restart dynflow-sidekiq@rex

root@modest-gator:/etc/foreman/dynflow# systemctl status dynflow-sidekiq@rex
● [email protected] - Foreman jobs daemon - rex on sidekiq
   Loaded: loaded (/lib/systemd/system/[email protected]; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-14 11:34:20 UTC; 2s ago
     Docs: https://theforeman.org
 Main PID: 1903 (ruby2.5)
    Tasks: 4 (limit: 38437)
   Memory: 94.4M
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/[email protected]
           └─1903 ruby2.5 /usr/share/foreman/vendor/ruby/2.5.0/bin/sidekiq -e production -r /usr/share/foreman/extras/dynflow-sidekiq.rb -C /etc/foreman/dynflow/rex.yml

May 14 11:34:20 modest-gator systemd[1]: Started Foreman jobs daemon - rex on sidekiq.
May 14 11:34:21 modest-gator dynflow-sidekiq@rex[1903]: 2020-05-14T11:34:21.852Z 1903 TID-gn2jupy2z INFO: GitLab reliable fetch activated!
May 14 11:34:21 modest-gator dynflow-sidekiq@rex[1903]: 2020-05-14T11:34:21.852Z 1903 TID-gn2k8l9cr INFO: Booting Sidekiq 5.2.8 with redis options {:id=>"Sidekiq-server-PID-1903", :url=>"redis://localhost:6379/0"}

And finally we can check it has more threads.

root@modest-gator:/etc/foreman/dynflow# ps -fp 1903
UID          PID    PPID  C STIME TTY          TIME CMD
foreman     1903       1  0 11:34 ?        00:00:14 sidekiq 5.2.8  [0 of 15 busy]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment