Skip to content

Instantly share code, notes, and snippets.

@holmberd
Last active September 25, 2024 05:44
Show Gist options
  • Save holmberd/44fa5c2555139a1a46e01434d3aaa512 to your computer and use it in GitHub Desktop.
Save holmberd/44fa5c2555139a1a46e01434d3aaa512 to your computer and use it in GitHub Desktop.
Adjusting child processes for PHP-FPM (Nginx)

Adjusting child processes for PHP-FPM (Nginx)

When setting these options consider the following:

  • How long is your average request?
  • What is the maximum number of simultaneous visitors the site(s) get?
  • How much memory on average does each child process consume?

Determine if the max_children limit has been reached.

  • sudo grep max_children /var/log/php?.?-fpm.log.1 /var/log/php?.?-fpm.log

Determine system RAM and average pool size memory.

  • free -h
  • All fpm processes: ps -ylC php-fpm7.0 --sort:rss
  • Average memory: ps --no-headers -o "rss,cmd" -C php-fpm7.0 | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }'
  • All fpm processes memory: ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | grep php-fpm

Calculate max_children

Based on RAM

  • pm.max_children = Total RAM dedicated to the web server / Max child process size

  • System RAM: 2GB

  • Average Pool size: 85Mb

  • pm.max_children = 1500MB / 85MB = 17

Based on average script execution time

  • max_children = (average PHP script execution time) * (PHP requests per second)
  • visitors = max_children * (seconds between page views) / (avg. execution time)

Configure

sudo vim /etc/php/7.0/fpm/pool.d/www.conf

pm.max_children = 17
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
pm.max_request = 1000
; Choose how the process manager will control the number of child processes.
; Possible Values:
;   static  - a fixed number (pm.max_children) of child processes;
;   dynamic - the number of child processes are set dynamically based on the
;             following directives:
;             pm.max_children      - the maximum number of children that can
;                                    be alive at the same time.
;
;             pm.start_servers     - the number of children created on startup.
;                                    this value must not be less than min_spare_servers 
;                                    and not greater than max_spare_servers.
;
;             pm.min_spare_servers - the minimum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is less than this
;                                    number then some children will be created.
;
;             pm.max_spare_servers - the maximum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is greater than this
;                                    number then some children will be killed.
; Note: This value is mandatory.
@Yv-o-rL
Copy link

Yv-o-rL commented Jan 25, 2019

Hi!

I'm trying to get more information on how PHP children work. I can't find any documentation, blog or anything which would give general answers. I was wondering if you could help me out.

Does one PHP worker equal to one CPU core? I mean having 64 workers on a 32 core VM is unnecessary since they can only use a maximum of 32 cores?

What does a request include (pm.max_requests)? An Nginx request like "index.php"?

When I check PHP status and see "active processes", what's that? Is that one request? One PHP function? A call to a MySQL DB? A PHP worker?

How does PHP handle the Nginx requests coming in regarding workers? I mean on a high level. Is there a queue? Does it depend on the functions, extensions in the PHP file? For example, having 2 PHP children and opening one "index.php" every second when the processing time is .9 seconds. What happens?

When does FPM log lines like the one below? When all the workers have stuff to do or when there's a request which can't be fulfilled since all workers are busy? As you see my questions above, I don't know how the requests(one dynamic file)->manual functions (calculate_cart_items_shipping_cost())->PHP functions(round(3.14159)) work in this regard.

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it.

Thank you!

@holmberd
Copy link
Author

holmberd commented Apr 12, 2019

@Yv-o-rL one php-fpm child process handles one request.

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it.

The warning means that you are hitting the upper limit of how many requests you can handle with your current configuration. The request will be queued until a worker becomes available to handle the request. Reason for this is that your pm.max_children is set too low, i.e. you have more requests than the number of workers can handle.

@endrureza
Copy link

is it not better to set pm as static for high traffic web ? i mean, use dynamic will be getting confused to the setting. Cause of the adjustment of pm.max_children , pm.start_servers, pm.min_spare_servers and pm.max_spare_servers.

@holmberd
Copy link
Author

@endrureza neither is better than the other. Each one can have a better fit for a particular use-case, and it's up to the developer of that system to consider the trade-offs.

Static

  • Each process sits in memory at all times, how much memory is consumed will vary for each server.
  • CPU experiences less load spikes when traffic spikes; since there is no overhead in spawning new processes.

Dynamic

  • Spawns new processes when traffic load increases.
  • Spawning processes will put some load on the CPU.
  • Frees up memory that can be used by other services when traffic is low.

@synetalsolutions
Copy link

i have a dedicated server which have 128 gb of ram and xeon processor with 1gbps bandwidth. I have installed centminmod on my server but the issue I am facing is that when my traffic raise to 3000 realtime the server stop responding.

What I am looking for what will be the configuration of php-fpm so it can handle 10000 realtime users
My configuration is this

`global]
; Log level
; Possible Values: alert, error, warning, notice, debug
; Default Value: notice
log_level = warning
pid = /var/run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/www-error.log
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s
include=/usr/local/nginx/conf/phpfpmd/*.conf

[www]
user = nginx
group = nginx

listen = 127.0.0.1:9000
listen.allowed_clients = 127.0.0.1
listen.backlog = 65535

;listen = /tmp/php5-fpm.sock
listen.owner = nginx
listen.group = nginx
listen.mode = 0660

pm = static
pm.max_children = 3072
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 100
pm.min_spare_servers = 10000
pm.max_spare_servers = 15000
pm.max_requests = 10000

; PHP 5.3.9 setting
; The number of seconds after which an idle process will be killed.
; Note: Used only when pm is set to 'ondemand'
; Default Value: 10s
pm.process_idle_timeout = 10s;

rlimit_files = 65536
rlimit_core = 0

; The timeout for serving a single request after which the worker process will
; be killed. This option should be used when the 'max_execution_time' ini option
; does not stop script execution for some reason. A value of '0' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_terminate_timeout = 0
; Default Value: 0
;request_slowlog_timeout = 0
slowlog = /var/log/php-fpm/www-slow.log

pm.status_path = /phpstatus
ping.path = /phpping
ping.response = pong

; Limits the extensions of the main script FPM will allow to parse. This can
; prevent configuration mistakes on the web server side. You should only limit
; FPM to .php extensions to prevent malicious users to use other extensions to
; exectute php code.
; Note: set an empty value to allow all extensions.
; Default Value: .php
security.limit_extensions = .php

; catch_workers_output = yes
php_admin_value[error_log] = /var/log/php-fpm/www-php.error.log
php_admin_value[disable_functions] = shell_exec.
`

@endrureza
Copy link

i have a dedicated server which have 128 gb of ram and xeon processor with 1gbps bandwidth. I have installed centminmod on my server but the issue I am facing is that when my traffic raise to 3000 realtime the server stop responding.

What I am looking for what will be the configuration of php-fpm so it can handle 10000 realtime users
My configuration is this

`global]
; Log level
; Possible Values: alert, error, warning, notice, debug
; Default Value: notice
log_level = warning
pid = /var/run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/www-error.log
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s
include=/usr/local/nginx/conf/phpfpmd/*.conf

[www]
user = nginx
group = nginx

listen = 127.0.0.1:9000
listen.allowed_clients = 127.0.0.1
listen.backlog = 65535

;listen = /tmp/php5-fpm.sock
listen.owner = nginx
listen.group = nginx
listen.mode = 0660

pm = static
pm.max_children = 3072
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 100
pm.min_spare_servers = 10000
pm.max_spare_servers = 15000
pm.max_requests = 10000

; PHP 5.3.9 setting
; The number of seconds after which an idle process will be killed.
; Note: Used only when pm is set to 'ondemand'
; Default Value: 10s
pm.process_idle_timeout = 10s;

rlimit_files = 65536
rlimit_core = 0

; The timeout for serving a single request after which the worker process will
; be killed. This option should be used when the 'max_execution_time' ini option
; does not stop script execution for some reason. A value of '0' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_terminate_timeout = 0
; Default Value: 0
;request_slowlog_timeout = 0
slowlog = /var/log/php-fpm/www-slow.log

pm.status_path = /phpstatus
ping.path = /phpping
ping.response = pong

; Limits the extensions of the main script FPM will allow to parse. This can
; prevent configuration mistakes on the web server side. You should only limit
; FPM to .php extensions to prevent malicious users to use other extensions to
; exectute php code.
; Note: set an empty value to allow all extensions.
; Default Value: .php
security.limit_extensions = .php

; catch_workers_output = yes
php_admin_value[error_log] = /var/log/php-fpm/www-php.error.log
php_admin_value[disable_functions] = shell_exec.
`

your ram is quite high, what about your cpu ? as for my problem, it has nothing to do with ram, it's the cpu. and i kinda solve this one not only from setting the php setting, but also using horizontal scaling (ex: load balancer)

@synetalsolutions
Copy link

I am looking for best server configuration which can handle around 10000 real time traffic or any recommendation.

@endrureza
Copy link

I am looking for best server configuration which can handle around 10000 real time traffic or any recommendation.

yeah, as for me i wouldn't tweak this setting too far in order to reach high traffic. i would consider using load balancer

@bonyod
Copy link

bonyod commented Feb 14, 2020

Hi, I have 8GB RAM and 2vCPU (m5.large AWS) any idea to maximize server performance? Avg pool size is : 32MB
I have use these settings:
pm.max_children = 192
pm.start_servers = 100
pm.min_spare_servers = 100
pm.max_spare_servers = 175
pm.max_request = 2000
is this ok?

@abdennour
Copy link

Guys! you must make everything auto/dynamic if you are running on top of Kubernetes cluster. When CPU or Memory reaches the thresholds, k8s HPA will do the work.
So now the question: how to make everything dynamic and open ?
Even nginx, I want to configure it like this :

worker_processes auto;
events {
  worker_connections  ??;
  multi_accept on;
  use epoll;
}

@Herz3h
Copy link

Herz3h commented Oct 20, 2020

Any reason you put php-fpm7.0 in process name instead of php-fpm for the Determine system RAM and average pool size memory.
part ?

@mubasshir
Copy link

@bonyod whats your final config?

@bonyod
Copy link

bonyod commented Nov 9, 2020

@bonyod whats your final config?

pm.max_children = 50
pm.start_servers = 12
pm.min_spare_servers = 5
pm.max_spare_servers = 25
pm.max_request = 0

But still not at best performance.

@pultho
Copy link

pultho commented Nov 9, 2020

@bonyod the prop is called pm.max_requests not pm.max_request 👀

When setting it to zero you're going to run into memory issues if garbage collection of PHP or some framework isn't working properly. You can set the number quite high to 20 - 50k requests if your app is performing well with memory but zero isn't recommended.

@mubasshir
Copy link

@pultho what should be the ideal config for m5.large?

@dwaqaddi
Copy link

dwaqaddi commented Apr 27, 2021

Hi @holmberd,

How did you came up with the values for the following?

pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

Is it safe to calculate it this way?

pm.start_servers = No. of cpu cores x 4
pm.min_spare_servers = No. of cpu cores x 2
pm.max_spare_servers = No. of cpu cores x 4

I have a t3.small server which has 2 cpus and 2 gb ram, is it okay to use the setting below with 'dynamic'?

Average Process = 70mb
RAM = 1900mb

pm.max_children = 27
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
pm.max_request = 1000

@thanksmia
Copy link

Hi, i did sudo nano on /etc/php/7.4/fpm/pool.d/www.conf

But the problem is I still get the same error log after increasing children to 17 i still get : WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

Help!!

@holmberd
Copy link
Author

holmberd commented Aug 5, 2021

@thanksmia I suggest you use a more appropriate channel to ask for help, e.g. https://stackoverflow.com

@nabtron
Copy link

nabtron commented Feb 14, 2022

@thanksmia this might be because your server is creating individual files for each virtual server / domain that you add. You need to check other files in the same folder of pool.d named as random numbers, and find the one that is for your server (it has the name of your server account on top of each file) and edit it as explained here: https://socalledhacker.com/index.php/2022/02/14/error-fpm-initialization-failed-solved/

@CRC-Mismatch
Copy link

CRC-Mismatch commented Jun 7, 2023

@abdennour I'm not sure you'd really want to set anything to "auto", since Kubernetes doesn't "replace" the presented values for available CPU cores or RAM size for each pod, it only applies its limits and that's it; if you run free -h or check for the number of CPU cores inside a pod, you'll see that the reported numbers are for the actual node this pod is running inside of. Setting everything to "auto" will only lead to a scenario where your HPA scales everything, but your ingress doesn't know that those pods are already over their limit, leading to a snowball effect inside each pod, leading to OOM kills and broken pipes for that pod while PHP-FPM restarts or tries to respawn dead workers that have no mCPUs or RAM available to work with (and the HPA starts going crazy, scaling to the max since from its POV, the scaling isn't solving the excessive resource consumption for some of the pods)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment