Skip to content

Instantly share code, notes, and snippets.

@kayintveen
Last active March 26, 2022 08:45
Show Gist options
  • Save kayintveen/7d72c068bc98ed5c91e9edffdfbbfb82 to your computer and use it in GitHub Desktop.
Save kayintveen/7d72c068bc98ed5c91e9edffdfbbfb82 to your computer and use it in GitHub Desktop.
Local Magento2 Development on Mac OS X Catalina and Mojave. Solving FD_SETSIZE Max processes and Max file descriptior issues.

Local Magento2 development on Mac OS X Catalina

Added the document to Medium and expanded it a bit with more configuration examples:

https://medium.com/@k.veen/setup-a-local-php-development-environment-on-mac-catalina-for-laravel-and-magento-2-development-3d7f1c587d86

After a lot of headace and sturggle i went back to what i trust. no more Laravelt Valet or Valet-plus or any other wrappers and workarounds. Sure a few things like enabling and disabling Xdebug is a tad bit more work. but its matter of seconds.

What will we be installing

  1. Nginx
  2. [email protected] [email protected] and [email protected] next to each other
  3. [email protected]
  4. Redis
  5. Xdebug
  6. Elasticsearch 2.4 5.8 and 6.4 next to each other without docker or other bullshit

Solve OS X Catalina File Descriptor exceeded FD_SETSIZE error

I added to configuration files underneath that solves the Max processes and Max files issues existing on both Mac OS X Catalina and Mac OS X Mojave. We need to configure these settings simply my upping the ulimit in your terminal. But i think its much easier to add a launchagent that will be applied on every reboot.

Add the files limit.maxproc.plist and limit.maxfiles.plist to the /Library/LaunchDaemons/ directory. and then load them in to your system

 sudo launchctl unload -w /Library/LaunchDaemons/limit.maxproc.plist
 sudo launchctl unload -w /Library/LaunchDaemons/limit.maxfiles.plist 

Start clean

Clean all your brew services you don't need. remove old php version, nginx, redis

brew services list

to list all your services, stop all php nginx, redis, optionally apache if its running.

brew services uninstall <package name>

Sometimes you get the message its KEG-Only and you need to run command to remove package. for examples:

sudo rm -rf /usr/local/Cellar/[email protected]

Install php

We are going to install all needed php version, in this case 7.1 7.2, 7.3 and optionally if you want the new [email protected]

brew install [email protected]
brew install [email protected]
brew install [email protected]

We do not start the packages yet because we are first going ot change a few files before we do this. Since we want to run php 7.1 through 7.3 next to each other we need to be sure they don't conflict. Change the following files: /usr/local/etc/php/"PHP_VERSION"/php-fpm.d/www.conf

user = <your mac user>
group = staff

for fastCGI proxy we setup different ports for each php version for [email protected] for example we use:

listen = 127.0.0.1:9071

for 7.2 we use

listen = 127.0.0.1:9072

also do this for 7.3 and 7.4 if needed. for performance also change the following

pm = dynamic
pm.max_children = 96
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 16384

Then we continue on to the php.ini files. Change the following files: /usr/local/etc/php/"PHP_VERSION"/php.ini And change the following lines

max_execution_time = 600
max_input_time = -1
max_input_vars = 50000
memory_limit = 1024M
upload_max_filesize = 64M

[opcache]
opcache.enable=1
opcache.enable_cli=0
opcache.memory_consumption=256
opcache.interned_strings_buffer=12
opcache.max_accelerated_files=65000
opcache.validate_timestamps=0
opcache.fast_shutdown=1

install Nginx

Install Nginx via brew

brew install nginx

Go to your nginx configuration file /usr/local/etc/nginx/nginx.conf and set the following configuration:

user  <your mac user> staff;
worker_processes  auto;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    gzip                on;
    gzip_comp_level     5;
    gzip_min_length     256;
    gzip_proxied        any;
    gzip_vary           on;

    gzip_types
    application/atom+xml
    application/javascript
    application/json
    application/rss+xml
    application/vnd.ms-fontobject
    application/x-font-ttf
    application/x-web-app-manifest+json
    application/xhtml+xml
    application/xml
    font/opentype
    image/svg+xml
    image/x-icon
    text/css
    text/plain
    text/x-component;

    upstream fastcgi_backend {
        server 127.0.0.1:9071;
    }

    include servers/*;
}

In this case we use server 127.0.0.1:9071; so this means we are using php 7.1. Change to 9072 or 9073 accordingly. You can also specify this in the specific server configuration to use php 7.1 on one and 7.2 on the other.

As you can see we include all files in servers. This is the place where we configure the different sites.

inside /usr/local/etc/nginx/servers add a new file for your local development location for example magento2.test I personally use a "sites" folder inside my users root directory. Create a folder for your magento2 install and add the following lines to the magento2.test file. You can add multiple files for multiple sites.

server {
    listen			80;
    server_name		magento2.test;

    set $MAGE_ROOT /Users/YOUR_MAC_USER/sites/magento2;

    include /Users/YOUR_MAC_USER/sites/magento2/nginx.conf.sample;
}

install Mysql version 5.7

Install [email protected] via homebrew

brew install [email protected]

Then open the my.cnf file from the following directoy /System/Volumes/Data/usr/local/etc/my.cnf It's possible that its not there. then you can create this file. And add the following lines:

[client]
user=root
password=root
host=localhost

[mysqld]
bind-address = 127.0.0.1
sql_mode="NO_ENGINE_SUBSTITUTION"
innodb_file_per_table=OFF
open_files_limit=999999
local_infile=ON
secure_file_priv=""
max_allowed_packet=1073741824
max_connections=100000
key_buffer_size=2G
innodb_buffer_pool_size=12G
query_cache_size=67108864
query_cache_type=1
query_cache_limit=4194304
table_open_cache=4096
innodb_buffer_pool_instances=24
innodb_sort_buffer_size=2G
sort_buffer_size=1G
innodb_flush_log_at_trx_commit=0
innodb_log_file_size=3G
interactive_timeout=3600
max_connect_errors=1000000
thread_cache_size=4096

[mysqld_safe]
open_files_limit=999999

Then restart the service

brew services restart [email protected]

After this you can locally just run mysql command without username and password to easily login.

Install Redis

for Magento 2 development it's smart to have Redis up and running

brew install redis
brew services start redis

installing and configuring Xdebug

To install xdebug we need to be sure we use pecl from the specific and correct version of PHP. Run the following command replacing the version number for your needed php version.

/usr/local/opt/[email protected]/bin/pecl install xdebug

This will automatically add and enable xdebug, this is something we do not want due to performance issues it can cause i like to be able to enable and disable xdebug only when i needed.

Remove the following files from php.ini:

zend_extension="xdebug.so"

Add a xdebug.ini file to the following directories: /usr/local/etc/php/"PHP_VERSION"/conf.d and add the following lines of code.

zend_extension="xdebug.so"
[xdebug]
xdebug.remote_enable=1
xdebug.default_enable=1
xdebug.remote_autostart=1
xdebug.remote_host=localhost
xdebug.remote_port=9000
xdebug.idekey=PHPSTORM

Now this is automatically read by php.ini.

To make a command to enable and disable xdebug with just one command add this to your ~/.bash_profile or ~/.zshrc After adding this be sure to reload by restarting your terminal or do source ~/.bash_profile or source ~/.zshrc

alias xdebugoff="mv /usr/local/etc/php/7.1/conf.d/xdebug.ini /usr/local/etc/php/7.1/conf.d/xdebug.ini.bk || mv /usr/local/etc/php/7.2/conf.d/xdebug.ini /usr/local/etc/php/7.2/conf.d/xdebug.ini.bk || mv /usr/local/etc/php/7.3/conf.d/xdebug.ini /usr/local/etc/php/7.3/conf.d/xdebug.ini.bk"
alias xdebugon="mv /usr/local/etc/php/7.1/conf.d/xdebug.ini.bk /usr/local/etc/php/7.1/conf.d/xdebug.ini || mv /usr/local/etc/php/7.2/conf.d/xdebug.ini.bk /usr/local/etc/php/7.2/conf.d/xdebug.ini || mv /usr/local/etc/php/7.3/conf.d/xdebug.ini.bk /usr/local/etc/php/7.3/conf.d/xdebug.ini"

If your xdebug is already off or on you will get a warning that the file does not exist, this is o.k.

Install Elasticsearch

I'm not a fan of running Elasticsearch via homebrew or docker. The quickest and easiest and also the most performance friendly solution is. Nothing worst than a elasticsearch instance running continiously on the background of your mac eating system resources.

First install java

brew cask install homebrew/cask-versions/adoptopenjdk8

Download the needed Elasticsearch version from the past-releases page https://www.elastic.co/downloads/past-releases I personally have a /etc/ directory inside my root directoy ~/etc i made a folder elasticsearch and a folder for each version. extract the files from the zip or tar you just downloaded and add them to your elasticsearch directory. i currently have ~/etc/elasticsearch/2.4 the only this i have to do is open a terminal window and run ~/etc/elasticsearch/2.4/bin/elasticsearch wait 2 sec till it starts up and elastic search is running untill i quit the terminal window.

To let it function with Magento2. you also need to install the icu and phonetic package Go to your elasticsearch folder and run

 bin/elasticsearch-plugin install analysis-phonetic
 bin/elasticsearch-plugin install analysis-icu

setup host file to server .test domains

There are many solutions like dnsmasq or ngrok. but the easiest is just setting up

 sudo vim /etc/hosts
 or
 sudo nano /etc/hosts

to change the host file under super user rights. and then add for example

 127.0.0.1 magento2.test

add SSL support to your local domains.

The easiest is to use mkcert tool.
https://github.com/FiloSottile/mkcert

 brew install mkcert

to install the mkcert package on your machine. Then make the root certificate with

 mkcert -install

Then create a directory to store you certicates. I personally made a ~/.ssl directory. Then cd into that directory and run the following command

 mkcert domain.test

with of course replacing the domain for your test domain. You can also do multiple in one certificate.

 mkcert domain1.test domain2.test domain3.test

This will result in two file in your directoy. A domain.test+3.pem for example. and a domain.test+3-key.pem. The files are needed for your server configuration. If you want one certificate for all domains you can also do

 mkcert *.local.test

for example be sure to add that to your hostfile. but then you have a wildcard for all. a *.test is not supported by many browsers.

A server configuration should look like this:

 server {
      listen			443 ssl;
      server_name		projectname.local.test;

     ssl_certificate /Users/kayintveen/.ssl/projectname.test+3.pem;
     ssl_certificate_key /Users/kayintveen/.ssl/projectname.test+3-key.pem;

      set $MAGE_ROOT /Users/kayintveen/sites/projectname;

      include /Users/kayintveen/sites/projectname/nginx.conf.sample;
 }
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>limit.maxfiles</string>
<key>ProgramArguments</key>
<array>
<string>launchctl</string>
<string>limit</string>
<string>maxfiles</string>
<string>262144</string>
<string>524288</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>ServiceIPC</key>
<false/>
</dict>
</plist>
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>limit.maxproc</string>
<key>ProgramArguments</key>
<array>
<string>launchctl</string>
<string>limit</string>
<string>maxproc</string>
<string>2784</string>
<string>4176</string>
</array>
<key>RunAtLoad</key>
<true />
<key>ServiceIPC</key>
<false />
</dict>
</plist>
server {
listen 80;
server_name magento_server_config.test;
set $MAGE_ROOT /Users/MAC_USER/sites/magento_server_config;
include /Users/MAC_USER/sites/bbb/nginx.conf.sample;
}
[client]
user=root
password=root
host=localhost
[mysqld]
bind-address = 127.0.0.1
sql_mode="NO_ENGINE_SUBSTITUTION"
innodb_file_per_table=OFF
open_files_limit=999999
local_infile=ON
secure_file_priv=""
max_allowed_packet=1073741824
max_connections=100000
key_buffer_size=2G
innodb_buffer_pool_size=12G
query_cache_size=67108864
query_cache_type=1
query_cache_limit=4194304
table_open_cache=4096
innodb_buffer_pool_instances=24
innodb_sort_buffer_size=2G
sort_buffer_size=1G
innodb_flush_log_at_trx_commit=0
innodb_log_file_size=3G
interactive_timeout=3600
max_connect_errors=1000000
thread_cache_size=4096
[mysqld_safe]
open_files_limit=999999
user MAC_USER staff;
worker_processes auto;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
gzip on;
gzip_comp_level 5;
gzip_min_length 256;
gzip_proxied any;
gzip_vary on;
gzip_types
application/atom+xml
application/javascript
application/json
application/rss+xml
application/vnd.ms-fontobject
application/x-font-ttf
application/x-web-app-manifest+json
application/xhtml+xml
application/xml
font/opentype
image/svg+xml
image/x-icon
text/css
text/plain
text/x-component;
upstream fastcgi_backend {
server 127.0.0.1:9071; // Change port according to php version
}
include servers/*;
}
[PHP]
engine = On
short_open_tag = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = -1
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
error_reporting = E_ALL
display_errors = On
display_startup_errors = On
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = On
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 8M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
extension_dir = "/usr/local/lib/php/pecl/20160303"
enable_dl = Off
file_uploads = On
upload_max_filesize = 2M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
[CLI Server]
cli_server.color = On
[Date]
[filter]
[iconv]
[imap]
[intl]
[sqlite3]
sqlite3.defensive = 1
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[Phar]
[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = Off
[SQL]
sql.safe_mode = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = On
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.sid_length = 26
session.trans_sid_tags = "a=href,area=href,frame=src,form="
session.sid_bits_per_character = 5
[Assertion]
zend.assertions = 1
[COM]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[sysvshm]
[ldap]
ldap.max_links = -1
[mcrypt]
[dba]
[opcache]
opcache.enable=1
opcache.enable_cli=0
opcache.memory_consumption=256
opcache.interned_strings_buffer=12
opcache.max_accelerated_files=65000
opcache.validate_timestamps=0
opcache.fast_shutdown=1
[openssl]
openssl.cafile = "/usr/local/etc/[email protected]/cert.pem"
openssl.capath = "/usr/local/etc/[email protected]/certs"
server {
listen 443 ssl;
server_name projectname.local.test;
ssl_certificate /Users/kayintveen/.ssl/projectname.test+3.pem;
ssl_certificate_key /Users/kayintveen/.ssl/projectname.test+3-key.pem;
set $MAGE_ROOT /Users/kayintveen/sites/projectname;
include /Users/kayintveen/sites/projectname/nginx.conf.sample;
}
[www]
user = MAC_USER
group = staff
listen = 127.0.0.1:9071 // Change 9071 to your php version
pm = dynamic
pm.max_children = 96
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 16384
zend_extension="xdebug.so"
[xdebug]
xdebug.remote_enable=1
xdebug.default_enable=1
xdebug.remote_autostart=1
xdebug.remote_host=localhost
xdebug.remote_port=9000
xdebug.idekey=PHPSTORM
@bonified2x
Copy link

Thank you for sharing this. I'm having the problem with mySQL 5.7.25 in MAMP Pro under macOS 10.14.6. Every so often I'm hitting the open files limit [Warning] File Descriptor 1030 exceeded FD_SETSIZE=1024

Can I simply load your 2 files limit.maxfiles.plist and limit.maxproc.plist reboot and be good?

@kayintveen
Copy link
Author

If you follow the text under "Solve OS X Catalina File Descriptor exceeded FD_SETSIZE error" that will solve your issue.

@bonified2x
Copy link

Ok which was this: I did that

I added to configuration files underneath that solves the Max processes and Max files issues existing on both Mac OS X Catalina and Mac OS X Mojave. We need to configure these settings simply my upping the ulimit in your terminal. But i think its much easier to add a launchagent that will be applied on every reboot.

Add the files limit.maxproc.plist and limit.maxfiles.plist to the /Library/LaunchDaemons/ directory. and then load them in to your system

sudo launchctl unload -w /Library/LaunchDaemons/limit.maxproc.plist
sudo launchctl unload -w /Library/LaunchDaemons/limit.maxfiles.plist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment