kubernetes nginx php-fpm pod

pod.yaml

# Create a pod containing the PHP-FPM application (my-php-app)
# and nginx, each mounting the `shared-files` volume to their
# respective /var/www/html directories.

kind: Pod
apiVersion: v1
metadata:
  name: phpfpm-nginx-example
spec:
  volumes:
    # Create the shared files volume to be used in both pods
    - name: shared-files
      emptyDir: {}
    
    # Add the ConfigMap we declared above as a volume for the pod
    - name: nginx-config-volume
      configMap:
        name: nginx-config

  containers:
    # Our PHP-FPM application
    - image: my-php-app:1.0.0
      name: app
      volumeMounts:
        - name: shared-files
          mountPath: /var/www/html
      # Important! After this container has started, the PHP files
      # in our Docker image aren't in the shared volume. We need to 
      # get them into the shared volume. If we tried to write directly
      # to this volume from our Docker image the files wouldn't appear
      # in the nginx container.
      #
      # So, after the container has started, copy the PHP files from this
      # container's local filesystem (/app -- added via the Docker image) 
      # to the shared volume, which is mounted at /var/www/html.
      lifecycle:
        postStart:
          exec:
            command: ["/bin/sh", "-c", "cp -r /app/. /var/www/html"]
    
    # Our nginx container, which uses the configuration declared above,
    # along with the files shared with the PHP-FPM app.
    - image: nginx:1.7.9
      name: nginx
      volumeMounts:
        - name: shared-files
          mountPath: /var/www/html
        - name: nginx-config-volume
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf

@HexalGJ, IMHO the file systems between them are not shared and by default are isolated, as you may use totally different base images.

To have common shared space you can use emptyDir{} but I have problem with it, because following reason:

1. emptyDirs start empty, so you need to initialize it.
2. emptyDirs usualy point to some directory in host side, so you can notice performance issue if your Kubernetes stand on slow virtual machine in cloud provider.
3. emptyDirs are not shared across replicas pods, so when you scale your deployments to 10 and in start initialize emptyDir by copying your web content it will consume on host side 10x space, so it`s huge waste of resources.

Another option is to have shared hostPath but again it`s not a perfect:

1. Again you use path that can bind to your vm storage with AKS, so have performance2 issue.
2. Your hostPath can be shared across replicas instances, but if you plan to have some blue-green deployments then you need to build own mechanism that will store data in different directories to avoid conflicts and finally purge data that belong to old not used deployments.

In theory you can use PVC as a storage for your www data but there is another set of problems:

1. You can bind them to one namespace, so you can forget to have logical separation of your domains in k8s namespaces.
2. In theory you can request many PVC, each one per namespace, but smaller PVC usually have less IOPS then big one and you will be charge for each separately(at least in azure).
3. Problem about blue-green deployment also occur here ;<

I don't know how this looks in different cloud providers, but at lest in azure aks there is no way to have single PVC based on SSD and share it across nodes. You can mount disk based on AzureFile but it's a samba share that use regular spinning disk, so it's slow.

@math

From the guide on using Kubernetes/PHP/Nginx

this solution is not working when you have 1 GB of software ... that cp command is taking long on copying stuf into var/www/html folder

Why mountPath: /var/www/html both image: my-php-app:1.0.0 and image: nginx:1.7.9 ?

Why mountPath: /var/www/html both image: my-php-app:1.0.0 and image: nginx:1.7.9 ?

php-fpm deals with php files, not assets like css/js/images

for PHP-heavy solutions like Drupal/Magento, this approach won't work. If you have several tens/hundres of thousands of files that you need to copy on every single scaled up POD, it will bring the app down to a crawl. I'm quite surprised that there's no straight-forward solution to sharing a folder from one container to another. for example, let's say I bake the Drupal code into the NGINX image; nginx will serve content from e.g. /usr/share/nginx/html/drupal , but the other container (php-fpm) won't have access to these files, so no dice, app won't run.

Seems we still need to rely on a distributed filesystem like NFS and the like. we need to precreate the volumes, copy over all the code and then deploy the app. This is fine (albeit cloud-based NFS solutions can be slow), however we can't do proper CI/CD for the code upgrades/patches. either we need to create a secondary volume with new codebase, or stop the app, upload the changes and then resume.

We've also looked into postStart actions, but given the number and size of the files, it will take a couple of minutes to finish up. this leads to errors as the entrypoint is executed before the postStart and perhaps half the files have been cloned/copied.

The copy operation should be performed from an init container, in that way pod will not be marked as available until the copy operation is finished.

Example for Laravel v6 / PHP 7.4

# ...
      initContainers:
        - name: shared-code
          image: org/custom-laravel-service:x.y.z
          command: [sh, -c]
          args: ['cp -r /var/www/html/* /data && chown www-data:www-data -R /var/www/html']
          imagePullPolicy: Always
          volumeMounts:
            - mountPath: /data
              name: shared-code
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 1000m
              memory: 256Mi
      containers:
        - name: nginx
         image: nginx:1.19-alpine
          volumeMounts:
            - name: shared-code
              mountPath: /var/www/html
              readOnly: true
#...

@JulienBreux - how many files do you have in /var/www/html ? In our custom Drupal code, we are reaching upwards of 50K files, which takes somewhere around 10min to copy. It makes Kubernetes a no go unfortunately.

The easy solution to this is not to copy stuff. Build an nginx container with your static resources in it; use a cloud storage solution to store dynamically uploaded content. see: https://carnage.github.io/2019/10/opencfp-on-gke for a full example

If you have large amounts of files, the simplest solution (although not very dockery) is to put both nginx and php-fpm on the same container and use something like supervisor. People don't like it, but the solution proposed here is a no-go when folders are large. Besides it being wasteful too.

For Drupal S3 module is now in Beta mode, and I have tested it out on our lower environments. It works fine. There is no need to copy all the files each time a container is created. Are you running composer install before creating the image or after the pod is created?

@JulienBreux - how many files do you have in /var/www/html ? In our custom Drupal code, we are reaching upwards of 50K files, which takes somewhere around 10min to copy. It makes Kubernetes a no go unfortunately.

One index.php ! Because our statics assets are on another public URL.

Since I keep seeing this same exact emptyDir technique (which gets initialized via initContainers / postStart) everywhere, let me share an idea that I’m bouncing around in my head that I may experiment with later:

• Single Dockerfile with multistage build
• Stage 1: Have a stage based on nginx, e.g. FROM nginx:1.2.3 AS nginx (or in my case Apache, because: legacy)
• Stage 2: Have a stage based on php, e.g. FROM php:7-fpm AS php
• In each stage copy files needed into each image via COPY
• During your build, just use --target=stage to build just that stage and also tag it separately (e.g. docker build . --target=nginx -t gcr.io/org/image:nginx)

Now you have two separate images, both already coming pre-bundled with their respective dependencies. Then you can run both totally separate pods/deployments (or as a sidecars if you really want). Plus, context is preserved and your single Dockerfile very clearly defines in plain readable terms that it generates two separate build artifacts using stages/targets.

The issue is that we have a combination of a few problems, basically all centered around depending on the same set of files:

• A web server of some sort that depends on (and serves) static files (e.g. nginx or apache)
• PHP scripts which often coexist adjacent to these static files which must also run in a separate process (php-fpm)

The approach I’m seeing others taking to this (because it is intuitive) is to bundle all of our project-specific assets together in one place (e.g. into the php-fpm image) and then try to copy them over at the point of delivery (e.g. in the Kubernetes pod). What I’m suggesting instead is to flip that to the build stage instead, since it’s already deterministic… i.e. we already know what’s going go be mounted/copied into each image head of time. Why repeat this work every single time your pod has to go live when it’s not necessary? Just ship it with each container. 🤔

p.s. I like this (running nginx and php-fpm independently instead of using some process to bundle both into the same image) because it allows Kubernetes to do it’s job and manage and scale the processes independently, as it was built. Plus I assume it’s better tested.

Thank you for this 🙏

Apparently, there are several approaches given the currently-available methods for mounting. I plan to move forward with the initContainers method. In my case, the startup time for copying files is ~20 seconds (and will not grow much), which is bearable.

I wanted to share my approach for others to see and critique:
I am solely mounting a shared directory between the initContainer and the nginx container. My php container already has the files, so no need to copy. I have an .env file secret mounted with environment values. Choosing to not mount the emptyDir volume in my php container maintains more separation - my .env secrets will only exist in the one container, limiting spread.

spec:
    containers:
        - name: php
          image: my/app:latest
          ...
          lifecycle:
              postStart:
                  exec:
                      command:
                          - "/bin/sh"
                          - "-c"
                          - "cp /path/to/secrets/.env /var/www/html"
          volumeMounts:
              - name: app-env
                mountPath: /path/to/secrets
                readOnly: true
        - name: nginx
          image: nginx:alpine
          ...
          volumeMounts:
              - name: web-root
                mountPath: /var/www/html
                readOnly: true
              ...
    initContainers:
        - name: setup-web-root
          command:
              - "/bin/sh"
              - "-c"
              - "cp -r /var/www/html/. /app"
          image: my/app:latest
          imagePullPolicy: Always
          volumeMounts:
              - name: web-root
                mountPath: /app
    volumes:
        - name: web-root
          emptyDir: {}
        - name: app-env
          secret:
              secretName: app-env-secret
              items:
                  - key: .env
                    path: .env

@lstellway methods work perfectly. But I wanted to know is there any issue running both nginx and php-fpm together?

@rsidhaarth This doc at least provides a reasonable argument against running them both together: https://cloud.google.com/architecture/best-practices-for-building-containers#package_a_single_app_per_container

I still feel like a good middle ground here might be to have a single Dockerfile (or multiple Dockerfile’s) based on the same core app image containing all your app files which then is used as a COPY --from source for your separate php-fpm and nginx daemon images. But, I know the practicality of that may depend heavily on your use case (e.g. build times). Personally I like shifting that to build time, since granted the images will be larger and the build time may increase (e.g. new nginx image for each deploy) but then the result is more granular and easily reproducible and less prone to ephemeral failures at runtime or lower startup cost (since files are already there). But that’s just my 2c since it’s hypothetical for me (right now I still roll php-apache which integrates PHP already in an Apache server container via mod_php, lol 😅)

@patricknelson thanks for your reply

I have already done that in the docker swarm setup. I got both nginx and php-fpm in a single image and deployed them there. If that is the best way, I will do the same for k8s.

Also, for persistent storage, what method is recommended?

Longhorn, NFS, GFS etc

I am trying to use K8s for Magento 2 setup.

Also, for persistent storage, what method is recommended?

I guess whatever the default is from your provider. Are you talking about shared storage? If so, I've used NFS in Google Cloud which is Filestore for shared persistent volumes (minimum is 1TB basic HDD @ $204/mo for a single instance). Or I guess you could roll our own GlusterFS or NFS server. Otherwise, just go with whatever the default is.

https://kubernetes.io/docs/concepts/storage/persistent-volumes/

Yes, I am speaking about shared storage that supports ReadWriteMany, and my provider needs to provide one. So I have to build my one storage. For Magento 2, the recommended is GFS, and I am exploring Longhorn now.

BTW, I just tried to deploy both nginx and php-fpm as a single images and now my deployment file is straightforward, and execution is also faster.

1. there is no need of initcontainer
2. there is no need for copy commands as the image has all the files
3. deployment is faster as it just going to pull one image and deploys one container

Next up to test/validate your cluster, do some pre-prod testing. Scale it up and do some load testing! Pound it with a bunch of traffic (e.g. Gatling). Then, maybe delete a line of code on one of the containers, see how easy it is for you to see/diagnose errors when only some of your requests are failing. When your new persistent storage is setup, put a 3mb .jpg on one of the containers in the shared volume and see if it’s available on all the others and/or how long it takes to sync.

… good luck!

For the record, I have since moved towards bundling PHP + the web server in the same image.
I believe I followed TrafeX/docker-php-nginx as an example.

For the record, I have since moved towards bundling PHP + the web server in the same image.
I believe I followed TrafeX/docker-php-nginx as an example.

Thanks for sharing this. I will make use of it :)

Same, that looks like a mature and well implemented image @lstellway.

One of those times I really wish Github Gists had reactions. That way we could just react with 👍 or whatever instead of spamming the thread. 😁