Some network diagram somewhere

pre-gliffy

               (INTERNET)
                   |
HAPROXY\           |    /HAPROXY-STANDBY
NFS- DRBD1 <------VIP   DRBD2 -NFS-STANDBY
MYSQL/ /  /|\ \         \MYSQL-STANDBY
PUPPET/  / | \ \STORAGE
        /  |  \
WEB1 WEB2 WEB3 WEB4 (apache's mpm-itk/nginx php5-fpm)




Things we cant limit :  Evil/Bad/Dumb PHP code
                        Users with bad passwords

Things we can limit:
                        PHP
                                Apache:
                                        mpm-itk takes care of setuid at apache level 
                                        (no need for suexec)
                                        CONCERN?: mpm-itk needs root access to files on NFS, so it has
                                        to be exported with no_root_squash. Is this a major security
                                        concern?

                                Nginx:
                                        fork php5-fpm processes by uid
                                        only exec files from defined directories
                                        chroot
                                # of processes (ulimit)
                                Mem Limit of processes (php.ini)
                                what is avaliable (no exec, no php-cli)
                        Mysql
                                queries per account per hour
                                updates per account per hour
                                # of connections per hour
                                # of concurent connetions

                        NFS
                                DISK QUOTAS:
                                                edquota peruser

                        HAPROXY
                                rate-limit sessions? (probably cant use this, as its global)

                       
Monitoring:
                       Remote Syslog server -Rsyslog (unified logging)
                                - Monitor logs per minute to generate alerts (or monitor errors/etc)
                                - check out splunk

                       Sar (systat package) (IO PREFORMANCE CRON) 
                             - munin monitors this -> $(sar | tail -n1 | awk {' print $6 '} | cut -d . -f1)
                       
                       lm-sensors (mother board sensors (IE temperature)


Security:                 
                           IPtables: I looked at the loadbalancer iptables-save and was scared.
                                     Set simple blocking on the public interface for the webservers (still get updates stop worms) start with logging stanza rather than reject to ensure nothing critical is blocked 
                                     MySQL is publicly open on db1 and db2 

                           OpenVZ: http://en.wikipedia.org/wiki/OpenVZ  (better than chroot) 

                           Apache/Ningx: 
                                     Modsecurity - application firewall: (detects oddd urls, leave in in permissive mode to create a profile of allowed actions (enable when there are no more false positives)
                       
                        Selinux: Enable, leave in permissive mode and generate alerts (email)
                                Eventually once you have cleared all the false positives, set in enforcing mode and generate alerts if selinux is triggered)

                       Pingdom/Blamestall/whatever (cheep and effective web monitoring

                       Intrusion Detection
                                http://sourceforge.net/projects/tripwire/ 
                                http://aide.sourceforge.net/


Current Issues:
                        NFS bottleneck
                                options:
                                        - distributed filesystem (Gluster?)
                                        - or is there a way to make local copies of popular sites, and      
                                          shadow other files to primary nfs?
                                        -memcached can store small files (you could blow the cache on ftp upload or just expire every 15 minutes. (cron detecintg date stamp change)


                        Too localised (Dallas)
                                architecture definition for standalone nodes / secondary clusters

                        Poor backup policy
                                short-term (3-month) should be on our server
                                good archival services offered by SoftLayer
                                setup mysql slave

                        Provisioning for new webserver cloud instances
                                need better puppet recipe