Setup Caching and Proxy with Nginx in Centos/Fedora P1

Recently I started to tackle a load problem on one of my personal sites, the issue was that of a poorly written but exceedingly MySQL heavy application and the load it would induce on the SQL server when 400-500 people were hammering the site at once. Further compounding this was Apache’s horrible ability to gracefully handle excessive requests on object heavy pages (i.e: images). This left me with a site that was almost unusable during peak hours — or worse — would crash the MySQL server and take Apache with it by frenzied F5ing from users.

I went through all the usual rituals in an effort to better the situation, from PHP APC then Eaccelerator, to mod_proxy+mod_cache, to tuning Apache timeouts/prefork settings and adjusting MySQL cache/buffer options. The extreme was setting up a MySQL replication cluster with MySQL-Proxy doing RW splitting/load balancing across the cluster and memcached, but this quickly turned into a beast to manage and memcached was eating memory at phenomenal rates.

Although I did improve things a bit, I had done so at the expense of vastly increased hardware demand and complexity. However, the site was still choking during peak hours and in a situation where switching applications and/or getting it reprogrammed is not at all an option, I had to start thinking outside the box or more to the point, outside Apache.

I have experience with lighttpd and pound reverse proxy, they are both phenomenal applications but neither directly handles caching in a graceful fashion (in pounds case not at all). This is when I took a look a nginx which to date I had never tried but heard many great things about. I fired up a new Xen guest running CentOS 5.4, 2GB RAM & 2 CPU cores….. an hour later I had nginx installed, configured and proxy-caching traffic for the site in question.

The impact was immediate and significant — the SQL server loads dropped from an average of 4-5 down to 0.5-1.0 and the web server loads were near non-existent from previously being on the brink of crashing every afternoon.

Enough with my ramblings, lets get into nginx. You can download the latest release from http://nginx.org and although I could not find a binary version of it, compiling was straight forward with no real issues.

First up we need to satisfy some requirements for the configure options we will be using, I encourage you to look at ‘./configure –help’ list of available options as there are some nice features at your disposal.

# yum install -y zlib zlib-devel openssl-devel gd gd-devel pcre pcre-devel

Once the above packages are installed we are good to go with downloading and compiling the latest version of nginx:

# wget http://nginx.org/download/nginx-0.8.36.tar.gz

# tar xvfz nginx-0.8.36.tar.gz

# cd nginx-0.8.36/

# ./configure –with-http_ssl_module –with-http_realip_module –with-http_addition_module –with-http_image_filter_module –with-http_gzip_static_module

# make && make install

This will install nginx into ‘/usr/local/nginx’, if you would like to relocate it you can use ‘–prefix=/path’ on the configure options. The path layout for nginx is very straight forward, for the purpose of this post we are assuming the defaults:

# ls /usr/local/nginx

conf  fastcgi_temp  html  logs  sbin

# cd /usr/local/nginx

# ls conf/

fastcgi.conf  fastcgi.conf.default  fastcgi_params  fastcgi_params.default  koi-utf  koi-win  mime.types  mime.types.default  nginx.conf  nginx.conf.default  win-utf

The layout will be very familiar to anyone that has worked with Apache and true to that, nginx breaks the configuration down into a global set of options and then the individual web site virtual host options. The ‘conf/’ folder might look a little intimidating but you only need to be concerned with the nginx.conf file which we are going to go ahead and overwrite, a copy of the defaults is already saved for you as nginx.conf.default.

My nginx configuration file is available here , be sure to rename it to nginx.conf or copy the contents listed below into ‘conf/nginx.conf’:

Replacing failed disk drives in a ZFS pool

If you have a failed disk in a ZFS Pool and needed to swap it out to restore the pool to an optimal state. To begin the swap out, we used the zpool utility to see which disk drive was faulty:

$ zpool status -v

pool: rz2pool

 state: DEGRADED

status: One or more devices could not be opened.  Sufficient replicas exist for

        the pool to continue functioning in a degraded state.

action: Attach the missing device and online it using ‘zpool online’.

   see: http://www.sun.com/msg/ZFS-8000-D3

 scrub: resilver completed with 0 errors on Tue Feb 13 14:12:37 2011

config:

        NAME          STATE     READ WRITE CKSUM

        rz2pool       DEGRADED     0     0     0

          raidz2      DEGRADED     0     0     0

            c1t9d0    ONLINE       0     0     0

            c1t10d0   ONLINE       0     0     0

            c1t12d0   ONLINE       0     0     0

            c2t1d0    ONLINE       0     0     0

            spare     DEGRADED     0     0     0

              c2t2d0  UNAVAIL      0     0     0  cannot open

              c2t3d0  ONLINE       0     0     0

        spares

          c2t3d0      INUSE     currently in use

Here, We used  zpool with the “replace” option to replace the failed drive in my pool:

$ zpool replace rz2pool c2t2d0 c2t2d0

After the replacement operation completed, I used zpool to monitor the resilvering of the replacement drive:

$ zpool status -v

pool: rz2pool

 state: DEGRADED

status: One or more devices is currently being resilvered.  The pool will

        continue to function, possibly in a degraded state.

action: Wait for the resilver to complete.

 scrub: resilver in progress, 0.10% done, 0h31m to go

config:

        NAME                STATE     READ WRITE CKSUM

        rz2pool             DEGRADED     0     0     0

          raidz2            DEGRADED     0     0     0

            c1t9d0          ONLINE       0     0     0

            c1t10d0         ONLINE       0     0     0

            c1t12d0         ONLINE       0     0     0

            c2t1d0          ONLINE       0     0     0

            spare           DEGRADED     0     0     0

              replacing     DEGRADED     0     0     0

                c2t2d0s0/o  UNAVAIL      0     0     0  cannot open

                c2t2d0      ONLINE       0     0     0

              c2t3d0        ONLINE       0     0     0

        spares

          c2t3d0            INUSE     currently in use

errors: No known data errors

All of this was done online, and with minimal interruption to the applications running on the host

You can want to know:

1. Basic guide to using ZFS in Solaris
2. How to Create and manipulate zpools
3. How to Autoinstall packages in Solaris
4. Optimize Solaris’s TCP for internet services
5. Add Swap space in solaris

Set up MySQL database replication

Database replication will require the use of two machines. The reason you want to use two machines is to ensure you will always have a working copy of your database (should one machine die). When using replication you think of your machines as Master and Slave. The Master is the machine that holds the original database. The Slave is the machine that holds the replicated database (the backup per say.) Both machines must have a working MySQL installation and must be networked together. You will need root access as well as access to the database administrator on both machines.

Setting up the Master

For purpose of example our database to replicate will be called “sample_database”.? The first thing you need to do is open up the /etc/mysql/my.conf file for editing. You need to search for the lines:

#skip-networking

#bind-address = 127.0.0.1

Uncomment out these lines (remove the “#” character). If these lines do not exist, add them. Now we have to inform mysql which database we plan on replicating. For this you will add the following lines:

log-bin = /var/log/mysql/mysql-bin.log

binlog-do-db=sample_database

server-id=1

The above lines do the following:

Line 1: Instruct mysql what log file to use.

Line 2: Instruct mysql which database to replicate

Line 3: Instruct mysql that this machine is the master.

Restart mysql on this machine with the following command:

# /etc/init.d/mysql restart

The next step is to log into the mysql terminal and create a user that has replication privileges. To do this issue the command:

# mysql -u root -p

You will prompted for the mysql admin password. Now we have to make some mysql magic. From the mysql prompt issue the following:

GRANT REPLICATION SLAVE ON *.* TO ‘USER’@'%’ IDENTIFIED BY ”;

Where NEW_PASSWORD is the actual password and USER is the actual username that will have replication privileges.

Now issue the command:

 FLUSH PRIVILEGES;

You’re not done with mysql yet. Now issue the command:

 USE sample_database;

And now the command:

FLUSH TABLES WITH READ LOCK;

And now you need to make sure mysql is seeing the Master and can show the status of the master with the command:

SHOW MASTER STATUS;

When you issue the above command you should see a listing printed out for your sample_database. Write this information down (you will see a Position number that you will need later).

Now you need to get tables and data from the sample_database. The method I will show you requires that the database on the Master be locked momentarily. To that end the database will be unavailable until the database is unlocked. Keep this in mind when setting this up.

 Now you’re ready to set up the Slave. Move over to the machine that will serve as the slave.

The first thing to do on the Slave is to create the database the Master will write to. On the slave open up the mysql shell with:

# mysql -u root -p

You will be prompted for the password. Now enter the following command to create the database:

 CREATE DATABASE sample_database;

And quit mysql with the “quit” command.

Now to configure MySQL to know it is the slave and open up the /etc/mysql/my.conf file for editing. Add the following lines:

server-id=2

master-host=IP_ADDRESS_OF_MASTER

master-user=USER

master-password=USER_PASSWORD

master-connect-retry=60

replicate-do-db=sample_database

Where IP_ADDRESS_OF_MASTER is the actual IP address of the master server, USER is the user created on the master for replication, and USER_PASSWORD is the password given to the replication user on the Master.

Save that file and restart mysql on the slave with the command:

# /etc/init.d/mysql restart

Now we have to load the data from the Master into the Slave. This is when the database will be locked. Here are the commands to load the data:

# mysql -u root -p

Enter the mysql admin password. Now from the mysql prompt enter the following command:

LOAD DATA FROM MASTER;

Now exit the mysql prompt with the command quit.

The next step is to stop the slave so you can finish up the Slave configuration. Issue the following:

# mysql -u root -p

You will prompted for the root password.

now enter the command:

SLAVE STOP;

Now the next command is a bit lengthy:

CHANGE MASTER TO MASTER_HOST=’IP_ADDRESS_OF_MASTER’, MASTER_USER=’USER’, MASTER_PASSWORD=’USER_PASSWORD’, MASTER_LOG_FILE=’mysql-bin.007′, MASTER_LOG_POS=NUMBER;

 Where:

IP_ADDRESS_OF_MASTER is the actual IP address of the Master server.

USER is the actual user you created on the Master.

USER_PASSWORD is the actual password you gave the user on the Master.

mysql-bin.007 is the File name from the output of the SHOW MASTER STATUS command from above.

MASTER_LOG_POS is the Position given in the output of the SHOW MASTER STATUS command from above.

Finally issue the command:

SLAVE START;

and then quit the mysql prompt with the command “quit”.

You’re done. If you have phpmyadmin installed you can check the status of the databases as they updated. You now have database replication up and running.

You can want to know:

1. Backup remote server – Rsync
2. 10 Ways to Automatically and Manually Backup MySQL Database
3. Use Rsync for Backups
4. Rsync tutorial, tips and examples
5. Deploying highly available Oracle databases with Sun Cluster 3.2