Amazon S3 and CloudFront with WordPress and DreamHost

Amazon web services

Amazon web services

For the last couple of days I have been testing out the Amazon CloudFront service using my blog as a sandbox. My goal was pretty simple: To set up Amazon CloudFront on my blog, enabling static files like images and styles to be distributed for fast access. This post describes my experimentation with it. It is not just straight forward work, and there are different approaches you may choose among. So I thought I should write a short walk through of the steps that I have taken. If you wonder what Amazon CloudFront is, or looking for information on how it could be used in conjunction with WordPress this post can give you an overview and help you getting started.
[% oiopub-banner-2-right %]

Let’s start with Amazon CloudFront and what this actually is. CloudFront is a Content Distribution Network (CDN). The service distributes your data out to multiple datacenter locations, ensuring faster access times. It works together with another service from Amazon S3 (Simple Storage Service). Amazon S3 is a data storage service where your files is stored initially. To be using CloudFront for distribution you need to have a S3 account and store the files there.

Setting up an Amazon Web Services account
As I have blogged about before, I use Amazon S3 already, so it was not necessary for me to create an Amazon Web Services account. If you do not have an Amazon Web Services account head over here to register. After you have registered, you must enable both the Amazon S3 service and the Amazon CloudFront service to your account (just follow the links and sign up).

With an AWS account and enabled both S3 and CloudFront to this account I was ready to start experimenting. Amazon Web Services is just services. This means that they only offers their services via APIs. Uploading a file to your S3 account must be done by a client. If you are a programmer it’s easy to just develop a client, but there are a lot of clients/programs around. An easy and simple client is S3Fox, a Firefox extension that that lets you upload files to Amazon S3 and manage the files’ access control. I use S3 Browser, a great application that handles the administration of my S3 files. Using S3Fox or S3 Browser makes it easy to test the S3 and CloudFront account, making sure it works. Objects (or files) are organized into buckets that can be created, listed, and retrieved using either a REST-style HTTP interface or a SOAP interface.

Update: As Andy says in a comment to this post (thanks Andy!), another client that you could try out is CloudBerry Explorer for Windows or S3HUB for Mac.

The first thing I had to do was to grab the AWS Access Key and the AWS Secret Access Key. Log in to the AWS account and click on ‘Security Credentials’ in the left menu. The first thing I did was to set up the S3 Browser account with this Access Key and Secret Key. I created my bucket and named it ‘static.larre.com‘, then uploaded some files to be sure everything worked:

Testing S3 bucket

Testing S3 bucket

Setting up CloudFront for my WordPress blog
Now I had an Amazon Web Services account, enabled Amazon S3, installed S3 Browser, created a bucket named ‘static.larre.com’, and uploaded files to this bucket. So far so good. Time to concentrate on the Amazon CloudFront and how to integrate it into my blog, that runs WordPress. There are several approaches that can be used. I have tried a couple of different approaches that I’ll try to describe.

Using Dreamhost
Besides hilarious monthly newsletters DreamHost is a great web hosting company. I use DreamHost for hosting of this blog. Last year DreamHost integrated CloudFront into their hosting service. A pretty amusing blog post from DreamHost about it is here. With CloudFront as a “Goodie-service” smoothly integrated in my hosting environment it is really simple to get started. The step is to create a cloudfront domain filling out this form:

CloudFront Domain

CloudFront Domain

What it does is to create a directory where you can put all files and directories that is going to be stored at S3 and distributed by CloudFront. A very simple admin view where you can handle simple tasks like delete, rescan etc is also part of this “goodies” service. When creating the CloudFront Domain, DreamHost creates a distribution for you at Amazon CloudFront. A distribution specifies the location of the original version of your objects. A distribution has a unique CloudFront.net domain name that you can use to reference your objects through the network of edge locations. If you wish, you can also map your own domain name to your distribution. DreamHost adds a CNAME record (in this case it was ‘static’), that has the cloudfront.net address as value. Using a CNAME enables me to use a domain name (in this case static.larre.com) instead of the domain name CloudFront provides. So looking at my own CloudFront distribution I have the following:

  • http://static.larre.com (my own mapped domain to the distribution)
  • http://d20o1nml9my7kd.cloudfront.net (the unique cloudfront.net domain)
  • http://static.larre.com.s3.amazonaws.com (the origin bucket at S3)

One of the most annoying thing about this service, is the lag from I request a rescan until my files are uploaded to S3. During the test period for the last couple of days it has taken everything from 10 minutes to several hours before DreamHost actually did rescan and upload/delete files in my S3 bucket. When DreamHost had finished scanning the directory and uploaded the files to S3, they where accessible. It seems that the only way to upload files is to do it manually within the DreamHost Admin Panel.

Using the service from Dreamhost may be a bit simple and manually copying all the needed files to the specified directory is not a very elegant solution. A more scalable solution is to make some scripts and jobs that pretty much does the work for you. Here is a blog post that describes one solution to make scripts and jobs for copying static files from Dreamhost to Cloudfront.

Adding a WordPress plugin
To work with my CloudFront Domain (http://static.larre.com), I tested a simple WordPress Plugin: ‘My CDN’. This plugin help you offloading javascript, css and theme files to your own CDN network, but only handle url rewriting not actual file transferring. Installing the plugin, activating it, and then adding three input fields under the settings section, was pretty much what I had to do (here is a screenshot of the settings page). One of the problems with this setup and solution, is that I have to manually put files in the specific directory (as described under the DreamHost section) unless I write a bunch of script-code. I think a better solution is to develop or find an already developed plugin that handles the prefixing of static content and also the upload to S3.

I’ve found one plugin: W3 Total Cache developed by Frederick Townes It’s a plugin improving the user experience of your blog by adding page caching, database caching, minify, content delivery network (CDN) functionality and more. Late last year they implemented integration with AWS and CloudFront. I have used the WP Super Cache until now, but it’s necessary to deactivate it before trying to install and activating the W3 Total Cache. Setting up this plugin is straight forward. After installing the plugin and activated it, I rolled over to the settings sections. First of all I had to activate the CDN and chose Amazon CloudFront as the CDN Type. Under the CDN settings I had to enter my Access Key, Secret Key, bucket and CNAME:

cdn settings

cdn settings

The plugin makes it easy to upload whatever static content you have on your WordPress blog to CloudFront. In the settings sections you specify extensions and files for upload to S3. During the testing period (a couple of days) I have found one issue with this plugin. I use WPtouch. It is a mobile theme, that automatically transforms your WordPress blog into a web-application experience when viewed from an iPhone, iPod touch, Android, or BlackBerry Storm touch mobile device. The default settings for W3 Total Cache captured the mobile devices and bypassed WPtouch. I solved it by adding mobile user agents in the corresponding field under the settings for page cache, minify and CDN.

As far as I can see, the DreamHost integration is not the solution for integrating my WordPress blog with Amazon CloudFront. With other needs than to get static WP content out to the edges it may be a nice functionality. But probably you will have to write some scripts and batch jobs working with the CloudFront directory before you get an elegant and effective solution. The W3 Total Cache plugin seems to be an easy and effective tool to use for mye goal. Setting up an AWS account, enabling S3 and CloudFront and adding a CNAME record is what you have to do before working with the plugin. Everything else seems to be achievable within the settings section.

Everything that is referenced with http://static.larre.com is now hosted by Amazon S3 and distributed out by Amazon CloudFront services.

  18 comments for “Amazon S3 and CloudFront with WordPress and DreamHost

  1. January 25, 2010 at 9:17 am

    Every Amazon S3 related blog gives you an impression that there is only one Amazon S3 client available – S3 fox. However there are many more freeware clients which are much superior. For instance, S3HUB for Mac (http://s3hub.com) and CloudBerry Explorer for Windows (http://s3.cloudberrylab.com/) .

  2. February 1, 2010 at 2:03 am

    Hi there.
    Thank you for a great post. It was very helpfull.
    Anyone reading this post should bookmark this guys contents.

    I have a new PC and needed some installation help so i went over to http://www.InstallSoftware.com but they did not provide me with the in depth
    info this guy did. he kicks all the bigger sites’ butts.

    Thanks Again

  3. February 18, 2010 at 5:42 am

    I came across this post when looking to troubleshoot an issue with WPTouch and W3 Total Cache. Kind of made me chuckle since I just did all of this myself on my blog last weekend…only difference is I don’t use Dreamhost, but that’s a minor difference.The cool thing, timing-wise, is that I randomly implemented all this caching and CDN stuff just in time! On Monday, my blog which usually gets about 100-200 visitors on a good day, received over 20,000 visitors from Reddit. If I hadn’t put all the fun CloudFront stuff in place, my server would have melted down from the load, I’m sure. Honestly, if you can offload all the images, stylesheets, and javascripts to the CDN, the straight HTML coming from your server is going to be pretty mild. I pushed over 8GB of CDN stuff this week, and that would have killed my server…but the static pages were nothing. Especially once you include the caching plugin to offset the database calls (caching worked GREAT in my case, as almost all the traffic was to the same post/page).Oh, and my Amazon bill for all of that? One dollar and seventy-six cents :)Here’s a bit more of the story – http://mattstratton.com/meta/tick-tick-boom

  4. February 18, 2010 at 10:16 pm

    Hi Matt!
    Thanks for your comment. I think more people should try out cloud services like aws. The sudden peak in traffic from reddit, digg etc is a problem. I have just experienced it a couple of times, but used Dreamhost’s services for enabling more memory and cpu on the fly. But it is actually pretty expensive. Using caching and CDN is a great solution for handling the peaks without going bankruptcy, and still have a decent response time ;-)

  5. May 15, 2010 at 5:45 am

    Great review. I am also testing Amazon S3, Cloudfront and W3 Total Cache on my website. It works great with a couple of issues:
    1. In my website I use @font-face in the original theme style.css in order to use custom fonts (Hebrew) and, unfortunately, Firefox refuses to download the font files if they are hosted on S3, therefore I should add an exception to the configuration of the plugin (exclude style.css of my theme) but I do not consider this problem as a bug.
    2. This one is definitely a bug of W3 Total Cache and I hope they will solve it soon. If you upload the Media library on S3, the original size files are not recognized and not transferred. If you use a lightbox-like effect, as I am, this issue is quite painful. An option could be to re-add the uploads as a custom directory but I think developers should address this problem.

    I have a question for you: how about auto-setting ACL’s on S3? I have googled for it but could not find any answer. Any help?

  6. May 15, 2010 at 9:46 pm

    Hi etzhaim, thanks for your comment!

    I have not having any problems with uploading the original file (size) to S3. This is done autmatically by the W3 Total Cache. If you look at my post: http://www.larre.com/2010/04/19/nypd-helicopter-on-top-of-empire-state-building/ you will see that all the photos has the original file uploaded (try to click one of the photos). What happens if you initiate a manual upload from the plugins settings page?

    I found the ACL a bit confusing. When I created a bucket trough Dreamhost the bucket did not have the correct ACL but as far as I can remember creating it through the clients I tested actually did.

    I have not found any settings on aws about auto-setting ACL´s.

  7. August 3, 2010 at 8:10 am

    Hi Catherine!
    Thanks for your mention!

    It still takes a couple of seconds to load my site but that has to do with other factors (some of my plugins).

  8. August 12, 2010 at 10:01 am

    I think this blog post was probably a solid beginning to a potential series of blog posts about this topic. Most users act like they know what they are preaching about when it comes to this area and most of the time, hardly anyone actually get it. You seem to grasp it however, so I think you need to run with it. Thanks!

  9. Pingback: Weekly linkage
  10. Roger Buckley
    October 12, 2010 at 4:42 am

    Hi Eyvind!

    Great article on a subject more and more of us are concerned!
    Still it seems no one talked about the fact that CDN in WordPress only works with the WP Media Library.

    WordPress Media Library is really lacking of options to sort and organize images. If you have a lot of pics, there is no comparison with the traditional system using FTP and separate folders.
    W3 Total Cache plugin has an option allowing re-import of images in WordPress and automatic adjusting of posts/pages link code.

    This is very nice except if you really preferred to keep your own way of organizing pics in separate folders.
    Or is there a way to upload manually my static images?

    Any help/feedback is much appreciated!

    THANKS!
    Roger

  11. October 14, 2010 at 10:06 pm

    Hi Roger and thanks for stopping by!
    I am sorry for the late respons (I’ve been on holiday).

    I have a pretty limited use of the media library so the topic you raise has not been a problem for me. I just use the Media Library as is and with small volume. Regarding the W3 Total Cache, you could send en email to the guy that has made the plugin: http://twitter.com/w3edge

    I am not sure but it could be that the only way to solve your problem (today) is to make your own scripts (or plugin) for copying/uploading files to Cloudfront (if there is no existing plugin that goes beyond the Media Library)

  12. April 12, 2011 at 11:17 pm

    Genuinely helpful post. Many thanks. I’m on Dreamhost and having performance issues so wanted to learn how to do this.

  13. Wordpress Business Themes
    January 5, 2012 at 1:51 am

    Very helpful post. Thank you! I hadn’t even considered using Amazon Cloudfront – didn’t realize you can use it with your current hosting company.

  14. June 3, 2013 at 7:32 am

    Now that DreamHost offers it’s own cloud solution, are you still using Amazon S3 for your cloud need?

  15. July 28, 2013 at 8:17 pm

    Hi Dhruv, and thanks for the comment. I have not looked into the topic after I wrote the post. Will do shortly – and then check out the Dreamhost cloud solution!

Leave a Reply

Your email address will not be published. Required fields are marked *