Amazon web services

Let’s start with Amazon CloudFront and what this actually is. CloudFront is a Content Distribution Network (CDN). The service distributes your data out to multiple datacenter locations, ensuring faster access times. It works together with another service from Amazon S3 (Simple Storage Service). Amazon S3 is a data storage service where your files is stored initially. To be using CloudFront for distribution you need to have a S3 account and store the files there.

Setting up an Amazon Web Services account
As I have blogged about before, I use Amazon S3 already, so it was not necessary for me to create an Amazon Web Services account. If you do not have an Amazon Web Services account head over here to register. After you have registered, you must enable both the Amazon S3 service and the Amazon CloudFront service to your account (just follow the links and sign up).

With an AWS account and enabled both S3 and CloudFront to this account I was ready to start experimenting. Amazon Web Services is just services. This means that they only offers their services via APIs. Uploading a file to your S3 account must be done by a client. If you are a programmer it’s easy to just develop a client, but there are a lot of clients/programs around. An easy and simple client is S3Fox, a Firefox extension that that lets you upload files to Amazon S3 and manage the files’ access control. I use S3 Browser, a great application that handles the administration of my S3 files. Using S3Fox or S3 Browser makes it easy to test the S3 and CloudFront account, making sure it works. Objects (or files) are organized into buckets that can be created, listed, and retrieved using either a REST-style HTTP interface or a SOAP interface.

The first thing I had to do was to grab the AWS Access Key and the AWS Secret Access Key. Log in to the AWS account and click on ‘Security Credentials’ in the left menu. The first thing I did was to set up the S3 Browser account with this Access Key and Secret Key. I created my bucket and named it ‘static.larre.com‘, then uploaded some files to be sure everything worked:

Testing S3 bucket

Setting up CloudFront for my WordPress blog
Now I had an Amazon Web Services account, enabled Amazon S3, installed S3 Browser, created a bucket named ’static.larre.com’, and uploaded files to this bucket. So far so good. Time to concentrate on the Amazon CloudFront and how to integrate it into my blog, that runs WordPress. There are several approaches that can be used. I have tried a couple of different approaches that I’ll try to describe.

Using Dreamhost
Besides hilarious monthly newsletters DreamHost is a great web hosting company. I use DreamHost for hosting of this blog. Last year DreamHost integrated CloudFront into their hosting service. A pretty amusing blog post from DreamHost about it is here. With CloudFront as a “Goodie-service” smoothly integrated in my hosting environment it is really simple to get started. The step is to create a cloudfront domain filling out this form:

CloudFront Domain

What it does is to create a directory where you can put all files and directories that is going to be stored at S3 and distributed by CloudFront. A very simple admin view where you can handle simple tasks like delete, rescan etc is also part of this “goodies” service. When creating the CloudFront Domain, DreamHost creates a distribution for you at Amazon CloudFront. A distribution specifies the location of the original version of your objects. A distribution has a unique CloudFront.net domain name that you can use to reference your objects through the network of edge locations. If you wish, you can also map your own domain name to your distribution. DreamHost adds a CNAME record (in this case it was ’static’), that has the cloudfront.net address as value. Using a CNAME enables me to use a domain name (in this case static.larre.com) instead of the domain name CloudFront provides. So looking at my own CloudFront distribution I have the following:

• http://static.larre.com (my own mapped domain to the distribution)
• http://d20o1nml9my7kd.cloudfront.net (the unique cloudfront.net domain)
• http://static.larre.com.s3.amazonaws.com (the origin bucket at S3)

One of the most annoying thing about this service, is the lag from I request a rescan until my files are uploaded to S3. During the test period for the last couple of days it has taken everything from 10 minutes to several hours before DreamHost actually did rescan and upload/delete files in my S3 bucket. When DreamHost had finished scanning the directory and uploaded the files to S3, they where accessible. It seems that the only way to upload files is to do it manually within the DreamHost Admin Panel.

Using the service from Dreamhost may be a bit simple and manually copying all the needed files to the specified directory is not a very elegant solution. A more scalable solution is to make some scripts and jobs that pretty much does the work for you. Here is a blog post that describes one solution to make scripts and jobs for copying static files from Dreamhost to Cloudfront.

Adding a Wordpress plugin
To work with my CloudFront Domain (http://static.larre.com), I tested a simple Wordpress Plugin: ‘My CDN’. This plugin help you offloading javascript, css and theme files to your own CDN network, but only handle url rewriting not actual file transferring. Installing the plugin, activating it, and then adding three input fields under the settings section, was pretty much what I had to do (here is a screenshot of the settings page). One of the problems with this setup and solution, is that I have to manually put files in the specific directory (as described under the DreamHost section) unless I write a bunch of script-code. I think a better solution is to develop or find an already developed plugin that handles the prefixing of static content and also the upload to S3.

I’ve found one plugin: W3 Total Cache developed by Frederick Townes It’s a plugin improving the user experience of your blog by adding page caching, database caching, minify, content delivery network (CDN) functionality and more. Late last year they implemented integration with AWS and CloudFront. I have used the WP Super Cache until now, but it’s necessary to deactivate it before trying to install and activating the W3 Total Cache. Setting up this plugin is straight forward. After installing the plugin and activated it, I rolled over to the settings sections. First of all I had to activate the CDN and chose Amazon CloudFront as the CDN Type. Under the CDN settings I had to enter my Access Key, Secret Key, bucket and CNAME:

cdn settings

The plugin makes it easy to upload whatever static content you have on your WordPress blog to CloudFront. In the settings sections you specify extensions and files for upload to S3. During the testing period (a couple of days) I have found one issue with this plugin. I use WPtouch. It is a mobile theme, that automatically transforms your WordPress blog into a web-application experience when viewed from an iPhone, iPod touch, Android, or BlackBerry Storm touch mobile device. The default settings for W3 Total Cache captured the mobile devices and bypassed WPtouch. I solved it by adding mobile user agents in the corresponding field under the settings for page cache, minify and CDN.

As far as I can see, the DreamHost integration is not the solution for integrating my Wordpress blog with Amazon CloudFront. With other needs than to get static WP content out to the edges it may be a nice functionality. But probably you will have to write some scripts and batch jobs working with the CloudFront directory before you get an elegant and effective solution. The W3 Total Cache plugin seems to be an easy and effective tool to use for mye goal. Setting up an AWS account, enabling S3 and CloudFront and adding a CNAME record is what you have to do before working with the plugin. Everything else seems to be achievable within the settings section.

Everything that is referenced with http://static.larre.com is now hosted by Amazon S3 and distributed out by Amazon CloudFront services.

What is Amazon S3, and Jungle Disk
Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. I use it to backup all my digital contents (music, photo, video). Amazon S3 does not offer any type of clients to typically access the service just a set of API’s. I first started out with implementing my own client but found out that writing my own client is just too geeky. I use Jungle Disk as a client to access my S3 account. This combination works perfect and I’m really happy with both S3 and Jungle Disk.

The problem with Content.IE5
I have not set up automatic backup but make a backup manually by copying folders from my network drive and over to my S3 account. Jungle Disk maps a drive to my Amazon S3 account for seamless integration. And here is where the problem or issue comes into action. What happens is that for each file that I copy from my local storage (read network drive) to my S3 area the file is temporarily stored at my local PC at:

C:\Documents and Settings\LocalService\Local Settings\Temporary Internet Files\Content.IE5

This folder is actually a hidden one (system folder) and not easy accessible for non-tech persons. I found out about this because my local drive almost got out of free space. This hidden temporarily folder was eating up my local Gigabytes.

The solution to the problem with Content.IE5
The solution to this problem is of course to delete the directories and files that are stored in this temporarily directory. This can be done manually or you could write a batch file that is executed after you have backed up files. This is what I have done.

Another solution is looking more into what is causing the local storage (caching). The problem is that Jungle Disk is running in the environment of the Local Service user. The files will be stored until the Local Service user reaches the cache limit. Her is a nice thread describing some of this issues. The solution is to ensure a cache limit that is small (or not extremely large) so that the system will clean up nicely. If we look at the solution described in the thread it describes where to set this limit in the registry. Using regedit I checked what the CacheLimit on my PC was: 256 MB. This is way lower than the 15 GB I had to delete the first time I found out of the the Content.IE5 folder issue.

I do not think this solution will work. I think that the Local Service is running under some other user (WindowsNT user?). But this is just a wild guess. Another issue is that to change the registry for the Local Service all processes by that user must be shut down. This could be a bit tricky, but most of all, it’s time consuming trying to find out what will work or not.

So I still run a manual clean of the Content.IE5 folder…

How well is the user interface when the user has 55 apps (as I have at the time this post is written)?. There are some factors to take into account:

• Native iPhone apps
• Third party apps
• Your own apps (could go as third party)
• Native apps in regular use
• Third party apps in regular use
• Third party apps for testing purpose
• Different categories like games, travel, office etc
• … (and more)

My experience is that if you put 16 applications in one page it’s a mess everytime you are looking for one; It’s impossible to find it. And if the application is not on the front page… where is it? On page 4…5? By using some of the factores above I have organized my iPhone to some extent, but sure would like to see Apple develop some great and innovative user interface that will work better than it does today. After organizing all my applications I found my self often slide over to the last page. Suddenly I was using the games more regular than first thought. If you know how to move an entire page at a time, please let me know (without using the jailbreaked Categories app). This is actually an extremely time-consuming process.

So, here is how I organized my iPhone:

1. What I did was to put Native and third party applications that I use often on my front page limiting them to 12.
2. On my second page I put all the native and third party apps that I use by regular basis.
3. On my third page is all my applications that supports me when I’m out traveling.
4. On my forth page is all my games.
5. On my fifth page I have the applications rarely in use
6. And the last one is for temporarily testing purpose.

UPDATE:Today I came over this nice post that is digging more into the issues in this post. Also.. 9 pages maximum?? I still have 2 pages until I will have trouble: http://venturebeat.com/2008/12/21/the-need-for-iphone-app-folders-and-search-is-quickly-approaching/

The first thing I did was to look into the information at Apple’s own site to see if they had written anything about this issue. And yes, they have. a firmware update should solve the issue (firmware update)… BUT it did not.

So I had to search the net to look for help. Whow, it seems that a lot of people is experiencing this problem. There is a lot of fancy and odd solutions and I tried everything I ever came over:

• Firmware Update
• Changing wlan channel
• Using network robustness
• Moving microwaves and wireless phones around
• Upgraded the network from 802.11b to 802.11g
• Using multiple speakers
• and more…

Here are some links:

• Apples list of actions
• 802.11b/802.11g
• IP phone, Microwave
• Multiple speakers

Nothing helped. So I then had to figure out what actually happend with my computer during the drop out. I looked at 2 things; Network traffic and CPU usage. What I found out whas that the network traffic was ok, nothing odd with it. But, when I looked at CPU usage, I found one correlation with the drop out. Whenever my computer was working or had a CPU usage that was more than typically a couple of %, I got a drop out.

So, what did I do?

When streaming music to my AirPort Express I shut down all other applications, typically MSN, Firefox, Word etc. This solved the problem. No drop outs.

My conclusion is that there is no problem with the AirPort Express, the problem is related to the client that streams.  AirPort Express drop out is not a problem for me anymore I just have to focus on the music not doing a lot of things at the same time

See also: Reflections after having implemented Facebook Connect and Google Friend Connect on my blog

First of all my goal was to use an existing web application, write a facebook integration module that communicates with the Facebook APIs, extracting some information/data and presenting them inside the existing web application. The web application I used is based on a j2ee framework (jsp, java). The architecture is much like Struts The first thing to do was to get the Developer Application to my existing Facebook account. Then I had to download the Client Library. I choosed the Java Library since my development environment was Java. When I extracted the Client Library, only the source files where included. So what I had to do first (besides looking into the source code), was to compile the java-files. I actually wrote an ant-task for this purpose, but a more easier way is just to use javac without using ant-framework. The next step was to take the output from the compile process and put it into an archived file (.jar). This is not necessary but I like to do it this way and the facebook.jar file I now had is easily dropped into my existing web application library. In my Facebook profile I had to create (set up) a new application. This is done by clicking the developer application that was installed earlier. Its some steps including getting a key (API key and secret key) that I used in my java-code but this process is straight forward.

Ok, now lets start the coding process. Here is the java code controlling the session and authentication:

String apiKey = “something”;
String secretKey = “something”;

FacebookRestClient frc = null;
HttpSession session = request.getSession();
String sessionKey = (String) session.getAttribute(”facebookSession”);
String token = request.getParameter(”auth_token”);

try {

if (sessionKey != null && sessionKey.length() > 0) {

frc = new FacebookRestClient(apiKey, secretKey, sessionKey);
this.doTheThing(request, response, frc);
} else if (token != null) {
frc = new FacebookRestClient(apiKey, secretKey);
session.setAttribute(”facebookSession”, sessionKey);
sessionKey = frc.auth_getSession(token);
session.setAttribute(”facebookSession”, sessionKey);
this.doTheThing(request, response, frc);

} else {
response.sendRedirect(”http://www.facebook.com/login.php?api_key=” + apiKey + “&v=1.0″);
}
} catch (FacebookException fe) {
} catch (IOException ioe) {
…

The doTheThing method is extracting the information and looks like this:

int myid = frc.users_getLoggedInUser();
EnumSet<ProfileField> fields = EnumSet.of(com.facebook.api.ProfileField.NAME, com.facebook.api.ProfileField.PIC, com.facebook.api.ProfileField.PIC_BIG, com.facebook.api.ProfileField.PIC_SMALL);
Collection<Integer> users = new ArrayList();
users.add(myid);

// Get my information
Document d = frc.users_getInfo(users, fields);
String myname = d.getElementsByTagName(”name”).item(0).getTextContent();
String mypicture = d.getElementsByTagName(”pic”).item(0).getTextContent();

// Get my friends id
Document d2 = frc.friends_get();
String s = d2.toString();
NodeList userIDNodes = d2.getElementsByTagName(”uid”);
int fcount = userIDNodes.getLength();

Collection<Integer> friends = new ArrayList<Integer>();
for (int i = 0; i < fcount; i++) {
Node node = userIDNodes.item(i);
String idText = node.getTextContent();
Integer id = Integer.valueOf(idText);
friends.add(id);
}

List l = new ArrayList();
Map m = new HashMap();
Document d3 = frc.users_getInfo(friends, fields);

// Get my friends information
for (int j = 0; j < fcount; j++) {
String name2 = d3.getElementsByTagName(”name”).item(j).getTextContent();
String picture2 = d3.getElementsByTagName(”pic”).item(j).getTextContent();
String picture3 = d3.getElementsByTagName(”pic_small”).item(j).getTextContent();
String picture4 = d3.getElementsByTagName(”pic_big”).item(j).getTextContent();

m.put(”name”, name2);
m.put(”picture”, picture2);
m.put(”picture_small”, picture3);
m.put(”picture_big”, picture4);

l .add(m);
m = new HashMap();

}
request.setAttribute(”friends”, l);
request.setAttribute(”myname”, myname);
request.setAttribute(”mypicture”, mypicture);
request.setAttribute(”numFriends”, new Integer(fcount));

So all the information from Facebook is now added to the request and can be displayed in a JSP. The code above is not very elegant and is not optimized(!). Its just a pice of code done after half an hour of work and my goal was to implement an integration module for testing purpose.

Hope this blog post can help others looking for hints when getting started with Facebook application development/integration.