Nested folders in Cloud Files

Cloud storages systems like Rackspace’s Cloud Files and Amazon’s S3 are great for storing large amounts of information. A common misconception is that these storage systems behave like traditional file systems, complete with byte-level manipulation and nested folders. It is the second of these that I want to talk about: how to simulate a nested directory (or folder) structure in Rackspace’s Cloud Files.

Cloud Files and S3 are better understood as storage systems, not file systems. Each have three basic parts: accounts, containers (buckets in S3), and objects. In Cloud Files, these three parts can be easily seen in the URL referencing an object. The URL one uses for the ReST API is of the form http://example.clouddrive.com/account/container/object. Containers are large-scale groupings of objects, operating at a higher level, conceptually, than folders. If objects were books, containers may be genres. Containers cannot be nested. That is, one cannot put a container inside of another container.

However, it is fairly easy to simulate a directory structure with objects. These “virtual directories” are not directories, per se, but object name prefixes over which one can iterate. An example should make this concept easy to understand. Suppose I wanted to store books in Cloud Files. From my analogy above, I can use the genre of the book as my container name. The object name will be of the form “author/title”. This way, I can list all books by a particular author (within a genre).

Let’s load the following books into Cloud Files:

  • The Pit and the Pendulum, Poe, Horror
  • The Masque of the Red Death, Poe, Horror
  • Pride and Prejudice and Zombies, Grahame-Smith, Horror
  • The Far Side Gallery, Larson, Comics
  • Something Under the Bed Is Drooling, Watterson, Comics
  • It’s A Magical World, Watterson, Comics

First, I will create two containers, horror and comics. Next I will name my files according to the pattern I laid out above. I will have the files “poe/the_pit_and_the_pendulum”, “poe/the_masque_of_the_red_death”, “larson/the_far_side_gallery”, etc. Then I will upload these files to their appropriate container. As a final step, I need to upload “directory marker” files. These are empty (zero-sized) files with a content-type of “application/directory”.

[The following gets technical. For those wishing to use this feature of Cloud Files and not wanting to program, I recommend using a third-party tool like Cyberduck (if you are using a Mac). Cyberduck handles virtual nested directories completely transparently.]

Now to take advantage of these “virtual directories”, I can do container listings and give an appropriate path value. In the Python language bindings, this would look similar to the following:

1
2
container = cf_connection.get_container('horror')
books_by_poe = container.get_objects(path='poe')

The path parameter on the get_objects call returns all objects in the given value. In this case, it returns the two books in the virtual “poe” directory. Similarly, if I had given the value “grahame-smith”, I would have found his adaptation of the classic love story.

In my example, I’ve used two genre containers and virtual directories only one level deep. I could just as easily put everything into one container and nested the authors under a genre virtual directory. An object name would then be like “comics/larson/the_far_side_gallery”. The only limitation to using this feature in Cloud Files is keeping the length of the object name (including all virtual directories) under the maximum allowed (1024 characters).

For more detailed information on how to implement virtual directories, see the Cloud Files developer guide. The relevant information is found in the “Pseudo hierarchical folders/directories” section.

Quickly uploading data to Cloud Files

Cloud Files is a great way to store information, either to take advantage of the CDN or to offload the infrastructure requirements of storing large amounts of data. However Cloud Files is used, though, one still must upload the data to the service before being able to use it.

Uploading the data is not problematic if it can be done in small chunks or spread out over time (images on a blog, for example). The Cloud Files language APIs offer a good way to upload data in these cases. Unfortunately, the language bindings can be terribly slow for uploading large numbers of files. While they do make some optimizations (like reusing connections when available), the code is written to be very generic. For example, the bindings make HEAD requests to ensure all proper data is set before allowing you to upload an object. While this is good in a general sense, these HEAD requests become superfluous when doing a large batch upload. One can achieve much better results by using the Cloud FIles ReST API directly.

As an example, let’s look at the following code which uses the Python API:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#!/usr/bin/env python
 
import os
import cloudfiles
 
username = 'xxxx'
apikey = 'xxxx'
 
conn = cloudfiles.get_connection(username, apikey)
 
container = conn.create_container('api_speed_test3')
data_list = ('test_data/%s'%x for x in os.listdir('test_data') if x.endswith('.dat'))
for filename in data_list:
    try:
        obj = container.create_object(filename)
        obj.load_from_filename(filename)
    except cloudfiles.errors.ResponseError, err:
        print err
print len(container.list_objects())

In my tests, using the above code takes about 5.5 minutes to upload 1000 16KB files to Cloud Files.

I wrote the same functionality using the ReST API directly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#!/usr/bin/python
 
import os
import httplib
 
username = 'xxxx'
apikey = 'xxxx'
 
# auth
conn = httplib.HTTPSConnection('api.mosso.com')
conn.request('GET', '/auth', headers={'x-auth-user': username, 'x-auth-key': apikey})
resp = conn.getresponse()
auth_token = resp.getheader('x-auth-token')
url = resp.getheader('x-storage-url')
conn.close()
# send data
send_headers = {'X-Auth-Token': auth_token, 'Content-Type': 'text/plain'}
container_path = '/'+'/'.join(url.split('/')[3:])+'/api_speed_test2'
conn = httplib.HTTPSConnection(url.split('/')[2])
conn.request('PUT', container_path, headers=send_headers)
conn.getresponse().read()
data_list = ('test_data/%s'%x for x in os.listdir('test_data') if x.endswith('.dat'))
for filename in data_list:
    f = open(filename)
    conn.request('PUT', container_path+'/'+filename, body=f, headers=send_headers)
    f.close()
    resp = conn.getresponse()
    resp.read()
    if resp.status >= 300:
        print resp.status, resp.reason, container_path+'/'+filename
conn.close()

Although slightly longer, the majority of the extra code is for the auth. In my tests, uploading 1000 16KB files took about 4.5 minutes. A whole minute improvement for only 1000 objects is a very good result. I would expect the difference to be even greater as the number of files increases.

All of the code above (plus code to generate the test data) can be found in my github account.

By using the ReST API directly, I can make certain assumptions about my data that are not possible in the generic language bindings. I do not need to do the HEAD requests because I know I have just created the container and I have not uploaded the files yet. I am explicitly setting all the data for each object upload. Further improvements would be to add some error handling and parallelization.

Cloud Files Object Copy

Cloud Files does not currently support object copying. However, a simple workaround is to re-upload the file with the new name. Implementing this workaround may be inconvenient, and one may miss some things like ensuring that metadata is updated. I have added a copy feature to my fork of the python-cloudfiles API that takes care of these details. This is a convenience function only and is not officially supported by Rackspace. Keep in mind that billable bandwidth will be used (unless the servicenet flag is set in the API). One option for renaming large files is to spin up a small Cloud server, use the API to copy over servicenet, and spin down the server. At $0.015 per hour, one could run a 256MB instance for 100 hours before equalling the transfer cost for copying one 5GB (Cloud Files max size) file over the billed network.

My python-cloudfiles fork on github: python-cloudfiles

Example script that copies the last file in a container to another container:

1
2
3
4
5
6
7
8
9
10
11
12
13
import cloudfiles
conn = cloudfiles.get_connection(username='myname', api_key='mykey')
container_name = 'example_container'
another_container = 'example_container2'
c = conn.get_container(container_name)
l = c.list_objects()
o = c.get_object(l[-1])
new_path = '%s/%s' % (another_container, o.name)
o.copy_to(new_path)
print 'copied', l[-1], 'to', new_path
new_list = conn.get_container(another_container).list_objects()
print new_list
assert o.name in new_list

When building a starship…

I’ve been watching Battlestar Galactica again, and I was struck by something that seems to be a very common plot device in space-based stories: Starships of all shapes and sizes are able to be reconfigured in seemingly infinite ways on the fly by the crew. In BSG, the crew networks disparate systems together to create a large compute cluster and implement a multi-layered firewall to protect against the Cylon virus. In Star Trek, the crew must constantly adjust the phasers by rerouting it through the main sensor dish or change the frequency of the shields to defeat some advanced enemy. In Star Wars, Han can easily reroute power to strengthen the shields.

Starships must be large and very complex systems. They have thousands of subsystems. Computer programs controlling doors, weapons, replicators, power, life support, sensors, water recycling, artificial gravity, and many other systems need to be written and tested before being put in to use in an operating starship. Designing and implementing this much code that is responsible for the life and death of all aboard is an impossible task for one company.

Now, for all these starships to be constantly reconfigured on the fly, either the requirement specs given to the developers describe all possible scenarios that the crew of the ship may face or must detail a standard API that all subsystems should adhere to.

There is no way one can predict all situations in which a piece of software may be used. All of these subsystems must have a standard interface that allows them to all interact with one another. Because you never know–you may need to reroute the water filtration through the warp core to kill the trans-dimentional virus that is infecting the crew.

But simply having a standard API isn’t enough. Starship crews need to be able to write programs themselves in order to tell these separate systems to work together in some custom way. Sure, their may be some large patch panel of sorts to redirect some output to some other input, but at some point, Wesley will need to do something that the system wasn’t designed to do. And, of course, bug fixes and patches will need to be applied periodically.

Given these realities, I’ve come to the conclusion that starships must run open-source software. The crew needs access to the source code to make changes as necessary. Disparate systems need to be modified to work together. Crew members (and captains, apparently) need to know exactly how these systems work. Proprietary software would probably not include the source code in the installation. Even if the code were included, it would most likely have so many disclaimers and warranty-voiding clauses in the EULA (or worse, DRM) to make any such modifications useless or impossible for the end-user.

Imagine Galactica docking and being told, “I’m sorry. You have modified your installation of the software, and we can’t upgrade you to the Cylon-proof Weapons Pro v12 unless you revert your changes.”

Or maybe Han is getting pulled in by a tractor beam (as he is wont to do). He engages the auxiliary power, and, “Thank you for using Power Systems 3.0. A new patch is available for this software. Would you like to download it now? Answering ‘No’ will disable the auxiliary power systems.”

The only way for all the complicated interconnected systems of a starship to be useful, reliable, and developed in an efficient manner is for it to be open source.

When building a starship, use open source software.

Use open source software–it’s the future.

Use open source software–don’t let the Borg/Cylons/Empire win.

South with the Sun to San Antonio

It’s official. I have accepted the formal offer to work at Rackspace. I will be working in the Cloud Files division in beautiful downtown San Antonio. This next month will be full of transition: leaving one job, moving, and starting a new job.

Update (08/07/09): We’ve moved. Everything went smoothly. The moving company was great. Our new washer and dryer were delivered today. We are in the process of unpacking a lot of boxes and learning our way around this new town. I’m looking forward to starting the new job on Monday.

PyGTK Chart widget beta release

We released a new version of pygtkChart today. This version is a beta release and allows for much more flexibility than the previous version. Some new features include the ability to independently address each part of a chart or graph and the ability to use GTK properties and signals. Mouse events are now supported, and hooks are available to click on individual areas of a chart.

The new version can be downloaded from http://github.com/notmyname/pygtkChart/downloads. As always, the latest source can be cloned from git://github.com/notmyname/pygtkChart.git.

OpenSolaris upgrade

OpenSolaris 2009.06 was released this month. Upgrading my home file server was pretty easy.

# pkg refresh && pkg image-update

Everything worked smoothly until the last part. The new boot environment did not activate correctly. Some further digging revealed that one of the two disks in my rpool was set with and EFI disk label (I’m not sure how that happened). I found some good information online, and soon enough, I was up and running with the new version of OpenSolaris.

ZFS was upgraded in this new version, so I upgraded it on my file server.

# zfs update

And like that, I was done. I will be interested to see if the drivers (rge) for my original NIC are better. If so, I may be able to set up trunking on the two NICs to double my throughput. I did notice that CIFS seems to work a little better. That is, I didn’t have to coerce the machine to share via CIFS like I used to.

PyGTK Chart Widget

pygtkChart is a chart widget for GTK that offers line graphs and pie charts. It’s simple to use, but it is lacking one feature that I really wanted: bar charts. I added a bar chart widget to the package, but I have not been able to get in touch with the original author to contribute the code back. So, here it is.

Download: Clone from git://github.com/notmyname/pygtkChart.git or view the source at http://github.com/notmyname/pygtkChart/tree

Installation: $ python setup.py build && sudo python setup.py install

Description: I have added two new classes: BarChart and MultiBarChart. BarChart provides a simple bar chart. MultiBarChart allows for grouped bars. The code is fairly well commented and should be easy to follow.

BarChart example

BarChart example


MultiBarChart example

MultiBarChart example

These images are screenshots of bar_chart_test.py and multi_bar_chart_test.py, both found in bar_chart_test.tgz

UPDATE: a new version has been released

Useful Tools

I use several online tools that make my life easier, and I’d like to share them in the hope that they make someone else’s life easier.

Readability
Readability is a bookmarklet that reformats your current page and makes it readable. It’s a wonderful tool in the world of web pages full of ads and useless junk.
Mint.com
Mint.com is a free online service to keep up with your finances. It offers a quick way to see where all of your money goes and shows where you can save money of get better deals (on bank accounts, credit cards, etc).
Nike+
Nike+ is a two-part service that allows you to keep track of running or walking with ease. The first part is the accelerometer you put in your shoe that connects to the receiver plugged in to an iPod. This records the run and offers real-time feedback during the workout. The second part is the website that keeps a history of your runs and allows you to compare and compete with the millions of other Nike+ users. The only non-free service on this list, the Nike+ will set you back $30 (assuming you already have an iPod nano or touch).
Print What You Like
Another free bookmarklet, print what you like allows you to selectively print sections of webpages without all of the cruft. Although I don’t print web pages very often, this tool is very helpful when I do.

Friday fun at work, part 2

I finished something up at work today that I have been working on part-time this week. My boss really likes it, and I’m pretty happy with how it turned out. I call it attachment scrubbing.

Last week, a friend and I were talking over lunch about some email woes our company was having. Specifically, some people were emailing around some large attachments, and they were putting a lot of strain on our mail server. By the time we finished eating, we had come up with the idea of putting something in place on the mail server to intercept attachments from emails. A user can send an attachment as usual, but when it gets to the email server, my program grabs it. I remove the attachment from the email, upload it to a web server, and put a message with a link to the attachment back in the email. When the recipient gets it, their email program won’t choke on some large file, and our servers are happier because we will only be storing one copy of the attachment for each message.

The real beauty of this system is that the user sending the email doesn’t have to do anything differently. And although the idea didn’t come from a desire to make external users happier, it can make their email experience more pleasant by not clogging up their inbox.

This was pretty fun to work on for the past week and a half, and it provided an interesting diversion from some mundane system administration I’ve had to do lately. I now have a lot more understanding of how my company’s email system is set up, and I think I could set up a similar system pretty easily. Email servers are pretty complicated, but there is nothing like installing your own and customizing it that will teach you how it works.