Posts tagged cloud files

Cloud Files is a great way to store information, either to take advantage of the CDN or to offload the infrastructure requirements of storing large amounts of data. However Cloud Files is used, though, one still must upload the data to the service before being able to use it.

Uploading the data is not problematic if it can be done in small chunks or spread out over time (images on a blog, for example). The Cloud Files language APIs offer a good way to upload data in these cases. Unfortunately, the language bindings can be terribly slow for uploading large numbers of files. While they do make some optimizations (like reusing connections when available), the code is written to be very generic. For example, the bindings make HEAD requests to ensure all proper data is set before allowing you to upload an object. While this is good in a general sense, these HEAD requests become superfluous when doing a large batch upload. One can achieve much better results by using the Cloud FIles ReST API directly.

As an example, let’s look at the following code which uses the Python API:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#!/usr/bin/env python
 
import os
import cloudfiles
 
username = 'xxxx'
apikey = 'xxxx'
 
conn = cloudfiles.get_connection(username, apikey)
 
container = conn.create_container('api_speed_test3')
data_list = ('test_data/%s'%x for x in os.listdir('test_data') if x.endswith('.dat'))
for filename in data_list:
    try:
        obj = container.create_object(filename)
        obj.load_from_filename(filename)
    except cloudfiles.errors.ResponseError, err:
        print err
print len(container.list_objects())

In my tests, using the above code takes about 5.5 minutes to upload 1000 16KB files to Cloud Files.

I wrote the same functionality using the ReST API directly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#!/usr/bin/python
 
import os
import httplib
 
username = 'xxxx'
apikey = 'xxxx'
 
# auth
conn = httplib.HTTPSConnection('api.mosso.com')
conn.request('GET', '/auth', headers={'x-auth-user': username, 'x-auth-key': apikey})
resp = conn.getresponse()
auth_token = resp.getheader('x-auth-token')
url = resp.getheader('x-storage-url')
conn.close()
# send data
send_headers = {'X-Auth-Token': auth_token, 'Content-Type': 'text/plain'}
container_path = '/'+'/'.join(url.split('/')[3:])+'/api_speed_test2'
conn = httplib.HTTPSConnection(url.split('/')[2])
conn.request('PUT', container_path, headers=send_headers)
conn.getresponse().read()
data_list = ('test_data/%s'%x for x in os.listdir('test_data') if x.endswith('.dat'))
for filename in data_list:
    f = open(filename)
    conn.request('PUT', container_path+'/'+filename, body=f, headers=send_headers)
    f.close()
    resp = conn.getresponse()
    resp.read()
    if resp.status >= 300:
        print resp.status, resp.reason, container_path+'/'+filename
conn.close()

Although slightly longer, the majority of the extra code is for the auth. In my tests, uploading 1000 16KB files took about 4.5 minutes. A whole minute improvement for only 1000 objects is a very good result. I would expect the difference to be even greater as the number of files increases.

All of the code above (plus code to generate the test data) can be found in my github account.

By using the ReST API directly, I can make certain assumptions about my data that are not possible in the generic language bindings. I do not need to do the HEAD requests because I know I have just created the container and I have not uploaded the files yet. I am explicitly setting all the data for each object upload. Further improvements would be to add some error handling and parallelization.