some thoughts » python

Compressed File Reader

John — Fri, 16 Apr 2010 03:19:13 +0000

Recently, I had need of streaming compressed data from an uncompressed file without buffering the entire file in memory. Using Python’s gzip library would require me to create a new file on disk. The zlib module offers streaming, but it does not produce the gzip headers. I wanted something that would produce gzip-compatible output in a streaming fashion.

To solve this, I wrote a class that wraps a file object and provides a read method to generate the compressed data.

class CompressedFileReader(object):
    def __init__(self, file_obj, compresslevel=9):
        self._f = file_obj
        self._compressor = zlib.compressobj(compresslevel,
                                            zlib.DEFLATED,
                                            -zlib.MAX_WBITS,
                                            zlib.DEF_MEM_LEVEL,
                                            0)
        self.done = False
        self.first = True
        self.crc32 = 0
        self.total_size = 0
 
    def read(self, *a, **kw):
        if self.done:
            return ''
        x = self._f.read(*a, **kw)
        if x:
            self.crc32 = zlib.crc32(x, self.crc32) & 0xffffffffL
            self.total_size += len(x)
            compressed = self._compressor.compress(x)
            if not compressed:
                compressed = self._compressor.flush(zlib.Z_SYNC_FLUSH)
        else:
            compressed = self._compressor.flush(zlib.Z_FINISH)
            crc32 = struct.pack(", self.crc32 & 0xffffffffL)
            size = struct.pack(", self.total_size & 0xffffffffL)
            footer = crc32 + size
            compressed += footer
            self.done = True
        if self.first:
            self.first = False
            header = '\037\213\010\000\000\000\000\000\002\377'
            compressed = header + compressed
        return compressed

This code, with some simple tests and examples, is available on github: http://github.com/notmyname/python_scripts/blob/master/compressed_file_reader_test.py

One potential use case is streaming compressed data to a web service. For example, one could use this class to compress data as it is streamed to cloud files.

conn = cloudfiles.get_connection(username, apikey)
container = conn.create_container('some_container')
test_object = container.create_object('file.gz')
test_object.content_type = 'application/x-gzip'
with open('path/to/large/uncompressed/file', 'rb') as f:
    compressed_f = CompressedFileReader(f)
    test_object.write(compressed_f)

Cloud Files CDN Stats

John — Tue, 09 Feb 2010 22:57:14 +0000

Cloud Files offers public content through Limelight’s CDN network. On public containers, one can opt in to save the logs for all content requested from the CDN. These logs are for the raw usage in an apache log format and are stored compressed in a container named “.CDN_ACCESS_LOGS”. One can then parse these logs with any commercial analytics tool or use a custom solution. Being a developer, I wrote a small Python script that loads these log files and aggregates the data.

The code can be found in my github repository.

After updating the code with your own Cloud Files credentials (or using your own cf_auth module), usage is similar to the following:

1	$ ./cf_stats.py obj_name

“obj_name” is one of the keys the stats can be grouped on. Others include “date”, “container_name”, and “user_agent”. The default is “obj_name” and any incorrect parameter will generate a usage message.

Sample output:

Object Name: my_file.pdf
Count: 11
User Agents: "Yandex/1.01.001 (compatible; Win16; I)"
Response: 200 304
Referrers: -
IPs: 1.2.3.4 1.2.3.5 1.2.3.6
Dates: 24/Jan/2010 25/Jan/2010 31/Jan/2010 01/Jan/2010 30/Dec/2009
Container Name: some_container

Any of the given fields can be used as a group. Even if the code output as-is is not to your liking, the script’s parsing and grouping functions my be a good starting point for writing your own log parser.

Quickly uploading data to Cloud Files

John — Sat, 19 Dec 2009 22:24:24 +0000

Cloud Files is a great way to store information, either to take advantage of the CDN or to offload the infrastructure requirements of storing large amounts of data. However Cloud Files is used, though, one still must upload the data to the service before being able to use it.

Uploading the data is not problematic if it can be done in small chunks or spread out over time (images on a blog, for example). The Cloud Files language APIs offer a good way to upload data in these cases. Unfortunately, the language bindings can be terribly slow for uploading large numbers of files. While they do make some optimizations (like reusing connections when available), the code is written to be very generic. For example, the bindings make HEAD requests to ensure all proper data is set before allowing you to upload an object. Additionally, at least in the Python language bindings, HEAD requests are issued when an instance of an object is created. While this is good in a general sense, these HEAD requests become superfluous when doing a large batch upload. One can achieve much better results by using the Cloud FIles ReST API directly.

As an example, let’s look at the following code which uses the Python API:

#!/usr/bin/env python
 
import os
import cloudfiles
 
username = 'xxxx'
apikey = 'xxxx'
 
conn = cloudfiles.get_connection(username, apikey)
 
container = conn.create_container('api_speed_test3')
data_list = ('test_data/%s'%x for x in os.listdir('test_data') \
             if x.endswith('.dat'))
for filename in data_list:
    try:
        obj = container.create_object(filename)
        obj.load_from_filename(filename)
    except cloudfiles.errors.ResponseError, err:
        print err
print len(container.list_objects())

In my tests, using the above code takes about 5.5 minutes to upload 1000 16KB files to Cloud Files.

I wrote the same functionality using the ReST API directly:

#!/usr/bin/python
 
import os
import httplib
 
username = 'xxxx'
apikey = 'xxxx'
 
# auth
conn = httplib.HTTPSConnection('api.mosso.com')
headers = {'x-auth-user': username, 'x-auth-key': apikey}
conn.request('GET', '/auth', headers=headers)
resp = conn.getresponse()
auth_token = resp.getheader('x-auth-token')
url = resp.getheader('x-storage-url')
conn.close()
# send data
send_headers = {'X-Auth-Token': auth_token, 'Content-Type': 'text/plain'}
container_path = '/'+'/'.join(url.split('/')[3:])+'/api_speed_test2'
conn = httplib.HTTPSConnection(url.split('/')[2])
conn.request('PUT', container_path, headers=send_headers)
conn.getresponse().read()
data_list = ('test_data/%s'%x for x in os.listdir('test_data') \
             if x.endswith('.dat'))
for filename in data_list:
    f = open(filename)
    conn.request('PUT', container_path+'/'+filename, body=f,
                 headers=send_headers)
    f.close()
    resp = conn.getresponse()
    resp.read()
    if resp.status >= 300:
        print resp.status, resp.reason, container_path+'/'+filename
conn.close()

Although slightly longer, the majority of the extra code is for the auth. In my tests, uploading 1000 16KB files took about 4.5 minutes. A whole minute improvement for only 1000 objects is a very good result. I would expect the difference to be even greater as the number of files increases.

All of the code above (plus code to generate the test data) can be found in my github account.

By using the ReST API directly, I can make certain assumptions about my data that are not possible in the generic language bindings. I do not need to do the HEAD requests because I know I have just created the container and I have not uploaded the files yet. I am explicitly setting all the data for each object upload. Further improvements would be to add some error handling and parallelization.

Cloud Files Object Copy

John — Thu, 19 Nov 2009 15:48:13 +0000

Update: This post is outdated and the referenced github branches no longer exist. The functionality described herein is now supported server-side in the latest version of Cloud Files (http://launchpad.net/swift). See my newer post for more details.

Cloud Files does not currently support object copying. However, a simple workaround is to re-upload the file with the new name. Implementing this workaround may be inconvenient, and one may miss some things like ensuring that metadata is updated. I have added a copy feature to my fork of the python-cloudfiles API that takes care of these details. This is a convenience function only and is not officially supported by Rackspace. Keep in mind that billable bandwidth will be used (unless the servicenet flag is set in the API). One option for renaming large files is to spin up a small Cloud server, use the API to copy over servicenet, and spin down the server. At $0.015 per hour, one could run a 256MB instance for 100 hours before equalling the transfer cost for copying one 5GB (Cloud Files max size) file over the billed network.

My python-cloudfiles fork on github: python-cloudfiles

Example script that copies the last file in a container to another container:

import cloudfiles
conn = cloudfiles.get_connection(username='myname', api_key='mykey')
container_name = 'example_container'
another_container = 'example_container2'
c = conn.get_container(container_name)
l = c.list_objects()
o = c.get_object(l[-1])
new_path = '%s/%s' % (another_container, o.name)
o.copy_to(new_path)
print 'copied', l[-1], 'to', new_path
new_list = conn.get_container(another_container).list_objects()
print new_list
assert o.name in new_list

PyGTK Chart widget beta release

John — Sat, 11 Jul 2009 19:17:42 +0000

We released a new version of pygtkChart today. This version is a beta release and allows for much more flexibility than the previous version. Some new features include the ability to independently address each part of a chart or graph and the ability to use GTK properties and signals. Mouse events are now supported, and hooks are available to click on individual areas of a chart.

The new version can be downloaded from http://github.com/notmyname/pygtkChart/downloads. As always, the latest source can be cloned from git://github.com/notmyname/pygtkChart.git.

PyGTK Chart Widget

John — Sat, 06 Jun 2009 16:11:14 +0000

pygtkChart is a chart widget for GTK that offers line graphs and pie charts. It’s simple to use, but it is lacking one feature that I really wanted: bar charts. I added a bar chart widget to the package, but I have not been able to get in touch with the original author to contribute the code back. So, here it is.

Download: Clone from git://github.com/notmyname/pygtkChart.git or view the source at http://github.com/notmyname/pygtkChart/tree

Installation: $ python setup.py build && sudo python setup.py install

Description: I have added two new classes: BarChart and MultiBarChart. BarChart provides a simple bar chart. MultiBarChart allows for grouped bars. The code is fairly well commented and should be easy to follow.

BarChart example

MultiBarChart example

These images are screenshots of bar_chart_test.py and multi_bar_chart_test.py, both found in bar_chart_test.tgz

UPDATE: a new version has been released

Three tier python imports

John — Sat, 28 Feb 2009 18:32:16 +0000

Suppose you want to write a python module, and, like a good software developer, you want to keep a separation between your development, model/testing, and production versions of your code. This is a simple task with a few tricks in __init__.py.

First, our tester code:

1
2
3

import module
x = module.test.Klass()
print x.status()

module/_dev/test.py is similarly simple (the model and prod versions respectively return “model” and “prod” instead of “dev”):

1
2
3

class Klass(object):
    def status(self):
        return '%s in dev' % str(self)

module/__init__.py:

import os
 
env = os.environ.get('ENV','dev').lower()
 
if env == 'prod':
    from _prod import *
elif env == 'model':
    from _model import *
else:
    from _dev import *

__init__.py in _dev, _model, and _prod:

import os
 
def _load_code():
    parent_dir = os.path.dirname(__file__)
    base = os.path.basename(os.path.dirname(parent_dir))
    modules = []
    for root, dirs, files in os.walk(parent_dir):
    for name in files:
        if name.endswith('.py') and \
          not name.startswith('__') and \
          not name.startswith('.'):
            module_name = '%s.%s.%s' % (base,
                        os.path.basename(parent_dir),
                        name.rsplit('.', 1)[0].replace(os.sep, '.'))
            __import__(module_name)
            modules.append(module_name)
    return modules
 
__all__ = []
 
def update__all__():
    global __all__
    __all__ = [x.rsplit('.',1)[-1] for x in _load_code()]
 
update__all__() # intial setup

When tester.py is run, it checks the environment variable ENV and imports the proper version of the file.

$ ENV=dev python ./tester.py
<module._dev.test.Klass object at 0x9d66e2c> in dev
$ ENV=model python ./tester.py
<module._model.test.Klass object at 0x9435e2c> in model
$ ENV=prod python ./tester.py
<module._prod.test.Klass object at 0x9f12e2c> in prod

Full source code.