Programmer Thoughts

Swift Tech Overview

2012-04-22T00:00:00+00:00

Openstack Object storage, called swift, is a distributed, fault-tolerant, eventually consistent object storage system. In this post, I’d like to go in to some detail about what that means.

Distributed

Swift is a distributed system. It is designed to be run on a cluster of computers rather than on a single machine. Swift is composed of three major parts: the proxy, storage servers, and consistency servers.

Proxy

The proxy server is a server process that provides the swift API. As the only system in the swift cluster that communicates with clients, the proxy is responsible for coordinating with the storage servers and replying to the client with appropriate messages. The proxy is an HTTP server that implements swift’s REST-ful API. All messages to and from the proxy use standard HTTP verbs and response codes. This allows developers building clients to interact with swift in a simple, familiar way.

Swift provides data durability by writing multiple complete replicas of the data stored in the system. The proxy is what coordinates the read and write requests from clients and implements the read and write guarantees of the system. When a client sends a write request, the proxy ensures that the object has been successfully written to disk on the storage nodes before responding with a code indicating success.

Storage Servers

The swift storage servers provide the on-disk storage for the cluster. There are three types of storage servers in swift: account, container, and object. Each of these servers provide an internal REST-ful API. The account and container servers provide namespace partitioning and listing functionality. They are implemented as sqlite databases on disk, and like all entities in swift, they are replicated to multiple availability zones within the swift cluster.

Swift is designed for multi-tenancy. Users are generally given access to a single swift account within a cluster, and they have complete control over that unique namespace. The account server implements this functionality. Users can set metadata on their account, and swift aggregates usage information here. Additionally, the account server provides a listing of the containers within an account.

Swift users may segment their namespace into individual containers. Although containers cannot be nested, they are conceptually similar to directories or folders in a file system. Like accounts, users may set metadata on individual containers, and containers provide a listing of each object they contain. There is no limit to the number of containers that a user may create within a swift account, and the containers do not have globally-unique naming requirements.

Object servers provide the on-disk storage for objects stored within swift. Each object in swift is stored as a single file on disk, and object metadata is stored in the file’s extended attributes. This simple design allows the object’s data and metadata to be stored together and replicated as a single unit.

Consistency Servers

Storing data on disk and providing a REST-ful API to it is not a hard problem to solve. The hard part is handling failures. Swift’s consistency servers are responsible for finding and correcting errors caused by both data corruption and hardware failures.

Auditors run in the background on every swift server and continually scan the disks to ensure that the data stored on disk has not suffered any bit-rot or file system corruption. If an error is found, the corrupted object is moved to a quarantine area, and replication is responsible for replacing the data with a known good copy.

Updaters ensure that account and container listings are correct. The object updater is responsible for keeping the object listings in the containers correct, and the container updaters are responsible for keeping the account listings up-to-date. Additionally, the object updater updates the object count and bytes used in the container metadata, and the container updater updates the object count, container count, and bytes used in the account metadata.

Replicators ensure that the data stored in the cluster is where is should be and that enough copies of the data exist in the system. Generally, the replicators are responsible for repairing any corruption or degraded durability in the cluster.

Fault-tolerant

The combination of swift’s pieces allows a swift cluster to be highly fault-tolerant. Swift implements the concept of availability zones within a single geographic region, and data can be written to hand-off nodes if primary nodes are not available. This allows swift to survive hardware failures up to and including the loss of an entire availability zone with no impact to the end-user.

An interesting consequence of this design is that upgrades and cluster resizes can be easily performed on a production cluster with zero end-user downtime. Swift provides both forward and backwards compatibility of its API, so a swift cluster can be running multiple versions of the swift software at the same time, as is common while the software is being upgraded. Similarly, during resizes, the incongruent data about where data lives is simply seen as a failure. Processes like replication ensure that the data will be moved to its correct location.

Eventually Consistent

Swift achieves high scalability by relaxing constraints on consistency. While swift provides read-your-writes consistency for new objects, listings and aggregate metadata (like usage information) may not be immediately accurate. Similarly, reading an object that has been overwritten with new data may return an older version of the object data. However, swift provides the ability for the client to request the most up-to-date version at the cost of request latency.

Example Request Flow

When an object PUT request is made to swift, the proxy server determines the correct storage nodes responsible for the data (based on a hash of the object name) and sends the object data to those object servers concurrently. If one of the primary storage nodes is unavailable, the proxy will choose an appropriate hand-off node to write data to. If a majority of the object servers respond with a success, then the proxy returns success to the client.

Similarly, when an object GET request is made, the proxy determines which three storage nodes have the data and then requests the data from each node in turn. The proxy will return the object data from the first storage node to respond successfully.

Client Data Designs

Using any storage system effectively means understanding the characteristics of the system and the guarantees that the system provides. Swift is optimized for high concurrency rather than single-stream throughput. The aggregate throughput of a swift cluster is much higher than what is available for a single request stream. A swift client can take advantage of this by distributing data across multiple containers within an account. For example, backups may be stored by day or week in a container that includes that information in its name. Or a photo-sharing application may store images across many containers by using a prefix of the hash of the photo in the container names.

Summary

Swift’s design provides robust software that can run effectively on unreliable (read: cheap) hardware. Modular processes allow deployers to optimize clusters based on client use cases. Fault-tolerance allows clusters to be effectively managed by a limited operations staff.

Swift is production-ready code that has been running at scale powering Rackspace Cloud Files for two years. It is being deployed around the world at large and small scale by public cloud service providers and for private, internal needs. Swift is 100% open source released under the Apache 2.0 license. For more information, you can read the technical docs, the admin guide, or the API guide. To get started building applications for swift, you can use either the stand-alone Python module included in swift’s code or any of Rackspace’s Cloud Files language bindings. If you have further questions, ask on the Openstack mailing list or in #openstack on freenode.

Swift State of the Project Spring 2012

2012-04-05T00:00:00+00:00

The last six months of OpenStack swift development have been the most active six-month period for the project since the code was first put into production. The developer community has grown, the code has improved, and adoption has increased. The past six months have covered the Openstack “Essex” release cycle. During this time, swift has made five releases: 1.4.4 through 1.4.8.

Where We Are

The easiest way to get an overview of swift’s evolution is to look at the version control logs.

Swift has had 125 non-merge commits:

git shortlog -nes --no-merges 1.4.3..1.4.8 | awk '{SUM+=$1} END {print SUM}'

Greg Holt has been the most prolific commiter:

git shortlog -nes --no-merges 1.4.3..1.4.8 | head -1

Swift has had contributions from Rackspace, SDSC, RedHat, Nebula, HP, SwiftStack, Internap, Memset, CERN and others.

The three largest commits in the last six months have been for the formpost middleware, man pages, and the expiring objects feature:

formpost 7fc1721d7d5290a6af278f9b6844cd3b96b7c7c3
    (11 files changed, 3359 insertions(+), 16 deletions(-))
man pages 0b0785e984d9164c1d1cd84f05dd9909bb7d37a8
    (27 files changed, 3148 insertions(+), 0 deletions(-))
expiring objects 872420efdb8e6e945cd2fe06994136b8c2ee153a
    (20 files changed, 2043 insertions(+), 53 deletions(-))

But looking at VCS logs doesn’t tell the whole story. What is in these commits?

Several important new features have been added to swift. Swift now supports expiring objects, HTML form POSTs with temporary signed URLs, and the Openstack auth 2.0 API in the swift CLI. Other new features include new config options, optional functionality in middleware, and more ops tools.

Expiring objects allow a swift user to set an expiry time or a TTL on an object, after which the object is no longer accessible and will be deleted from the system. This feature enables new use cases for swift. For example, this feature could be used by a document managements system with data retention requirements.

The new formpost and tempurl middleware modules allow a swift user to create a URL with write access and then use that URL as the target of an HTML form POST. This feature is aimed at a control panel use case. Since swift uses an auth method based on information in request headers, browsers typically can’t access swift directly. With these two new middleware modules, someone building a swift control panel can have the browser directly upload content into the swift cluster. Since the requests are going directly to swift and don’t have to be proxied through the control panel web servers for auth, the control panel deployer only has to scale infrastructure based on the control panel usage, not swift usage.

In addition to new features, many bugs have been squashed as well. Swift developers have found and fixed memory leaks, improved data corruption detection, improved replication, and improved the way rings are built.

Swift’s documentation has also been greatly improved in the last six months. Thanks to Marcelo Martins, an ops engineer at Rackspace, swift now has a full set of man pages. Additionally, swift’s self-auditing tool (swift-recon) now has full documentation.

Beyond the code, swift’s community has grown quite a bit. In addition to many private deployments, several companies have announced public deployments or their internal usage of swift. Softlayer, Haylix, and Aptira have all announced public clouds that use swift. Wikimedia Foundation has announced that all thumbnails on wikipedia are now served from a swift cluster, and they are migrating all of their media files to a swift back end.

Swift now has fifty-nine contributors listed in the AUTHORS file. Twenty-seven have been added in the last six months. This is incredible growth (nearly 50%), and many of these new contributors come from companies that had not previously contributed to swift. This growth speaks to the increasing rate of adoption of swift and builds a strong developer base that will ensure swift’s success in the furture.

Where We’re Going

However, swift is by no means “finished” or “complete”. There are always bugs to fix and edge cases that can be handled better. There are new features and use cases that can and should be solved. Some examples include solving multi-site deployments and keeping very large containers performant. Both of these improvements will allow swift to grow beyond its current use case, but they involve tremendous complexity to implement well. It is unlikely that serious attempts to solve these issues will be attempted until they become pain points for swift deployers. As one of the swift developers said, “Swift has solved all the easy problems. All we have left are the really hard problems.”

The biggest challenges facing swift are not technical; they are about the developer community. Expect the swift community to continue to grow. More companies are deploying swift. More developers will be contributing to swift. A larger developer community will of course bring new challenges, but much can be learned from other Openstack projects like nova. Bringing more developers to swift will allow swift to become more robust and more adaptable to a wider variety of use cases.

The next six months for swift should bring more community education and a larger ecosystem. More companies will deploy swift, and their unique experiences will allow swift to become more robust and feature-filled. Swift’s future is bright as both public and private clouds continue to grow.

Storage is important. Everyone has data, and it’s always growing. You should have ownership of everything that touches your data. OpenStack gives you that power.

Storage Systems Overview

2012-02-19T00:00:00+00:00

Storage requirements are getting huge. Data is incredibly sticky: it doesn’t move or ever get smaller. These two realities make your choice of a storage system vitally important. However, there is no storage system that provides a single solution for all use cases. You must pair the right use case with the right system.

When the word “storage” is mentioned, some people think of different types of hardware like hard drives (both spinning media and solid state) and tape drives. Some think of databases. Some think of deployment patterns like NAS, DAS, or SAN. Others may think of specific vendors. NetApp, Isilon, and EMC all offer storage products.

Types of Storage

However, there are only three types of storage: block, file, and object. Each type offers their own advantages and has their own use cases.

Block Storage

Block storage gives you access to the “bare metal”. There is no concept of “files” at this level. There are just evenly sized blocks of data. Generally, using block storage offers the best performance, but it is quite low-level. Database servers often times can take advantage of block storage systems. An example of a common block storage system is a SAN.

File Storage

File storage provides access to a file system. This is the most familiar kind of storage–it’s what we interact with most on a daily basis. Users of file storage have access to files and can read and write to either the whole file or a part of it. File systems are what operating systems provide on all of our personal computers. In a shared environment, file storage is often seen as a network drive.

Object Storage

Object storage is probably the least familiar type of storage to most people. Object storage doesn’t provide access to raw blocks of data. It doesn’t offer file-based access. Object storage provides access to whole objects, or blobs of data and generally does so with an API specific to that system. Unlike file storage, object storage generally does not allow the ability to write to one part of a file. Objects must be updated as a whole unit. Three of the most common commercial object storage systems are Amazon’s S3, EMC’s Atmos, and Rackspace’s Cloud Files. Object storage excels at storing content that can grow without bound. Perfect use cases include backups, archiving, and static web content like images and scripts. One of the main advantages of object storage systems is their ability to reliably store a large amount of data at relatively low cost.

Each type of storage has advantages and disadvantages. Trade-offs come when you try to grow your storage or layer the storage abstractions. What happens when you try to grow your storage system beyond a few dozen terabytes? What about beyond a petabyte or beyond fifty petabytes? Systems at this scale must make trade-offs in some areas.

CAP Theorem

In computing, the CAP theorem states that distributed systems must choose two out of consistency, availability, and tolerating network failure. For example, a system can be consistent (ie all reads get the most current data) and handle network failures, but it must sacrifice availability to do so. Or a system can choose to handle network failures and have perfect availability, but it must sacrifice consistency to do so. Distributed systems must always handle network failures, so they must choose to sacrifice either availability or consistency.

Storage systems become distributed as they grow. Openstack Swift (the basis for Rackspace’s Cloud Files product) chooses to sacrifice consistency for availability and network failure tolerance. This choice allows the system to scale to enormous levels and provide massive uptime, but it also means that in certain scenarios, some data may not be updated throughout the entire system. For example, a container listing may not be up-to-date immediately after writing an object. Swift will queue the container listing update and allow the object write to succeed. This sort of consistency model is called “eventual consistency”.

Object Storage Use Cases

Since object storage is the least familiar to people, I’d like to review some use cases for these type of systems. The most common use cases are backups, archiving, and web content. These use cases are fairly straightforward and easy to understand.

One of the most exciting use cases for object storage is as the back end for a storage appliance. Since the storage appliance presents the object storage system as local or network storage, the distinction between on-site and off-premise storage goes away. Those using a storage appliance backed by an object storage system can start to think about “data to be stored” and not have to worry about where the data is stored or managing separate backups.

Document storage is another interesting use case for an object storage systems. At first glance, this seems to be the same use case as archiving, but document management systems will go beyond simple archiving and also add in policies for each document describing permissions and retention.

Similarly, an object storage system can be used for disaster recovery. Since a system like Swift can store large amounts of data cheaply and reliably, it makes sense for a business to store a copy of their data in a remote Swift cluster. Generally an object storage system works well for this use case since the DR data will be large and needs to be updated and restored using high concurrency.

Object storage is also good for storing large data sets. Scientific research gathers enormous amounts of data as humanity looks at the biggest and smallest things in our universe. This data is oftentimes impossible to replace and becomes the basis for further research for decades to come. Object storage systems can reliably store this information in a cost-effective manner.

Object storage systems can also be used as the basis for other storage systems. For example, an object storage system could be used as the basis for a block storage system. Each “block” in the block storage system would be represented as an object in the object storage system. Layering abstractions like this will allow your block system to grow to a very large scale, but it will have certain performance penalties. A block system build on top of an object system will have much higher latency than a traditional block storage system.

Choose the Right System

Block storage, file storage, and object storage each excel in different areas. Understanding the storage landscape and the advantages and costs of each type of storage allows savvy users to choose the right system for their use case.

Democratization of Data

2011-10-28T00:00:00+00:00

You should have control over your data. If you want to host your own data, you should be able to. If you want to pay someone else to host your data, you should be able to interact with their systems in the same way you interact with your local system. You should be able to change hosting providers without changing your interface. You should be able to pull your data from a hosting provider to host it locally. You should be able to host your data across many providers seamlessly. You should be able to move your compute needs to your data storage. You should be able to separate your concerns over distributing your data from who is distributing your data.

You can do three things with data: store it, compute it, and deliver it. You should be in control of where you data is stored, how you compute on it, and how you distribute it. This is the promise of Openstack: a common infrastructure that puts you in control of your data. This promise is the democratization of data.

Ultimately, the democratization of data comes down to storage. Openstack provides a high-quality object storage system called swift. Swift is ideal for unstructured data that can grow without bound. Backups and static web content are perfect examples of good use cases for swift.

Computing on your data is provided by another Openstack project: nova. Nova enables the management of large numbers of dynamic virtual machines. It is directly comparable to AWS EC2 and Rackspace Cloud Servers.

Openstack currently lacks integration with content delivery networks. Such integration should facilitate simple distribution and management of data with exiting CDN providers (or perhaps even allow you to run your own CDN).

These three pillars, along with various complementary projects like identity management, queueing, and block storage, provide a foundation upon which we can build the future.

Swift State of the Project Fall 2011

2011-10-13T00:00:00+00:00

Swift has been running in production at Rackspace for over a year with near 100% uptime. Rackspace’s swift clusters store billions of objects and petabytes of data. Several companies, including Internap, KT, SDSC, HP, and others, have also deployed and are running swift clusters in production. These clusters range in size from fairly small to several petabytes. Other organizations, including CERN, are evaluating swift for production use. Overall, swift is a success.

Swift’s active developers are all currently Rackspace employees. Other companies have talked about features and promised to contribute code, but so far, no major patches have been forthcoming. Unfortunately this means that the day-in day-out needs for swift come from Rackspace. While many needs can be anticipated and met simply by looking at Rackspace’s needs, some key areas of development are missed. For example, Rackspace does not often deploy new swift clusters, therefore automatic deployment tools are neglected. Similarly, Rackspace’s needs focus on a general use case for a broad set of customers. Other clusters may have more specific needs based on different use cases. Until swift developers see these other use cases, swift is not likely to optimize for them.

Although other companies are not currently contributing swift code, many companies are active in the community. Piston, Nebula, Voxel, HP, and others are actively engaging the developer and user communities. They sponsor biweekly meetups, engage on the mailing lists, contribute in IRC, participate in the design summits, and generally talk about what they are doing and what needs to be done. For this, I am grateful. It seems that swift currently meets the needs of these groups. I hope that as they grow and use swift more, they will see areas to improve the software and contribute those improvements back to the community.

As we move forward with swift development, certain fundamental things must be preserved, protected, and encouraged. We must maintain a healthy project. We must ensure good feedback channels with users. We must encourage other companies to continue to participate and even submit patches. We must do what we can to encourage and support an active ecosystem of tools for swift. The universe of end-user tools, automation software, and monitoring systems all factor in to a decision to use swift or not. If we fail in these fundamental areas, we might as well pack up and go do something else.

With these concerns in mind, I see three realms of future swift development. Realm one is improving swift by fixing bugs and adding features. Realm two of swift development is data-compute locality. Most (if not all) data processing tasks can be improved by reducing the latency between where the data is stored and where it is processed. Realm three moves beyond data-compute locality by a single swift deployer and solves data federation.

We are currently working on realm one: improve swift by fixing bugs and adding features. The main goals are around very large and very small clusters. This is generally an ongoing task, and even when large and small deployments are better served, there will always be bug fixes and smaller feature improvements. Some features will be large, and some will be small. The work here mostly focuses on filling out the feature gaps in swift for specific use cases.

Realm two is waiting on nova stability. After large nova clusters are running in production, we can start to explore what it will take to unify the clusters. The goal is to bring compute to “near” the data in a network sense. The closest “near” can be is local to the same server, but it could perhaps be more simply solved by only being in the same cabinet or availability zone. Since data is “sticky” and hard to move, oftentimes bringing the compute to the data is more realistic. I do not foresee swift ever merging with nova; rather, I would like to see swift and nova cooperate in such a way that swift’s ring can be used as a scheduler for nova VMs. Currently nova is in a state of flux and needs to focus on maturity before large problems like swift integration are tackled. I expect nova-swift integration to be on hold for about another 12 months while nova matures.

Realm three is the ultimate goal. Federating compute is a fairly simple concept to understand. “Bursting into the cloud” is common enough to have become a marketing phrase. Federating storage still needs to be defined even before it can be understood. I believe it involves datasets distributed and replicated across many storage providers and dynamically balancing access to them. This is something I’ve talked about in a previous blog post.

Solving these problems will take a lot of work and a lot of time. As we move from one realm to the next, we must not consider work to be “done” in the previous realms. We must always listen to feedback and continue to polish the system as a whole.

Swift has been actively developed for a little over two years now. It was revealed to the world about one year ago and has made tremendous progress since. I’m quite proud to have been a part of the project. We have all learned a lot and had a lot of fun. Swift is in a great place: openstack momentum is growing, more users are deploying swift, and the vast majority of the feedback we hear is positive. Swift’s first two years have been a success. As we remember the fundamental things and work together as part of an active community, swift’s future will be even brighter than its past.

Setting Permissions (ACLs) on Openstack Swift containers

2011-08-04T00:00:00+00:00

I frequently see people in the #openstack IRC channel on freenode asking about how to set up ACLs in swift. Here’s a short tutorial.

First, set up two accounts. How to do this is specific to your auth system. Fir this example, I’ll use the default tempauth that ships with swift.

In your proxy server config file, under the tempauth section, add the accounts:

user_test_tester = testing .admin
user_test_tester2 = testing2

The first user (“tester”) has admin privilages on the account (“.admin”). The second user (“tester2”) is in the test account, but will only have access to what the first user grants him. The two accounts don’t need to have the same tempauth account (the “test” part).

Auth the first user and create a container. Then an read permissions on that container for the second user:

$ curl -i -H "X-Auth-User: test:tester" -H "X-Auth-Key: testing" \
    http://swift/auth/v1.0
$ curl -i -XPUT -H "X-Auth-Token: token1" http://swift/v1/AUTH_test/container
$ curl -i -XPOST -H "X-Auth-Token: token1" -H "X-Container-Read: test:tester2" \
    http://swift/v1/AUTH_test/container

Note that in the last curl command the proper value for the ACL is <account>:<user>.

Now, auth the second account. Note that the second account cannot list the containers or do anything but read what’s in the container called “container”.

$ curl -i -H "X-Auth-User: test:tester2" -H "X-Auth-Key: testing2" \
    http://swift/auth/v1.0
$ curl -i -H "X-Auth-Token: token2" http://swift/v1/AUTH_test/
$ curl -i -H "X-Auth-Token: token2" http://swift/v1/AUTH_test/container/
$ curl -i -XPUT --data-binary 1234 -H "X-Auth-Token: token2" \
    http://swift/v1/AUTH_test/container/foo

If one desires, adding the X-Container-Write header to a container will similarly grant write access.

Quickly Uploading Data To Cloud Files, Part 2

2011-06-27T00:00:00+00:00

Revisiting an old blog post on how to quickly upload data to cloud files, I figured an improved example would be beneficial.

Below is an updated script that calls the API directly and uses eventlet for concurrency. The code is available on github.

 1 #!/usr/bin/env python
 2 
 3 import eventlet
 4 eventlet.monkey_patch()
 5 
 6 import sys
 7 import httplib
 8 import os
 9 
10 from cf_auth import username, apikey
11 
12 container_name = sys.argv[1]
13 
14 # auth
15 conn = httplib.HTTPSConnection('auth.api.rackspacecloud.com')
16 conn.request('GET', '/auth',
17              headers={'x-auth-user': username, 'x-auth-key': apikey})
18 resp = conn.getresponse()
19 AUTH_TOKEN = resp.getheader('x-auth-token')
20 URL = resp.getheader('x-storage-url')
21 CONNECTION_ENDPOINT = URL.split('/')[2]
22 conn.close()
23 
24 SEND_HEADERS = {'X-Auth-Token': AUTH_TOKEN, 'Content-Type': 'text/plain'}
25 CONTAINER_PATH = '/' + '/'.join(URL.split('/')[3:]) + '/' + container_name
26 
27 # create the container
28 conn = httplib.HTTPSConnection(CONNECTION_ENDPOINT)
29 conn.request('PUT', CONTAINER_PATH, headers=SEND_HEADERS)
30 conn.getresponse().read()
31 conn.close()
32 
33 
34 def run(filename_list):
35     conn = httplib.HTTPSConnection(CONNECTION_ENDPOINT)
36     for filename in filename_list:
37         with open(filename, 'rb') as f:
38             conn.request('PUT', CONTAINER_PATH + '/' + filename, body=f,
39                      headers=SEND_HEADERS)
40         resp = conn.getresponse()
41         resp.read()
42     conn.close()
43 
44 data_list = ['test_data/%s' % x for x in os.listdir('test_data')
45              if x.endswith('.dat')]
46 concurrency = 20
47 pool = eventlet.GreenPool(size=concurrency)
48 [pool.spawn(run, data_list[concurrency*i:concurrency*(i+1)])
49     for i in xrange(len(data_list)/concurrency)]
50 pool.waitall()

This code is dramatically faster than either of my two previous examples. While writing this post, the python-cloudfiles example completed in 9 minutes 11 seconds. The direct API example completed in 7 minutes 51 seconds. The code above using eventlet completed in only 1 minute 13 seconds. This is nearly ten times faster than using the language bindings and without much tweaking at all.

A Vision of the Future of Storage

2011-05-22T00:00:00+00:00

Businesses are beginning to embrace “The Cloud” as a cost-effective way to offload internal compute and storage needs. However, many businesses still keep data in-house. Unfortunately, most of this data is useful when one can compute on it, and transfer costs to move the data to the compute centers can be cost-prohibitive. This implies that by storing data in-house, businesses also must host compute resources in-house.

Therefore, in an effort to save money, businesses have searched for ways to “burst into the cloud”–a marketing phrase which simply means temporarily using someone else’s public compute infrastructure to do some work and only paying for what is used. Unfortunately, this presents difficulty when it comes to storage. The data lives in-house. “Bursting” doesn’t make sense for storage.

There is an answer: federated storage.

Federated storage is where the storage cluster becomes a network of storage clusters. Upload data to a storage endpoint. Metadata on your data provides rules as to where the data should be stored. For example, rules can be set to require that the data is stored in geographically distinct regions. Other rules can be set to require that data is stored with at least two companies. This gives you both geographic and business redundancy. If an earthquake knocks out a data center or a company goes out of business, the data is protected.

Storage providers can enter into peering relationships that allow them to offer a storage network to their customers. For example, imagine that Dell, Rackspace, Internap, and ATT all have storage clusters, each with their own customers. Each company has different capacity and locations available. If these companies entered in to a federated storage agreement, they could share customers, capacity, and geographic redundancy.

This federated storage network is similar to how airlines organize today. American Airlines and British Airlines service different parts of the world, but they are both part of the One World network that allows passengers to collect rewards with one company even when using the partner airlines.

In the same way, smaller companies like Internap and Rackspace can partner with larger companies like ATT and Dell. A Rackspace customer can choose to store data both in one of Rackspace’s data centers and also in geographic regions that Rackspace does not serve. Perhaps Dell builds a data center in South Asia. Rackspace customers would then be able to store data redundantly in both Rackspace’s data center in Chicago and also in Dell’s data center on the other side of the world.

Other storage federations could also arise. Users then are able to choose where to store their data based on the entire offering of the partnership of providers rather than on one company’s merits. Companies benefit by having access to infrastructure without the need for expensive cap-ex costs. Customers benefit by having more storage options.

As another benefit to customers and businesses alike, a storage federation also implies that data is portable. Although data is sticky, customers would be able to move their data from one company or federation to another using the same API that the companies within the federation use to manage the federation itself. At a provider level, storage could be temporarily moved to another system for things like planned downtime or unplanned outages.

However, if data is portable between companies, this implies that the companies must compete on something other than simply having the infrastructure. A smaller company could join a federation and suddenly allow their customers access to a previously unavailable network of storage. Companies would need to differentiate on service, user interface, price, or myriad other things other than the availability of the storage itself. Of course, these federations would be governed by contracts established by the cooperating companies, and, so, the specifics of any one federation would be unique.

Federated storage networks become very interesting when the associated compute resources are accounted for. Rules for storage could take into account the available compute resources in particular locations or with particular providers. Using the companies from the example above, ATT may provide a large number of available compute resources, but Dell may provide particular types of compute–GPU vector processing, for example. A Rackspace customer, then, can choose to make their backups of their Rackspace-hosted web application stored in Rackspace’s storage infrastructure but the generated data could be stored with a rule requiring a copy be made available in ATT’s storage network near ATT’s large availability of compute resources. Or, perhaps a Dell customer could require that data be made available only within Internap’s storage infrastructure because of some particular government regulation Internap meets.

Federated storage allows data to break free of a single company. Customers gain more control over their data by being able to choose where their data is stored. Businesses gain the ability to store and process their data in the most cost-effective way possible. Users are no longer limited to their own infrastructure. Federated storage allows truly unlimited data.

I’m a developer on Openstack’s object storage system, and I believe that Openstack is uniquely positioned to achieve this vision.

The Story of an Openstack Feature

2010-12-10T00:00:00+00:00

Openstack is a fairly large open-source project with a set of core developers. Anyone can submit patches for bugfixes or new features, but sometimes the process can be a little mysterious, especially for larger features or for developers that haven’t contributed to open-source projects before.

For the swift project (Openstack storage), we have a mature codebase running in a production environment. Any patches that are accepted must not have adverse effects for the scalability or performance of the system a whole.

One of the features currently being developed for swift is large object support. The feature has gone through many iterations in both design and code, but perhaps the most important development came at the Bexar design summit. As the developers on the project, we knew that files larger than 5GB were important, but we did not have a good use case. We did not want to develop a solution for large files that did not meet the needs of user who would actually want the feature. At the design summit, we were able to talk with Openstack users at NASA who had specific uses in mind for large objects. A launchpad blueprint was developed, and the existing coding work was refocused to meet the needs of the users.

Update: The large object feature has now been approved and lives in the swift trunk.

There are several ways to implement large object support. First, and most simply, is to raise the object size limit constant. The constant determines how big a file can be, but relying on it has limits, and raising it has some nasty side effects. Since an object is stored on one physical drive (per replica), one can only raise the object limit constant to the size of the smallest drive in the cluster. If a cluster is filled with 2TB drives, this means that the largest object can only be 2TB¹. Additionally, since objects are spread evenly throughout the cluster, the balance of the fullness of each drive in the cluster is related to the ratio of the max object size to the size of the drives in the cluster. At scale, each drive in a swift cluster will differ from the average fullness of all drives in the cluster by an amount proportional to the max object size.

If simply raising the max object size constant won’t work, another way to support large objects is to split the object into chunks and tell the system to treat groups of objects as one large object. The naive implementation is to split the objects as they are streamed into the system. As an object is loading into swift, swift could split the object after a certain number of bytes and then write the next bytes to a different location in the cluster. This implementation has the advantage of not requiring the user to know anything about object chunks or having a final “commit” step to finalize the large object write. This implementation was actually written, but it was rejected for its enormous complexity and various failure condition edge cases. One of its biggest disadvantages is that it does not allow for users to upload parts of the large uploads in parallel. Asking a user to upload terabytes of data as one upload simply isn’t practical.

After talking to swift users at the Bexar design summit, especially those from NASA, we realized that a large object solution that is implemented with client-side chunking would be sufficient to meet the need of our users and offer some advantages without the disadvantages of the server-side chunking implementation.

A client-side chunking implementation of large objects requires users to upload the chunks of the large objects as normal objects, but with a unique prefix. For example, one could upload three 5GB files (obj/1, obj/2, and obj/3). Then the user creates a manifest object that defines the prefix of the objects (“obj/” in this example). Now, the user can upload or the chunks concurrently, but if the manifest file is downloaded, the system will stream the concatenated chunks to the client. This allows for great flexibility for the user and still allows the system to support very large objects. With the current proposed implementation, the only limit of large files is the size of the cluster itself. If the operators can deploy servers faster than the user can upload the data, the object size is truly unlimited. This is much better than the similar feature in S3 that was announced today. This feature in swift will allow a manifest file to be created for existing content. Additionally, a manifest can be created for a large object, and content can be added to that large object at a later time without updating the manifest. Possible applications beyond simply storing large single objects include streaming all data in a single container to a client as one large object, appending to objects, maintaining sym links to files, and an upload pseudo pause and resume.

This is a feature that will be included in the swift codebase very soon, and is something we are very excited about. We think it balances the needs of the system (scalability and performance) with the requirements of the users. This feature would not have been implemented nearly as well without input from the community. The conversations we had with people at the design summit were invaluable to the design of this feature.

As always, patches are welcome in swift. If you have bug fixes or an idea for new features, we welcome contributions. Talk to us; submit your code; give us your use cases–we want swift to be the best it can be. The swift code is hosted on Launchpad, and the developers can be found on IRC in #openstack on irc.freenode.net.

Community input in the Openstack project is vital. I’m excited about where the project has been, but even more excited to see where the community takes it in the future.

Actually, it would be less than 2TB. A swift cluster is full when the first hard drive in the cluster is full. Therefore, it is wise to limit the fullness of the drives to about 80% of their capacity.↩

Simple script to upload content to Cloud Files

2010-11-20T00:00:00+00:00

I recently realized that, on the infrequent occasion that I need to share screenshots and other things with others, I don’t have a good way to quickly make content available to others. Sure, I can upload something to imgur or even use something more complicated like the Cloud Files control panel, but I’d like something simpler. Little apps like ShoveBox and TinyGrab can be helpful, but what if I want something more generic? I’m a Cloud Files developer, so I naturally assumed that the idea solution would be to put stuff in to a CDN-enabled container in Cloud Files.

I wrote a little command-line script that takes pathnames as parameters, uploads the files to a specific container, and prints the new object’s CDN URL to stdout. Then I aliased the script to “upload” in .bash_profile, and now I simply type upload some/file, and I get the sharable URL.

The code is available on my github account.

Features I Would Like to See in Swift

2010-11-09T00:00:00+00:00

Swift is a great way to store large amounts of data cheaply. This week I’m at the OpenStack design summit, and I’ve been thinking of features I would like to see added to swift.

WebDAV support: WebDAV support would allow swift users to mount public containers as network drives in any modern operating system. It could probably be implemented as WSGI middleware, and would therefore be an optional feature for any swift deployment.
Auto-compression: I would like to see swift support dynamic compression and decompression of responses based on the Accept header in the request. This feature too could be implemented as WSGI middleware and be optional for any swift deployment. One concern, however, is the extra CPU cycles required for the compression and decompression. The swift proxy servers can see high CPU load under heavy traffic conditions.
Object versioning: A more complicated feature, object versioning would allow old objects to be accessed after newer data has overwritten them. New semantics for accessing old versions (or even enabling/disabling this feature) would have to be created, and many questions relating to failure scenarios would need to be answered.
stop_marker query parameter in listings: Currently, account and container GETs can be filtered with a marker query parameter. The marker parameter will cause the listing to return values that are greater than the marker. In a container of 100 items a marker equal to item 50 will return items 51 through 100. However, if one wants to only fetch items less than item 60, the listing has to be filtered by the client. A stop_marker query parameter would return anything less than or equal to the parameter value and would be able to be used in conjunction with the other query parameters. This would be a relatively simple feature to add without any obvious (to me) risks to the system as a whole. Update: This feature is now supported in swift: https://code.launchpad.net/~notmyname/swift/end_marker.

If anyone would like to add any of these features to swift, please grab the code and submit your patch.

Swift (OpenStack Object Storage) Overview

2010-11-06T00:00:00+00:00

What is it?

Swift is a highly scalable redundant unstructured data store designed to store large amounts of data cheaply. “Highly scalable”, means that it can scale to thousands of machines with tens of thousands of hard drives. Swift is designed to be horizontally scalable–there is no single point of failure. In most large-scale deployments, swift should become more performant as the cluster grows larger. In the CAP theorem, swift sacrifices C for A and P. Most operations happen synchronously, but consistency is sacrificed in failure scenarios.

“Redundant” means that swift stores multiple copies of each entity in the system. Each copy is stored in physically distinct availability zones, so common failures like hard drive failure network issues are highly unlikely to cause data loss or downtime.

“Unstructured data store” means that swift simply stores bits. Swift is not a database. Swift is not a block-level storage system. Swift stores blobs of data. Swift offers namespace groupings within accounts as containers, but no other relation between objects is stored.

For more information on the internal workings of swift, see http://swift.openstack.org/overview_architecture.html.

What can it do?

Although swift is a key-value store, it is optimized for highly available reads and writes. This makes it ideal for storing backups and static web content. Swift is well-suited to storing and serving server backups, VM snapshots, database backups, image libraries, scripts and stylesheets, or or any other static content that needs to be accessed frequently.

Also, because swift guarantees that objects will be available for reading as soon as they are successfully written, swift can be used to store content that changes frequently.

How does one use swift?

Swift has a ReST-ful API. All communication with swift is done over HTTP, using the HTTP verbs to signal the requested action. A swift storage URL looks like

swift.example.com/v1/account/container/object

Swift’s URLs have four basic parts. Using the example above, these parts are:

Base: swift.example.com/v1/
Account: account. An account is determined by the auth server when the account is created. The devauth server that ships with swift creates URLs of the form AUTH_uuid.
Container: container. Containers are namespaces used to group objects within an account
Object: object. Objects are where the actual data is stored in swift. Object names may contain /, so pseudo-nested directories are possible.

One may get a list of all containers in an account with a GET on the account: GET http://swift.example.com/v1/account/

One may create new containers with a PUT to the container: PUT http://swift.example.com/v1/account/new_container

One may list all object in a container with a GET on the container: GET http://swift.example.com/v1/account/container/

One may create new objects with a PUT on the object: PUT http://swift.example.com/v1/account/container/new_object

Additionally, one may use POST to change metadata on containers and objects.

Get it

Swift is completely open-source released under the Apache 2.0 license. Find it at http://swift.openstack.org. Current documentation is found at http://swift.openstack.org. Patches are welcome.

git vs. bzr

2010-09-18T00:00:00+00:00

Whole wars can be (and have been) fought over the best version control system. I’ve been using git for the last couple of years or so, and I’m very happy with it. Recently, a project I contribute to switched to use bzr and launchpad; I’ve now been using bzr and launchpad nearly exclusively for the last few months.

Both git and bzr have their peculiar ways of doing version control. Both support similar functionality, but the biggest difference I’ve noticed is that of speed. git is hands-down faster. The slowness of bzr is painful and slow enough to cause me to loose momentum while I’m coding.

I tracked a few numbers this morning. The git commands are running against a project hosted on github, and the bzr commands are running against a project hosted on launchpad. Although the speed of these two sites contribute to the overall numbers, development processes often dictate pushing code to a hosting site. Therefore, these examples show the performance of the entire version control system, including a remote hosting site.

$ time git pull
Already up-to-date.
real	0m1.953s
user	0m0.148s
sys	0m0.138s

$ time bzr pull
No revisions to pull.                                                                                                                                
real	0m7.968s
user	0m0.651s
sys	0m0.277s

$ time git status
# On branch master
nothing to commit (working directory clean)
real	0m0.026s
user	0m0.008s
sys	0m0.008s

$ time bzr status
real	0m7.732s
user	0m0.528s
sys	0m0.330s

In my experience, git is fast enough so as not to be noticed. bzr is painfully slow–slow enough for me to actively try to not commit and push changes as often as I could or should.

Intro to VCS and Workflows

2010-08-28T00:00:00+00:00

Attached below are my slides for the talk I gave at PyTexas on August 28, 2010.

VCS talk slides

Server-side Object Copy in OpenStack storage

2010-07-24T00:00:00+00:00

OpenStack storage (codenamed swift supports server-side object copy.

Suppose you upload a file with the wrong object name or you needed to move some objects to another container. Without a server-side copy feature, you would need to reupload the same content and delete the existing object. With server-side object copy, you can save the step of re-uploading the content and thus also save the associated bandwidth charges, if any were to apply.

There are two ways to copy an existing object to another object in swift. One, do a PUT to the new object (the target) location, but add the “X-Copy-From” header to designate the source of the data. The header value should be the container and object name of the source object in the form of “/container/object”.

The second way to do an object copy is similar. This time, do a COPY to the existing object, and include the “Destination” header to specify the target of the copy. The header value is the container and new object name in the form or “/container/object”.

With both of these methods, the destination container must exist before attempting the copy.

If you were wanting to perform a move of the objects rather than a copy, you would need to send a DELETE request to the old object. A move simply becomes a COPY + DELETE.

All metadata is preserved during the object copy. Note that you can set metadata on the request to copy the object (either the PUT or the COPY) and the metadata will overwrite any conflicting keys on the target (new) object. One interesting use case is to copy an object to itself and set the content type to a new value. This is the only way to change the content type of an existing object.

I'm a dad twice over

2010-04-30T00:00:00+00:00

My second son, Seth, was born last Monday. Seven pounds, eleven ounces; nineteen and a half inches long.

Compressed File Reader

2010-04-15T00:00:00+00:00

Recently, I had need of streaming compressed data from an uncompressed file without buffering the entire file in memory. Using Python’s gzip library would require me to create a new file on disk. The zlib module offers streaming, but it does not produce the gzip headers. I wanted something that would produce gzip-compatible output in a streaming fashion.

To solve this, I wrote a class that wraps a file object and provides a read method to generate the compressed data.

class CompressedFileReader(object):
    def __init__(self, file_obj, compresslevel=9):
        self._f = file_obj
        self._compressor = zlib.compressobj(compresslevel,
                                            zlib.DEFLATED,
                                            -zlib.MAX_WBITS,
                                            zlib.DEF_MEM_LEVEL,
                                            0)
        self.done = False
        self.first = True
        self.crc32 = 0
        self.total_size = 0
    
    def read(self, *a, **kw):
        if self.done:
            return ''
        x = self._f.read(*a, **kw)
        if x:
            self.crc32 = zlib.crc32(x, self.crc32) & 0xffffffffL
            self.total_size += len(x)
            compressed = self._compressor.compress(x)
            if not compressed:
                compressed = self._compressor.flush(zlib.Z_SYNC_FLUSH)
        else:
            compressed = self._compressor.flush(zlib.Z_FINISH)
            crc32 = struct.pack("<L", self.crc32 & 0xffffffffL)
            size = struct.pack("<L", self.total_size & 0xffffffffL)
            footer = crc32 + size
            compressed += footer
            self.done = True
        if self.first:
            self.first = False
            header = '\037\213\010\000\000\000\000\000\002\377'
            compressed = header + compressed
        return compressed

This code, with some simple tests and examples, is available on github: http://github.com/notmyname/python_scripts/blob/master/compressed_file_reader_test.py.

One potential use case is streaming compressed data to a web service. For example, one could use this class to compress data as it is streamed to cloud files.

1 conn = cloudfiles.get_connection(username, apikey)
2 container = conn.create_container('some_container')
3 test_object = container.create_object('file.gz')
4 test_object.content_type = 'application/x-gzip'
5 with open('path/to/large/uncompressed/file', 'rb') as f:
6     compressed_f = CompressedFileReader(f)
7     test_object.write(compressed_f)

Deleted File Recovery

2010-02-14T00:00:00+00:00

Karen got a portable video camera (a Creative Vado) for Christmas. We’ve recently started playing with it, recording Ian’s activities for posterity. Last week, Karen and Ian spent a whole day making and decorating cookies. They had a great time, and it was full of some fun “firsts” for Ian. Karen got most of the activities on the camera and is planning on editing them down to a highlights reel for the rest of the family. As they were reviewing the footage (on the camera), Ian got a hold of the camera and started pushing buttons. Play and pause weren’t a problem, but on this particular camera, the trash (delete) button is flush with the edge of the camera and easy to push. Ian pushed the delete button, and Karen, as she saw what was happening, reached to take the camera from him. In the handoff, someone pushed the confirm button, and the video was gone.

Or was it?

Like most digital recorders, the Creative Vado stores its data in flash memory. The Vado’s storage uses a FAT16 filesystem. When something is deleted, the data is not immediately overwritten. The space that the delete data used is simply marked as available. So, if nothing new has been recorded, there is a pretty good chance that the data can be recovered.

After some Internet searching, I found a program called testdisktestdisk. testdisk allowed me to (mostly) recover the deleted video file. That is, I was able to recover a 140MB chunk of data that had the right name. Now I was stuck with a 140MB data file that I knew must have the data in it, but it would not open in any video player. A little more online searching, and I found DivFix++, a tool that can read and fix corrupted AVI files. Unfortunately, it didn’t work. DivFix++ didn’t even recognize the file as an AVI movie.

Nearly ready to give up, I decided to follow one last hunch I had on how to recover the movie and be a hero for the day. I started looking through the recovered file and the other known good AVI files with hexdump, a tool that shows the raw data in a file. I noticed that all of the good AVI files started with the same 16 bytes, so I guessed that AVI files start with some sort of header or index that describes the video data in the file. Using dd and cat, I created a new file that was the 16 byte header + the recovered data file. It didn’t work. However, with this new file, DivFix++ gave me a more helpful error message, leading me to believe that I was on the right track. Not knowing where the headers ended, I took the first 512 bytes of a good AVI file and prepended them to the recovered data. Right off the bat, I was able to watch the first 5 seconds of the 5 minute video. Running the new file through DivFix++ resulted in the missing file. The only side effects are a few slightly corrupted frames at the beginning of the movie. The day was saved, and the memories were preserved.

The other side of the story, and a link to the recovered video, can be found on Karen’s blog.

Cloud Files CDN Stats

2010-02-09T00:00:00+00:00

Cloud Files offers public content through Limelight’s CDN network. On public containers, one can opt in to save the logs for all content requested from the CDN. These logs are for the raw usage in an apache log format and are stored compressed in a container named “.CDN_ACCESS_LOGS”. One can then parse these logs with any commercial analytics tool or use a custom solution. Being a developer, I wrote a small Python script that loads these log files and aggregates the data.

The code can be found in my github repository.

After updating the code with your own Cloud Files credentials (or using your own cf_auth module), usage is similar to the following:

1 $ ./cf_stats.py obj_name

“obj_name” is one of the keys the stats can be grouped on. Others include “date”, “container_name”, and “user_agent”. The default is “obj_name” and any incorrect parameter will generate a usage message.

Sample output:

1 Object Name: my_file.pdf
2 Count: 11
3 User Agents: "Yandex/1.01.001 (compatible; Win16; I)"
4 Response: 200 304
5 Referrers: -
6 IPs: 1.2.3.4 1.2.3.5 1.2.3.6
7 Dates: 24/Jan/2010 25/Jan/2010 31/Jan/2010 01/Jan/2010 30/Dec/2009
8 Container Name: some_container

Any of the given fields can be used as a group. Even if the code output as-is is not to your liking, the script’s parsing and grouping functions my be a good starting point for writing your own log parser.

Nested folders in Cloud Files

2010-01-22T00:00:00+00:00

Cloud storages systems like Rackspace’s Cloud Files and Amazon’s S3 are great for storing large amounts of information. A common misconception is that these storage systems behave like traditional file systems, complete with byte-level manipulation and nested folders. It is the second of these that I want to talk about: how to simulate a nested directory (or folder) structure in Rackspace’s Cloud Files.

Cloud Files and S3 are better understood as storage systems, not file systems. Each have three basic parts: accounts, containers (buckets in S3), and objects. In Cloud Files, these three parts can be easily seen in the URL referencing an object. The URL one uses for the ReST API is of the form http://example.clouddrive.com/account/container/object. Containers are large-scale groupings of objects, operating at a higher level, conceptually, than folders. If objects were books, containers may be genres. Containers cannot be nested. That is, one cannot put a container inside of another container.

However, it is fairly easy to simulate a directory structure with objects. These “virtual directories” are not directories, per se, but object name prefixes over which one can iterate. An example should make this concept easy to understand. Suppose I wanted to store books in Cloud Files. From my analogy above, I can use the genre of the book as my container name. The object name will be of the form “author/title”. This way, I can list all books by a particular author (within a genre).

Let’s load the following books into Cloud Files:

The Pit and the Pendulum, Poe, Horror
The Masque of the Red Death, Poe, Horror
Pride and Prejudice and Zombies, Grahame-Smith, Horror
The Far Side Gallery, Larson, Comics
Something Under the Bed Is Drooling, Watterson, Comics
It’s A Magical World, Watterson, Comics

First, I will create two containers, horror and comics. Next I will name my files according to the pattern I laid out above. I will have the files “poe/the_pit_and_the_pendulum”, “poe/the_masque_of_the_red_death”, “larson/the_far_side_gallery”, etc. Then I will upload these files to their appropriate container. As a final step, I need to upload “directory marker” files. These are empty (zero-sized) files with a content-type of “application/directory”.

Update: Since the original writing of this post, Cloud Files has added support for a “delimiter” query parameter on directory listings that eliminates the need to maintain directory marker objects. From the Cloud Files developer guide:

You can also use a delimiter parameter to represent a nested directory hierarchy without the need for the directory marker objects. You can use any single character as a delimiter. The listings can return virtual directories - they are virtual in that they don’t actually represent real objects. like the directory markers, though, they will have a content-type of application/directory and be in a subdir section of json and xml results.

The following gets technical. For those wishing to use this feature of Cloud Files and not wanting to program, I recommend using a third-party tool like Cyberduck. This program handles virtual nested directories completely transparently.

Now to take advantage of these “virtual directories”, I can do container listings and give an appropriate path value. In the Python language bindings, this would look similar to the following:

1     container = cf_connection.get_container('horror')
2     books_by_poe = container.get_objects(path='poe')

The path parameter on the get_objects call returns all objects in the given value. In this case, it returns the two books in the virtual “poe” directory. Similarly, if I had given the value “grahame-smith”, I would have found his adaptation of the classic love story.

In my example, I’ve used two genre containers and virtual directories only one level deep. I could just as easily put everything into one container and nested the authors under a genre virtual directory. An object name would then be like “comics/larson/the_far_side_gallery”. The only limitation to using this feature in Cloud Files is keeping the length of the object name (including all virtual directories) under the maximum allowed (1024 characters).

For more detailed information on how to implement virtual directories, see the Cloud Files developer guide. The relevant information is found in the “Pseudo hierarchical folders/directories” section.

Quickly uploading data to Cloud Files

2009-12-19T00:00:00+00:00

Cloud Files is a great way to store information, either to take advantage of the CDN or to offload the infrastructure requirements of storing large amounts of data. However Cloud Files is used, though, one still must upload the data to the service before being able to use it.

Uploading the data is not problematic if it can be done in small chunks or spread out over time (images on a blog, for example). The Cloud Files language APIs offer a good way to upload data in these cases. Unfortunately, the language bindings can be terribly slow for uploading large numbers of files. While they do make some optimizations (like reusing connections when available), the code is written to be very generic. For example, the bindings make HEAD requests to ensure all proper data is set before allowing you to upload an object. Additionally, at least in the Python language bindings, HEAD requests are issued when an instance of an object is created. While this is good in a general sense, these HEAD requests become superfluous when doing a large batch upload. One can achieve much better results by using the Cloud Files ReST API directly.

As an example, let’s look at the following code which uses the Python API:

 1 #!/usr/bin/env python
 2  
 3 import os
 4 import cloudfiles
 5  
 6 username = 'xxxx'
 7 apikey = 'xxxx'
 8  
 9 conn = cloudfiles.get_connection(username, apikey)
10  
11 container = conn.create_container('api_speed_test3')
12 data_list = ('test_data/%s'%x for x in os.listdir('test_data') \
13              if x.endswith('.dat'))
14 for filename in data_list:
15     try:
16         obj = container.create_object(filename)
17         obj.load_from_filename(filename)
18     except cloudfiles.errors.ResponseError, err:
19         print err
20 print len(container.list_objects())

In my tests, using the above code takes about 5.5 minutes to upload 1000 16KB files to Cloud Files.

I wrote the same functionality using the ReST API directly:

 1 #!/usr/bin/python
 2  
 3 import os
 4 import httplib
 5  
 6 username = 'xxxx'
 7 apikey = 'xxxx'
 8  
 9 # auth
10 conn = httplib.HTTPSConnection('auth.api.rackspacecloud.com')
11 headers = {'x-auth-user': username, 'x-auth-key': apikey}
12 conn.request('GET', '/auth', headers=headers)
13 resp = conn.getresponse()
14 auth_token = resp.getheader('x-auth-token')
15 url = resp.getheader('x-storage-url')
16 conn.close()
17 # send data
18 send_headers = {'X-Auth-Token': auth_token, 'Content-Type': 'text/plain'}
19 container_path = '/'+'/'.join(url.split('/')[3:])+'/api_speed_test2'
20 conn = httplib.HTTPSConnection(url.split('/')[2])
21 conn.request('PUT', container_path, headers=send_headers)
22 conn.getresponse().read()
23 data_list = ('test_data/%s'%x for x in os.listdir('test_data') \
24              if x.endswith('.dat'))
25 for filename in data_list:
26     f = open(filename)
27     conn.request('PUT', container_path+'/'+filename, body=f,
28                  headers=send_headers)
29     f.close()
30     resp = conn.getresponse()
31     resp.read()
32     if resp.status >= 300:
33         print resp.status, resp.reason, container_path+'/'+filename
34 conn.close()

Although slightly longer, the majority of the extra code is for the auth. In my tests, uploading 1000 16KB files took about 4.5 minutes. A whole minute improvement for only 1000 objects is a very good result. I would expect the difference to be even greater as the number of files increases.

All of the code above (plus code to generate the test data) can be found in my github account.

By using the ReST API directly, I can make certain assumptions about my data that are not possible in the generic language bindings. I do not need to do the HEAD requests because I know I have just created the container and I have not uploaded the files yet. I am explicitly setting all the data for each object upload. Further improvements would be to add some error handling and parallelization.

Cloud Files Object Copy

2009-11-19T00:00:00+00:00

Update: This post is outdated and the referenced github branches no longer exist. The functionality described herein is now supported server-side in the latest version of Cloud Files (http://launchpad.net/swift. See my newer post for more details.

Cloud Files does not currently support object copying. However, a simple workaround is to re-upload the file with the new name. Implementing this workaround may be inconvenient, and one may miss some things like ensuring that metadata is updated. I have added a copy feature to my fork of the python-cloudfiles API that takes care of these details. This is a convenience function only and is not officially supported by Rackspace. Keep in mind that billable bandwidth will be used (unless the servicenet flag is set in the API). One option for renaming large files is to spin up a small Cloud server, use the API to copy over servicenet, and spin down the server. At $0.015 per hour, one could run a 256MB instance for 100 hours before equalling the transfer cost for copying one 5GB (Cloud Files max size) file over the billed network.

My python-cloudfiles fork on github: python-cloudfiles.

Example script that copies the last file in a container to another container:

 1 import cloudfiles
 2 conn = cloudfiles.get_connection(username='myname', api_key='mykey')
 3 container_name = 'example_container'
 4 another_container = 'example_container2'
 5 c = conn.get_container(container_name)
 6 l = c.list_objects()
 7 o = c.get_object(l[-1])
 8 new_path = '%s/%s' % (another_container, o.name)
 9 o.copy_to(new_path)
10 print 'copied', l[-1], 'to', new_path
11 new_list = conn.get_container(another_container).list_objects()
12 print new_list
13 assert o.name in new_list

When building a starship...

2009-10-06T00:00:00+00:00

I’ve been watching Battlestar Galactica again, and I was struck by something that seems to be a very common plot device in space-based stories: Starships of all shapes and sizes are able to be reconfigured in seemingly infinite ways on the fly by the crew. In BSG, the crew networks disparate systems together to create a large compute cluster and implement a multi-layered firewall to protect against the Cylon virus. In Star Trek, the crew must constantly adjust the phasers by rerouting it through the main sensor dish or change the frequency of the shields to defeat some advanced enemy. In Star Wars, Han can easily reroute power to strengthen the shields.

Starships must be large and very complex systems. They have thousands of subsystems. Computer programs controlling doors, weapons, replicators, power, life support, sensors, water recycling, artificial gravity, and many other systems need to be written and tested before being put in to use in an operating starship. Designing and implementing this much code that is responsible for the life and death of all aboard is an impossible task for one company.

Now, for all these starships to be constantly reconfigured on the fly, either the requirement specs given to the developers describe all possible scenarios that the crew of the ship may face or must detail a standard API that all subsystems should adhere to.

There is no way one can predict all situations in which a piece of software may be used. All of these subsystems must have a standard interface that allows them to all interact with one another. Because you never know–you may need to reroute the water filtration through the warp core to kill the trans-dimentional virus that is infecting the crew.

But simply having a standard API isn’t enough. Starship crews need to be able to write programs themselves in order to tell these separate systems to work together in some custom way. Sure, their may be some large patch panel of sorts to redirect some output to some other input, but at some point, Wesley will need to do something that the system wasn’t designed to do. And, of course, bug fixes and patches will need to be applied periodically.

Given these realities, I’ve come to the conclusion that starships must run open-source software. The crew needs access to the source code to make changes as necessary. Disparate systems need to be modified to work together. Crew members (and captains, apparently) need to know exactly how these systems work. Proprietary software would probably not include the source code in the installation. Even if the code were included, it would most likely have so many disclaimers and warranty-voiding clauses in the EULA (or worse, DRM) to make any such modifications useless or impossible for the end-user.

Imagine Galactica docking and being told, “I’m sorry. You have modified your installation of the software, and we can’t upgrade you to the Cylon-proof Weapons Pro v12 unless you revert your changes.”

Or maybe Han is getting pulled in by a tractor beam (as he is wont to do). He engages the auxiliary power, and, “Thank you for using Power Systems 3.0. A new patch is available for this software. Would you like to download it now? Answering ‘No’ will disable the auxiliary power systems.”

The only way for all the complicated interconnected systems of a starship to be useful, reliable, and developed in an efficient manner is for it to be open source.

When building a starship, use open source software.

Use open source software–it’s the future.

Use open source software–don’t let the Borg/Cylons/Empire win.

South with the Sun to San Antonio

2009-07-13T00:00:00+00:00

It’s official. I have accepted the formal offer to work at Rackspace. I will be working in the Cloud Files division in beautiful downtown San Antonio. This next month will be full of transition: leaving one job, moving, and starting a new job.

Update (08/07/09): We’ve moved. Everything went smoothly. The moving company was great. Our new washer and dryer were delivered today. We are in the process of unpacking a lot of boxes and learning our way around this new town. I’m looking forward to starting the new job on Monday.

PyGTK Chart widget beta release

2009-07-11T00:00:00+00:00

We released a new version of pygtkChart today. This version is a beta release and allows for much more flexibility than the previous version. Some new features include the ability to independently address each part of a chart or graph and the ability to use GTK properties and signals. Mouse events are now supported, and hooks are available to click on individual areas of a chart.

The new version can be downloaded from http://github.com/notmyname/pygtkChart/downloads. As always, the latest source can be cloned from git://github.com/notmyname/pygtkChart.git.

OpenSolaris upgrade

2009-06-11T00:00:00+00:00

OpenSolaris 2009.06 was released this month. Upgrading my home file server was pretty easy.

# pkg refresh && pkg image-update

Everything worked smoothly until the last part. The new boot environment did not activate correctly. Some further digging revealed that one of the two disks in my rpool was set with and EFI disk label (I’m not sure how that happened). I found some good information online, and soon enough, I was up and running with the new version of OpenSolaris.

ZFS was upgraded in this new version, so I upgraded it on my file server.

# zfs update

And like that, I was done. I will be interested to see if the drivers (rge) for my original NIC are better. If so, I may be able to set up trunking on the two NICs to double my throughput. I did notice that CIFS seems to work a little better. That is, I didn’t have to coerce the machine to share via CIFS like I used to.

PyGTK Chart Widget

2009-06-06T00:00:00+00:00

pygtkChart is a chart widget for GTK that offers line graphs and pie charts. It’s simple to use, but it is lacking one feature that I really wanted: bar charts. I added a bar chart widget to the package, but I have not been able to get in touch with the original author to contribute the code back. So, here it is.

Download: Clone from git://github.com/notmyname/pygtkChart.git or view the source at http://github.com/notmyname/pygtkChart/tree.

Installation:

$ python setup.py build && sudo python setup.py install

Description: I have added two new classes: BarChart and MultiBarChart. BarChart provides a simple bar chart. MultiBarChart allows for grouped bars. The code is fairly well commented and should be easy to follow.

These images are screenshots of bar_chart_test.py and multi_bar_chart_test.py, both found in bar_chart_test.tgz

UPDATE: a new version has been released

Useful Tools

2009-05-25T00:00:00+00:00

I use several online tools that make my life easier, and I’d like to share them in the hope that they make someone else’s life easier.

Readability

Readability is a bookmarklet that reformats your current page and makes it readable. It’s a wonderful tool in the world of web pages full of ads and useless junk.
Mint.com

Mint.com is a free online service to keep up with your finances. It offers a quick way to see where all of your money goes and shows where you can save money of get better deals (on bank accounts, credit cards, etc).
Nike+

Nike+ is a two-part service that allows you to keep track of running or walking with ease. The first part is the accelerometer you put in your shoe that connects to the receiver plugged in to an iPod. This records the run and offers real-time feedback during the workout. The second part is the website that keeps a history of your runs and allows you to compare and compete with the millions of other Nike+ users. The only non-free service on this list, the Nike+ will set you back $30 (assuming you already have an iPod nano or touch).</dd>
print what you like

Another free bookmarklet, print what you like allows you to selectively print sections of webpages without all of the cruft. Although I don’t print web pages very often, this tool is very helpful when I do.

Friday fun at work, part 2

2009-05-08T00:00:00+00:00

I finished something up at work today that I have been working on part-time this week. My boss really likes it, and I’m pretty happy with how it turned out. I call it attachment scrubbing.

Last week, a friend and I were talking over lunch about some email woes our company was having. Specifically, some people were emailing around some large attachments, and they were putting a lot of strain on our mail server. By the time we finished eating, we had come up with the idea of putting something in place on the mail server to intercept attachments from emails. A user can send an attachment as usual, but when it gets to the email server, my program grabs it. I remove the attachment from the email, upload it to a web server, and put a message with a link to the attachment back in the email. When the recipient gets it, their email program won’t choke on some large file, and our servers are happier because we will only be storing one copy of the attachment for each message.

The real beauty of this system is that the user sending the email doesn’t have to do anything differently. And although the idea didn’t come from a desire to make external users happier, it can make their email experience more pleasant by not clogging up their inbox.

This was pretty fun to work on for the past week and a half, and it provided an interesting diversion from some mundane system administration I’ve had to do lately. I now have a lot more understanding of how my company’s email system is set up, and I think I could set up a similar system pretty easily. Email servers are pretty complicated, but there is nothing like installing your own and customizing it that will teach you how it works.

A little puzzle

2009-05-06T00:00:00+00:00

I was listening to NPR on the way home today when a story about college entrance essays got me thinking. How do admissions officers judge the merits of an entrance essay? Is it based on some fixed scale? I can’t imaging that something as subjective as an essay can be effectively judged on a fixed scale. Is it judged relative to the other essays? This makes more sense to me, but I don’t understand the logistics. So I thought of a simpler problem that has the same elements, but is easier to identify with.

Suppose I were to come to you and offer you 100 boxes. In each box is a random amount of money between $0.01 and $100. After each box is offered to you, you must decide to keep it or to return it to me. If you return it to me, I will show you the amount of money in the box, but you will never see it again (and you won’t get the money). You are allowed to keep the first 10 boxes you choose. If you have not taken all of your 10 boxes by the time I offer the last boxes, you must take the remaining boxes to round out your total of 10. When you pick your tenth box, I will not offer any more boxes. What strategy optimizes your earnings?

Or suppose you are in a fishing tournament. At the end of the day, you must weigh in with your 10 biggest fish. If the game warden catches you with more than 10 fish in your boat, you will be disqualified from the competition. What strategy will assure your highest total weight?

Network Shares

2009-05-02T00:00:00+00:00

After setting up ZFS, I need to set up network shares and connect them to our home computers.

I originally chose NFS because of its simplicity to set up.

zfs set sharenfs=on tank/Media
zfs set sharenfs=on tank/twiddle
zfs set sharenfs=on tank/backups

That’s it for the OpenSolaris side of things. The only thing left to do is connect these NFS shares to my Mac. This is a simple task. Although I could mount them like any other server, I instead chose to use the automount facility provided by Directory Services (Note: this feature has changed in Snow Leopard). This way, the share is mounted when it is needed—a helpful tool when I am using a share for my iTunes music.

One big disadvantage of NFS, though, is that it shares based on user id. That is, user id number 1000 on the client will connect as user id 1000 on the NFS server. The practical implications of this is that each client computer should have unique user ids. Unfortunately, without a common user management system like NIS or LDAP, managing unique users across independent clients very quickly becomes tedious.

One alternative to NFS is to use CIFS/SMB. CIFS/SMB is normally used to make shares accessible to Windows computers, but it can also be used when one needs to maintain separate user accounts on server and client machines. In my case, I have two OS X computers, each with the primary user account #501. In this case, I can set up two user accounts on the file server and use SMB to access them from the Macs.

Setting up CIFS/SMB is not terribly complicated, but it does require installing a couple of packages.

pkg install SUNWsmbs SUNWsmbskr

Next, enable the smb server:

svcadm enable -r network/smb/server

And finally, set the ZFS mount to use smb:

zfs sharesmb=name=Media tank/Media
zfs sharesmb=name=twiddle tank/twiddle

At this point I could connect to the shares from my Mac via SMB, but I noticed that newly created files had no permissions. The solution was to set up ACLs on the file server.

/bin/chmod -R A=owner@:full_set:d:allow,\
owner@:full_set:f:allow,\
everyone@:rxaARWcs:d:allow,\
everyone@:raARWcs:f:allow \
/tank/Media /tank/twiddle

This simply sets an ACL that allows the owner full access and gives read access to everyone else. Note that this command uses /bin/chmod, and not the OpenSolaris default of /usr/gnu/bin/chmod. For more details on ACLs, see the excellent posts at Daz’s bits and bobs.

On the file server, I have two users (john_remote and karen_remote), both members of the same group (remote), that I use for SMB sharing. I changed the ownership and group of the files that are shared on the file server, and I can now connect from the Macs with no problem. Thinking longer term, the SMB shares have the further advantage of still being usable as computers are added or replaced. The key is just the user id and password.

Note: I did have some problems getting smb to start after a reboot. The code below fixed it.

# svcadm disable network/smb/server
# rem_drv smbsrv
Invalid argument
Cannot remove major number binding for 260
# add_drv smbsrv
# svcadm enable -r network/smb/server
# svcs | grep smb
online 20:59:18 svc:/network/smb/server:default
#

More adventures of home tech support

2009-05-02T00:00:00+00:00

Recently, Karen’s laptop refused to boot. After a few attempts to fix the problem (OS X’s Disk Utility and Disk Warrior), I connected the drive to my computer, made a complete backup, and reformatted it. This is the first time I have ever had to reinstall OS X on any Mac I have owned. I suspect her problems were caused by bad sectors on the disk, so before I reinstalled the operating system, I erased the disk by writing zeros to the entire disk. This way, if bad sectors are detected, the drive could substitute them for the good sectors it keeps in reserve.

After reinstalling the OS, she reinstalled her programs, and I restored her documents from the backup. The last thing too do was to connect he back to our home file server.

Originally, I used NFS to share our files. When I reinstalled her OS, though, Karen’s user id reverted back to the OS X default. Unfortunately, this was the same user id that I used on my computer. Because NFS based its permissions on the client’s user id, I had two choices. One, I could change her user id on her fresh install. Or two, I could use something other than NFS on the file server.

The second option seemed to be a better long-term solution, so I looked in to setting up SMB. SMB has the advantage of requiring a user id and password to connect. This means that from any computer, no matter the users on that computer, I can connect to the file server with the proper permissions.

Setting up the new network shares turned out to be less painful that I anticipated, and, for now, all is right in the world of our home computers.

Three tier python imports

2009-02-28T00:00:00+00:00

Suppose you want to write a python module, and, like a good software developer, you want to keep a separation between your development, model/testing, and production versions of your code. This is a simple task with a few tricks in init.py.

First, our tester code:

1     import module
2     x = module.test.Klass()
3     print x.status()
4

module/_dev/test.py is similarly simple (the model and prod versions respectively return "model" and "prod" instead of "dev"):

1     class Klass(object):
2         def status(self):
3             return '%s in dev' % str(self)

module/__init__.py:

 1     import os
 2 
 3     env = os.environ.get('ENV','dev').lower()
 4 
 5     if env == 'prod':
 6         from _prod import *
 7     elif env == 'model':
 8         from _model import *
 9     else:
10         from _dev import *

__init__.py in _dev, _model, and _prod:

 1     import os
 2 
 3     def _load_code():
 4         parent_dir = os.path.dirname(__file__)
 5         base = os.path.basename(os.path.dirname(parent_dir))
 6         modules = []
 7         for root, dirs, files in os.walk(parent_dir):
 8         for name in files:
 9             if name.endswith('.py') and \
10               not name.startswith('__') and \
11               not name.startswith('.'):
12                 module_name = '%s.%s.%s' % (base,
13                             os.path.basename(parent_dir),
14                             name.rsplit('.', 1)[0].replace(os.sep, '.'))
15                 __import__(module_name)
16                 modules.append(module_name)
17         return modules
18 
19     __all__ = []
20 
21     def update__all__():
22         global __all__
23         __all__ = [x.rsplit('.',1)[-1] for x in _load_code()]
24 
25     update__all__() # intial setup

When tester.py is run, it checks the environment variable ENV and imports the proper version of the file.

    $ ENV=dev python ./tester.py
    <module._dev.test.Klass object at 0x9d66e2c> in dev
    $ ENV=model python ./tester.py
    <module._model.test.Klass object at 0x9435e2c> in model
    $ ENV=prod python ./tester.py
    <module._prod.test.Klass object at 0x9f12e2c> in prod

Full source code.

Setting Up Zfs

2009-02-18T00:00:00+00:00

Now that my hardware is selected and I have OpenSolaris installed, it’s time to set up the storage pool and automatic snapshots. I have four 750GB drives that I want to set up in a RAID-Z1 format, so a quick format to find the drive identifiers, and we’re off to the races.

zfs create tank raidz c4d2t0 c4d3t0 c4d4t0 c4d5t0

I want to have three filesystems on tank: one for media (music, photos, movies, TV shows), one for Karen’s work, and one for backups. Initially, I wanted to create the backup as an iSCSI volume, but I had some problems with OpenSAN’s iSCSI initiator, and so, for now, I’ll just be using nfs to share all three filesystems.

The commands to set this up are very simple:

zfs create tank/Media
zfs create tank/twiddle
zfs create tank/backups

Since this file server won’t be doing much work besides just serving files, I have CPU cycles to spare. I’ll turn on compression, and, since this is a data store and not a place for programs, I will turn off setuid and exec.

zfs set compression=on tank
zfs set setuid=off tank
zfs set exec=off tank

Although all of the data in tank is protected against a disk failure thanks to RAID-Z, I would still like to take snapshots of the data to get historical copies. This can be enabled very easily with the zfs-auto-snapshot service.

zfs set com.sun:auto-snapshot=false rpool/ROOT
svcadm enable svc:/system/filesystem/zfs/auto-snapshot:frequent
svcadm enable svc:/system/filesystem/zfs/auto-snapshot:hourly
svcadm enable svc:/system/filesystem/zfs/auto-snapshot:daily
svcadm enable svc:/system/filesystem/zfs/auto-snapshot:weekly
svcadm enable svc:/system/filesystem/zfs/auto-snapshot:monthly

The first line prevents the auto-snapshot service from taking snapshots of rpool/ROOT. The only reason I turn it off is because the services do not start successfully when older versions of the system exist but are not mounted (more details and another solution). Perhaps one of the best things about the auto-snapshot service is that it deletes old snapshots according to schedule.

I like Time Machine’s integration with the desktop, and, if at all possible, I want to keep using it. Unfortunately, my original plan of using an iSCSI share did not work out. For now, I set up a cron job to sync my home directory to my backups mount point every hour.

0 * * * * rsync -aH --inplace --numeric-ids --delete --ignore-errors
    --exclude=\.zfs --exclude=\.Trash --exclude=\.Spotlight-V100
    --exclude=\.fseventsd /Users/john john_remote@fileserver:/tank/backups/
    >>/Users/john/Dropbox/Documents/rsync_backup.log 2>&1

While not as elegant as Time Machine, when combined with the ZFS snapshots, it does offer the “oops I deleted that” protection I’m looking for. Eventually, I hope to find a good solution that allows me to use Time Machine again. The --ignore-errors switch allows the delete to work even if an IO error occurs, and the --exclude switches prevent backups of those specific directories (which I don’t have read access to anyway). Finally, I redirect any output to a file so I can check the status at any time.

Finally, I want to be able to easily check on the health of my file server. To do that, I set up a crontab on the file server to run a few commands and upload them to my web server. Now I can see my file server status from any Internet-connected computer.

Power supplies, hard drives, and network drivers, oh my!

2009-02-17T00:00:00+00:00

I think I finally have a working file server.

The replacement power supply is working great. I’m not sure what happened to the first one. Once I had the server up and running, I started doing some tests of copying data over. My initial tests seemed successful, but I started getting some errors reported from one of the hard drives. A quick replacement from Seagate, and things seem to be going well. My zpool is resilvering the new drive now.

After my initial tests, I tried copying over some larger data sets (my 153GB iTunes library). Initially, things seemed to go fine, but, after a while, the network connection dropped. After quite a bit of searching online, I ran across some fantastic info about this exact problem. I tried his solution and things, so far, are working well. I have copied nearly 200GB of data to the file server so far. If things keep working reliably for the next week or so, I’ll start moving the data to the file server and removing it from my computer.

I’ve had to abandon the OpenSAN iSCSI initiator that I was hoping to use for Time Machine backups. That initiator seems to be buggy in OS 10.5, and it kept causing my computer to hang. Although I like the idea of using iSCSI for backups, there is one big drawback to that method. The OS X install CD does not have an iSCSI initiator, so it would be tricky at best to restore from Time Machine backups stored on an iSCSI target. My next task will be to get Time Machine to work over NFS.

Testing And Hardware

2009-02-07T00:00:00+00:00

I have recently been in need of a good storage solution. After seeing what is available on the market, I decided to build my own system and use OpenSolaris and ZFS.

In order to test my ideas without spending a lot of money up front, I downloaded and installed VirtualBox, a great little tool for running virtual computers.

Everything worked great in VirtualBox. Installation and setup were really simple. The ZFS commands are quite simple and easy to use. I found a great ZFS-based NAS tutorial online.

Everything is set up and ready to go. Let’s go shopping for hardware.

I chose to buy all of my parts at newegg.com because of their good prices, good return policies, and my good experiences ordering things there in the past. I decided on the following parts:

The tutorial I mentioned above recommended the Antec case for its quality and sonic properties. Thus far, I have been quite satisfied with it. I got the Antec power supply bundled with the case. The discount was nice, and since the case has an unusual power supply location, I figured that the same company would make a power supply with long enough cables. I feel a little more comfortable with Intel chips than I do with AMD, so I got the lowest power Core 2 I could find and a motherboard sold as a bundle with it. The Biostar motherboard also has six SATA ports. Four GB of RAM should be a good start (and it has a neat looking heat sink). I need the DVD drive to install the OS, and for less than $20, it can’t really hurt. Finally, I wanted a decent gigabit switch to connect everything together. I had previously purchased four 750GB Seagate drives, and had on hand another couple of 500GB drives.

Once the hardware is up and running and OpenSolaris is installed, it’s time to set up ZFS.

File server update

2009-02-06T00:00:00+00:00

Unfortunately, I haven’t made much progress on my new file server. I got OpenSolaris installed last weekend. After I figured out that I had accidentally turned off the SATA controllers in the BIOS, the install went flawlessly.

I didn’t set up the storage pool right after I installed OpenSolaris because I didn’t have all of my hard drives. When I first ordered my four hard drives for the Drobo, one of them was bad, so I sent it back for a replacement. The replacement came on Monday. I opened up the case, screwed in the hard drive, plugged in the cables, neatly arranged everything, and closed the case back up. I pushed the power button, and…nothing. No beeps, no clicks, no anything. Quite disappointed, I opened up the case again and took a look. Nothing smelled bad, and I couldn’t see anything that looked burned, so I assumed the magic blue smoke that makes all electronics work was still inside. I pulled the power supply (no easy task) and tried it in a computer at work. No change. I boxed up the supply and sent it back to get replaced. UPS says it was delivered today, so I hope to get a new one next week.

Karen and I finished watching season two of Battlestar Galactica last night. Unfortunately, without a working file server, I don’t have enough hard drive space on my computer to download season three. I hope to get the replacement power supply soon.

I’ve added a page that gives more specific details on setting up my home file server, including parts, screenshots, and example commands.

Finding a good storage solution

2009-01-27T00:00:00+00:00

I realized the other day that I have 110GB of stuff in my iTunes library. I don’t even think I have enough hard drive space left to get the last two seasons of Battlestar Galactica. I guess I should move everything to my 500GB external drive I bought last year.

Oh wait…it died. But I’ll get to that later.

And Karen is running out of space on her external drive, too. (How does she use up space so quickly? Why are Photoshop files so big?)

So it looks like it’s time to find a good solution for all of my home storage needs.

To tell the whole story, I need to go back a bit. Karen’s laptop has a 100GB internal drive. Not bad, but by no means sufficient when doing a lot of graphic design work. A few years ago, we got a Western Digital Passport drive. I really liked it. It has a great form factor–you could literally put it in your pocket. It was powered by the USB connection. It was relatively cheap. All-in-all, I thought we were getting a really good deal.

And then Karen’s files started getting corrupted. You know, the files that she uses to run her business? So much for Western Digital.

Shortly after that saga, Apple released OS 10.5. It included Time Machine, a truly elegant home backup solution that is seamlessly integrated into the desktop. It has saved me from accidental deletes on more than one occasion.

So to take advantage of Time Machine and to replace Karen’s failed Passport drive, we ordered two 500GB LaCie drives. Again, they were a pretty good deal: cheap, lots of connections on the back, lots of space. I used mine for backup and Karen used hers for backup and storage for work documents and the thousands of pictures we have of Ian.

I think you can guess where this is going.

LaCie drives have a problem with their power supplies. Wait, that’s not quite right. I think the LaCie representative on the phone said, “This is a rare problem, but since you have 2 drives, I’ll go ahead and send you the replacement power supply for the broken one and an extra so you will have a spare.” LaCie, you aren’t really giving me a lot of confidence in your product.

Karen is still using her LaCie drive (but running out of space), and it seems to be working just fine, but mine died. It seems like the power cable is fine, but some of the electronics in the case went bad. Now I don’t have any backups on my computer, and I’m running out of space.

So I’ve been in the market, as it were, for something that is a little more reliable. Since I’m looking at something new, I figured I could dream a little and think of what I would really like. What kind of things do I want in a home storage system?

Lots of space
Future proof. That is, I want to be able to grow my space.
Reliable. It should not break if a hard drive breaks.
Usable with Time Machine for backups
In total, cost less than $1000. Preferably, much less
Easy to use. I don’t want to have to be a full-time system admin at home every evening. I don’t have time for that, and I do it enough at work.
Small and quiet would be nice
Network accessible so Karen can get to her files when she is working in another room

Doing a lot of research online, I discovered that there aren’t a lot of good options out there. One product I looked at is Netgear’s ReadyNAS NV+. It looks like a good product, but I had two qualms about it. First, it is somewhat expensive. Without any disks, it is about $700. Second, it doesn’t have any official support for Time Machine. On the other hand, the ReadyNAS has some really nice features I like. There are a lot of people that use it and love it. It has built-in network support. People have been able to get it to work with Time Machine (although this use is unsupported). Overall, it seems like a good product; my main beef is the price.

I came across another product I liked: Drobo. Data Robotics makes a little toaster-sized box called Drobo that holds four hard drives, protects against disk failure, connects to a computer with USB or Firewire, and makes management about as complicated as using an iPod. It doesn’t have built in network support (but you can add a DroboShare for another $200), and it is a little pricey for what you get. Retail price for a Drobo without any drives is about $500. In my opinion, $300 to $400 would be a much better price for what you get. One thing that is not mentioned in the marketing material for the Drobo is the $50 annual licensing fee that lets you get firmware updates and tech support for the Drobo.

Not liking the price a whole lot, but liking the promise of hands-off storage, I ordered a Drobo and four 750GB hard drives. They were delivered, I took it home, and the tragedy of errors continued.

I spent quite a bit of time looking online about the best way to use the Drobo for Time Machine backups and storage. I connected the Drobo to my computer, thinking that I could set it as a shared drive so Karen could see it on her machine. That saves me the $200 for the DroboShare, right? Well, since Karen’s info is “mission critical” to her business, I figured I would risk my data before using her as a guinea pig. I set up a sparsebundle file on the Drobo to use for Time Machine backups. This lets me limit the amount of space used by Time Machine. I didn’t want to fill up the Drobo with backups and have no space for any media, and since the Drobo is reported to the OS as being bigger than it really is (a nice feature that allows for seamless future expansion), Time Machine, when left in an unlimited state, could try to save backups to hard drive space I don’t have. Everything seemed to be going very well. I started my initial backup of all my data (including my iTunes media), and let it churn. Hmmm….did that really just take the better part of a day to copy 180GB to the Drobo?

Finally, the backup is done. Mount the sparsebundle, browse the Time Machine backup, everything seems to be working well. Yay for the Drobo. So I moved my iTunes library over (another long wait to copy 110GB). Things are going well. I can see my data in iTunes. I can play the music. Well I guess I don’t need to keep using all the space on my hard drive, right? The data is on my Drobo, safe and secure. Move the iTunes folder to the Trash. Gulp. Empty trash. That was kinda scary, but now I’m in the brave new world of redundant data storage. Even if something goes wrong, I have a Time Machine backup of my iTunes music. I can just restore it and go on with my life.

Unless the backup is corrupted.

AUGH!!

The first sign of trouble was when Time Machine reported that it couldn’t save the backup because the target disk was read-only. “That’s weird,” I thought. Everything looks ok to me. So I eject the Drobo and plug it back in.

“This disk has errors that could not be recovered and has been made available to you with limited functionality. Please back up your data and reformat the drive.”

Errors? I looked down at the Drobo. Four soothing green lights stared back saying, “Everything’s ok here.”

A little more digging and the damage is worse than I feared. iTunes is listing all of my music, but dozens of files are reported as missing. I open the folders on the Drobo and searched for them. The folders were there, but they were all empty. Some are files that I ripped from CDs. Some are files I bought on iTunes.

A feeling of dread begins to set in.

Scared of making it worse, I didn’t touch the computer again for three days.

Having steeled myself to the task ahead, I started to look for the missing data. It’s nowhere to be found. I copied what files I had back to my internal hard drive and vowed to never, ever, delete it again. I tried to think back to what I had done with the Drobo that had caused me to loose data. Although little more than a guess, I think I have an explanation for the problem.

At some point in my first adventures with the Drobo, I restarted my computer. However, the Drobo (or something) seemed to hang and not let the computer shut down properly. Seeing that, I used the power button on my Mac to turn the computer off. Having done it many times before with no ill effects, I didn’t think anything of it. My actions seemed to offend the Drobo mightily, and the filesystem on it got corrupted. Unfortunately, this is one of the worst kinds of problems. I know immediately if a drive dies, because I can’t see it anymore. When the filesystem got corrupted, there was no warning. I didn’t know about the problem until it was too late. Although I don’t completely blame the Drobo for this error, it didn’t do anything good for our relationship. After all, isn’t the Drobo supposed to protect against data corruption errors? That’s why I bought the thing.

Several hours and $120 later, I had finished running Disk Warrior on the Drobo. Disk Warrior fixed the corruption of the Time Machine backup and I was able to restore all of my iTunes library. If your data is worth more than about $100 to you, get it. Disk Warrior is one of the best programs I have ever used.

As an aside, I want to write programs like Disk Warrior. I want to be a hero when my programs are run. Those developers have my gratitude and admiration. They have made a wonderful program, and, if they are ever in town, drinks are on me (that’s Dr Pepper for all my conservative family members reading this).

So now that I have my data back, I’m left looking at a little black box, and I can’t help but thinking of the phrase “black hole.” Do I really want to try this again, or was this just a $500 mistake? To say the least, my confidence was shaken.

I spent the rest of the week carefully reseting and using the Drobo. Things seemed to be working better the second time around, but I still wasn’t using it as any sort of primary storage. I copied (not moved) my iTunes files back over to the Drobo and found another problem. The Drobo is slow. I could not watch HD TV shows that were stored on the Drobo without any stuttering of the picture or sound.

The last strike against the Drobo was when one of my drives was reported as bad (it’s a week-old hard drive). I took out the drive, put it back in, and haven’t had any problems with it. Now I’m really unsure about what the Drobo tells me.

Overall, my opnion of the Drobo is that it is a great little box that would be good for backups, but I can’t recommend it as primary storage for data. And, unfortunately, there are other options that offer cheaper backups out there, if all you are looking for is lots of space and hard drive failure protection.

Unfortunately, with the Drobo having struck out, I’m back to square one. I have no backups for my computer. Karen is running out of space on her external drive. I’m running out of space on my hard drive.

I went back to my list of requirements and started dreaming. If I could have anything, what would I have? What I want is a Drobo that is faster and more reliable.

A few years ago, I started hearing about this wonderful new filesystem called ZFS. In a nutshell, it’s free, it offers tremendous file protection that is completely transparent to the user, and it can hold so much information that I could literally record every second of my life and not fill it up. There are several implementations of ZFS, but the official version is included in OpenSolaris, an open-source operating system made by Sun that is similar to Linux.

I’m fairly proficient at technical things, and although I don’t relish the thought of maintaining a file server at home, I know that I have the knowledge to do so. Plus, ZFS looks really easy to use. I should be able to set it up once and not worry about it. So I started looking for a computer that I could use as a home file server running ZFS. I decided that I wanted a little more customization than I get just ordering something from Dell, so I started putting pieces together at newegg.com to see what my costs would be. It turns out, you can get a really nice computer that does everything the Drobo + DroboShare does and more for about the same cost as the Drobo + DroboShare. Looking good so far.

Not wanting to order a computer without knowing for sure if ZFS would meet my needs, I decided to run a little test. Sun makes a great little (free) program called VirtualBox that lets you run different operating systems in a window on your current computer. I installed VirtualBox and made an OpenSolaris machine in it that is just like what I would put together myself. I started it up, followed some simple ZFS tutorials I found online, and quickly realized that I could do everything I want. I can have a large pool of storage for iTunes media and pictures. I can have a second large pool for Karen’s work files. I can set up volumes that look like blank hard drives to our current computers that we can use for Time Machine backups. And everything is protected from disk failure and corruption by ZFS. I think I found the greener pasture.

So this morning I mailed back my Drobo for a refund (minus a restocking fee, grrr), and placed my order for my new computer parts. Let’s hope this one works out.

Wordle tag cloud

2009-01-20T00:00:00+00:00

Tag clouds are all the rage, and wordle has created a site that makes some really pretty ones. I have seen their creations around on the web (most recently with today’s inauguration speech), and I finally got around to making one of my own. Below is a tag cloud accurate as of this posting.

Darwin and Church History, Part 3

2008-11-19T00:00:00+00:00

As promised, I am offering up some answers to the questions I posed earlier.

Must one choose between young-earth creationism and an atheistic interpretation of evolution by natural selection? Is there a middle ground?

I feel I have answered this question in my previous post.
Is your view consistent with as-yet-undiscovered scientific facts? In other words, is your argument based on a current unknown remaining unknown?

Because my response to question 1 incorporates science and human reason, I do think my viewpoint accounts for new discoveries. In this, I find myself tending to the views of Augustine and Spurgeon. Both were of the opinion that Scripture must be interpreted in light of human knowledge. By this, I mean that if some discovery is not compatible with an interpretation of Scripture, the interpretation is flawed, not Scripture. I do not think this view removes the role of divine revelation or the existence of miracles.
Supposing that the modern understanding of evolution is true, life, even sentient life, may have developed some other place in the universe. How does your theology handle this possibility?

This is a hard question. In included it because it was asked in one of my sources for my paper, and I consider it to be a serious challenge. I do not think that life (bacteria, plants, furry woodland creatures, etc.) offer any serious challenge to Christian teachings but sentient life does. There is no evidence of sentient life (or any life, for that matter) in the universe apart from that found on Earth, and my faith requires that I say that sentient life on Earth is unique. I would struggle greatly if sentient alien life is ever found.
What is the definition of science?

From Wikipedia:

Science is the effort to discover, and increase human understanding of how the physical world works. Through controlled methods, scientists use observable physical evidence of natural phenomena to collect data, and analyze this information to explain what and how things work. Such methods include experimentation that tries to simulate natural phenomena under controlled conditions and thought experiments.

I think the important thing is to remember that scientists (not science) can very easily stray towards making philosophical claims about the world. These claims are not science and should not be accepted as such.</li>
What is to be done about the science upon which evolution is based (chiefly geology and paleontology)? > Keep the science and ignore the philosophizing. My faith does not require a young Earth to hold together, so I see not big conflicts with my faith and these foundational sciences.
How does your viewpoint account for the problem of evil in the world?

Christianity assigns blame for evil and the decay of creation to human disobedience to God. Unfortunately, this view can raise additional hard questions that, at best, have uncomfortable answers. Although a difficult position to hold at times, I too will explain evil in the world as the result of human disobedience to God. I don’t have answers to all the questions this perspective raises, and in such cases, I will freely say, “I don’t know.”

Update: Part 1 and Part 2

Update: Due to migrating my blog to a commentless system, original comments on this discussion are reproduced below.

Tim Dolch

John–sorry I never got back to your blog to see you follow-up here. I’d like to send you my paper on causal closure, as we discussed, but I do not have your e-mail. I have started a blog too, as you can see.

I have a few reactions to your answers here. I would say if a discovery conflicts with scripture, it could be our interpretation of scripture that is wrong, or it could be that our “discovery” is more theoretical or conjectural than we suppose, and thus possible also an erroneus interpretation. Allowing correction in only one direction doesn’t seem right. Also, I think the definition of science from Wikipedia is too narrow; it suggests that science never makes hypotheses! It also conflates simulating phenomena with explaining how things work, probably because the latter has to involve hypothesizing entities or laws that are not just given in the phenomena. With that said, I canot agree that philosophy and science are so neatly separable. I am happy, though, that we seem to basically agree on #3 and #6.

I have recently had the chance to read portions of Roy Clouser’s The Myth of Religious Neutrality, which is all about scientific and philosophical theorizing. I heartily recommend it. Clouser is not a young-earth creationist, but his treatments of religion and theory are very lucid and worth consideration.

Housekeeping

2008-11-14T00:00:00+00:00

I have started getting spam comments on this blog (Mexican pain killers, anyone?), and so I have changed the commenting policy accordingly. Now, in order to post a comment, you must have a previously approved comment. I regret having to do this, but I do not want to simply respond to the spam comments—I want to be proactive in heading them off. For all you other bloggers out there, how do you handle spam comments?

Don’t worry, I will still respond to our ongoing discussion about the relation of science and theology. Stay tuned…

Darwin and Church History, part 2

2008-11-05T00:00:00+00:00

Charles Darwin’s ideas presented the church one of its greatest challenges. With his publication of Origin of Species in 1859, Darwin completed the Scientific Revolution begun by Copernicus and firmly divided science and theology in the Western world. The church’s response to this challenge has varied—liberals have changed the very meaning of Christianity and conservatives have all but rejected the advances of science. However, I do think there is a middle ground where one can both accept scientific truths and hold to orthodox Christian theology.

The most liberal view I have come across claims that the evolutionary process of nature itself reconciles us with God. In this view, the human mind, developed from lesser to greater to greater still, is the means to know God. As our mind developed from lesser species, our ability to comprehend God has similarly grown. This process will continue on as the human species continues to evolve and adapt.

The problem with this liberal view is that it completely negates the need for the Incarnation. This view focuses on higher moral values and has little need for sin or redemption (Bowler 166). Other views have claimed the Incarnation is the culmination of the entire process of nature, but again, the focus is on nature and natural processes as the means of spiritual renewal. These liberal viewpoints significantly depart from Christian orthodoxy on several important points.

On the other end of the spectrum, strict conservatives, often known as fundamentalists, embrace biblical literalism and treat the word therein as accurate scientific and historical truth. In the fundamentalist view, God has preserved Scripture as inerrant and a single departure from the clear meaning of the text erodes the very foundation upon which Christianity is built. If Genesis cannot be trusted as being true, how can we accept any part of Scripture as true?

The conservative viewpoint has trouble reconciling the clear meaning of conflicting passages in Scripture (from two Creation stories to what day Jesus was crucified) and understanding other problematic verses (pi = 3 from I Kings 7:23-26). Additionally, the fundamentalist perspective tends to limit scientific research and advancement in areas deemed sacred or unknowable. A fundamentalist view leads to a harsh, legalistic view of Christianity that ends up doing more harm to the Kingdom of God that good.

Both the liberal and fundamentalist views have merit, however. The liberal view encourages human progress and greater knowledge of our world. The fundamentalist view reminds us of the sacredness of Scripture and the need for a saving God.

Both sides’ advocates truly believe their side is correct, so any attempt to find a balance can be easily attacked by both camps. Nevertheless, I will be brave and press on.

A bit of housekeeping first: To find a Christian response to Darwinian evolution, one must recognize certain basic assumptions. One, every perspective in this debate is colored by context (Mike Stroope has some great thoughts on this topic). To remain civil, we must try to take other perspectives into consideration. Two, humanity has been struggling to reconcile theology and natural knowledge for millennia. Any conclusions we may draw will not end the debate.

Likewise, I refer to those that follow the strict conservative view as fundamentalists. This is not meant to be a derogatory term. I use it to differentiate from other, more moderate, conservative viewpoints and because these views rose to prominence as part of the fundamentalist movement of the 1920s.

Finally, to properly respond to Darwin, we must understand his ideas. The theory of evolution by means of natural selection was fist presented by Darwin in Origin of Species and has been further developed over the last one hundred fifty years. Evolution simply states that the vast diversity of life can be explained by small changes over time. Natural selection claims the evolutionary changes are the result of environmental pressures that “select” for certain attributes beneficial for the survival of an individual of group. It is an unguided process with no outside influence—it is random and has no final purpose.

As an example, consider a population of Jabberwocks. Suppose some members of the group develop a mutation that allows them to hide from their predators slightly more effectively than their peers. These Jabberwocks will be more likely to produce more offspring, and soon the entire colony of Jabberwocks will have acquired the helpful mutation. Similarly, if two populations of Jabberwocks are genetically separated (by a mountain range or large body of water, for example), each population will continue to develop their own unique characteristics. These characteristics may lead to an inability to mate and produce fertile young Jabberwocks; two new species are created from the one common ancestor.

Often, the theory of evolution is criticized because it is “just a theory.” A theory is just a guess, right? Not according to the National Academy of Sciences:

In everyday usage, “theory” often refers to a hunch or a speculation. When people say, “I have a theory about why that happened,” they are often drawing a conclusion based on fragmentary or inconclusive evidence. The formal scientific definition of theory is quite different from the everyday meaning of the word. It refers to a comprehensive explanation of some aspect of nature that is supported by a vast body of evidence.

So the theory of evolution is not just a vague idea about life. It is a well-established and well-researched explanation of the likely causes of the variation of life in the world around us.

“But wait,” you say. “That’s not good science. It’s not really science at all. We can’t go back in time and watch it happen, and we can’t duplicate evolution in a lab. The best you can hope for is to see small variation within a single species, but we certainly can’t see one species change into another.”

Like a courtroom trial, though, periodic observational evidence, as scientists have for evolution, is sufficient to provide a convincing explanation for past events. One may surmise a likely sequence of events based on observed evidence. If new evidence comes to light, however, the conclusion may be drastically altered. Evidence for evolution is abundant. For a high-level overview for the evidence of evolution see evolution.berkeley.edu or talkorigins.org. One specific study completed this year shows the evolution of a complex trait not generally considered a characteristic of the original species. Many other sites can provide similar information.

The observed evidence of evolution does not preclude the existence of God, nor does it negate the role of theology in our lives. Science is concerned with natural explanations of the natural world. The Christian God, by definition, is not of this world. Science cannot comment on the existence or non-existence of God. The absence of observed evidence does not conflict with a God that is essentially “other.” Fundamentally, science and theology are concerned with two separate parts of human life: science can answer the questions of how or what and theology gives meaning to answers.

It is possible for science and theology to coexist. Science cannot confirm religious belief, but it can confirm or deny facts about religious belief. Likewise, religion can judge science by guiding the direction of study or commenting on the meaning of scientific truths (Sweet 225-227).

In my opinion, science and theology go together like architecture and aesthetics. Solid architecture does not mean a structure is beautiful; likewise, a structure’s aesthetic appeal does speak to its structural soundness. One may admire the architectural principles of a building or see order in its aesthetic properties, but great structures require attention to both architectural and aesthetic principles. Similarly, individuals must incorporate scientific principles and religious truth to determine meaning and purpose in life.

Darwin Bibliography (pdf): bibliography for my research on the paper I recently wrote on a similar topic

Darwin paper (pdf): a paper I recently wrote that started this whole discussion

Stay tuned for more. I’ve been accused of always sitting on the fence, so I’ll try to follow up with more assertive statements about how science and theology can work together.

Update: Part 1 and Part 3

Update: Due to migrating my blog to a commentless system, original comments on this discussion are reproduced below.

Tim Dolch

I just posted my response to your last blog and then noticed this one. I have some concerns, which I hope you’ll consider.

My biggest concern, I guess, is that based on your bibliography it seems your sources for creation are all theological (e.g. MacArthur’s sermons). If you look at scientific sources on evolution, but only theological sources on creation, then it will appear that evolution is science and creation is religion. You mention the talk.origins archive; have you looked, for example, at the true.origins archive run by creation scientists?

Concerning science, creationists accept natural selection and even the existence of beneficial mutations. What they claim is undiscovered and undiscoverable is a natural process that produces new genetic information. In the bacteria example that you cite, the article states that the precise genetic change is not yet known. The quote at the end of the article, that complex traits are what creationist say can’t happen, is not correct. The jury is still out on whether this bacteria produced new genetic information.

I have to object to your claim that conservatives have trouble reconciling Gen 1-2, and with the pi issue. These are manufactured problems, not real ones. For starters, take a look at http://www.answersingenesis.org/creation/v17/i2/pi.asp and http://www.tektonics.org/jedp/creationtwo.html. Also, I recommed Jordan’s In Six Days, which gives a very sophisticated young-earth interpretation of Gen 1-2.

I challenge you to produce one example support your claim that “the fundamentalist perspective tends to limit scientific research and advancement in areas deemed sacred or unknowable.”

At the end, I understand you, to be taking the non-overlapping magisteria view (NOMA, coined by S.J. Gould), that science and religion are concerned with absolutely separate topics, fact and meaning. This view is problematic philosophically because the fact/value distinction is controversial. It is problematic theologically because it sounds like gnosticism. To say that God is “essentially other” suggests the view of Marcion and others that God is totally separate from the world. Christianity says God is distinct from the the world, but also that God created the world, and that the world to some extent bears witness to its creator.

I do think you are on the right track in emphasizing the incarnation and that God saves.

I should add that I like your architecture/aesthetic illustration. The relationship is not just one of complementarity, though, but of foundation. Architectrual soundness is foundational to aesthic beauty, in that you have to have a sound structure to have a beautiful structure, but you don’t need beauty to have a sound structure.

Likewise with Christianity, the historical facts are foundational to the religious meaning. The early creeds are basically claims about historical fact. You acknowledge the importance of the factuality of the resurrection. Young-earthers contend that the historical factuality of Adam’s fall with death as its consequence is also an essential historical fact, and that all of the facts of science cohere with it.

myself

Tim, thanks for your insights. As a postscript to my post, I will agree with you in that most issues of reconciling Scripture (like that of pi = 3) are “manufactured issues” and pose no real challenge to the validity of Scripture. My point is that strict literal readings of the text rarely happen, and even the most hard-core biblical literalist still will interpret to a degree. For example, Romans says that if you confess with your mouth, you will be saved. But what about people who can’t speak? Can they be saved? I think that is a silly conclusion, but unless one interprets the meaning and message behind the text, such silly conclusions can be made (similarly, pi = 3).

As a rebuttal to your thoughts on the E. coli study, the scientists said they don’t know the exact genetic change(s) the bacteria made. They did not say they don’t have an explanation for the mechanism of change. Also, there is a very easy example of a natural process producing new genetic information: Down’s syndrome.

Tim Dolch

Thanks for replying.

Throughout Christian history literal interpretation has meant according to the intention of the (human) author. Thus Augustine’s De Genesi ad Litteram, and the fourfold way of literal-moral-analogical-anagogical. I don’t know anyone who claims literal = noninterpretive.

How can you explain the mechanism for a change when you don’t know what the change was? Only in general terms like “natural selection.” But what, exactly, was selected, and where did it come from?

Down’s syndrome duplicates a chromosome, ie pre-existing information. It does not produce any new information. And it is hardly adaptive. Nice suggestion though.

Jon Norvell

Read the paper…I’ll comment on a thing or two later. For now, I was wondering if you ran across anything about Erasmus Darwin, Charles Darwin’s grandfather? I just read a great deal about him in Roy Porter’s The Creation of the Modern World: The Untold Story of the British Enlightenment. He actually proposed a strikingly similar theory of evolution to that of his grandson, though with a different engine. Apparently, no one has really written about this connection. Just thought you might be interested.

myself

@Jon: I linked the paper because it goes in to more depth about the response of the church to Darwin’s ideas. Please disregard the last few pages where I speculate about what would be different had Darwin not lived. I only wrote that to fulfill the requirements of the assignment. I am not willing to defend any of that speculation.

Charles was aware of his grandfather’s theory, at least in a superficial sense (see page 49 in Darwin’s Autobiography). You are right in that Charles did not “invent” evolution; Lamarck, E. Darwin, and others had proposed theories of the development of life that involved evolution. See Appendix One in Darwin’s Autobiography for more info on Charles’s relationship with these other authors. The significance of Origin is not evolution. Evolution was a well-known idea and commonly accepted in the nineteenth century. Darwin’s work is significant for the mechanism of evolution proposed: natural selection—a process describable without the need for the Divine.

@Tim: It’s quite common to know the mechanism of change without knowing the exact change that takes place. For a simplistic example, remember the old hypercolor t-shirts? Put you hand on it or put it in the sun and the color changes. You and I know that something changed, and that the change is caused by heat, but we don’t know the exact chemistry behind the change. Similarly, we know that genetic information controls an organism (I’m not getting into a nature/nurture debate) and that the organism’s behavior has changed (the ability to metabolize citrate). Although not knowing the exact codons that changed in the organism, a convincing explanation for the mechanism of change is natural selection. It is a model that closely fits the observed evidence and predicts the change that did in fact occur.

I used Down’s syndrome as an example of a mutation leading to new genetic information, not because it is a beneficial mutation, but because it is commonly known. Any mutation that duplicates genetic material produces new genetic information. Perhaps what you mean is that duplication does not produce any unique information. That may be true, but only for the first generation. If the trait gets passed down to other generations, small mutations in the duplicated genetic material may end up coding for some beneficial trait. Of course, and more likely, the mutation will either be non-affective or harmful. But over time, the originally duplicated genetic material changes independently from the original source material, and thus new information is produced. For Down’s syndrome, the new information coded by the duplicate genetic material can be seen by its effects on the individual. If there were no new information coded in trisomy-21, there would be no differences in individuals with it and individuals without it (aside from general variation from person to person).

Tim Dolch

John, I would not describe knowing that sun changes the color of a hyper-color shirt as knowing a mechanism. I would describe it as knowing a phenomenal pattern. Natural selection is like that, a phenomenal pattern. We can observe it every population, and Darwin is rightfully famous for describing it and recognizing how it accounts for biodiversity. But “accounts” here has to be understood in a limited sense. Observing patterns is part of scientific knowledge, but patterns themselves need to be accounted for. Darwin had no explanation of where new traits come from and how inhertance works (genetics). His extrapolation from shifting traits within populations to common descent of different phyla and orders was simply imaginative speculation. So I have no doubt that the citrate metabolism is subject to natural selection, I just don’t think that saying so actually says much at all. It’s almost like saying “God willed it”–yeah, that surely is true, but it is not news to anyone.

“Perhaps what you mean is that duplication does not produce any unique information.” Yes. By “new information” creationists mean a new DNA sequence-type, not a sequence-token.

I understand, as you describe it, how (token-)duplication can be conceived–imagined–as a step in the production of a new, beneficial sequence-type. Maybe that is what happened with the citrate metabolism–if it could be proved, it would be very significant. (Although it still might not be legitimate to extrapolate to eukaryotes.)

But I am not sure that even non-harmful changes in a duplicated DNA sequence could in principle produce a new information-type. Information is more than a unique sequence of codons or letters. Mutations over time can only produce a random sequence, and I think statistics and information theory can prove that random sequencing will not produce meaningful information regardless of the time allowed.

I myself am not able to take the discussion any farther than this, but this article: http://www.trueorigin.org/schneider.asp looks to me like the latest scientific creationist response on the issue.

Justin Allison

I’m pretty sure that most Old Testament scholars would say that Genesis 1 and 2 are indeed different stories. Off the top of my head I can name a few evangelicals – Bruce Waltke, John Goldingay, Tim Pierce, even several SWBTS professors.

One Old Testament scholar (not a conservative), Terrence Fretheim, says that we need to take scientific research into account when we study Genesis so that we get a holistic picture.

Interestingly, John Goldingay who teaches at Fuller Theological seminary would be an example of one who subscribes to God-guided evolutionary processes. This debate is of course much less important amongst more liberal OT scholars such as Terrence Fretheim who claims that the evolutionary process is a part of the way that God is still creating today (for many liberals creation is not a completed project, but a continuing process).

I’d like to say more, but I have a crying kid…

Darwin and Church History

2008-10-22T00:00:00+00:00

I’m currently writing a paper on Charles Darwin and his influence on church history. What is your perspective on the relation of science and theology? If I get some interesting responses, I’ll follow up with my thoughts.

Questions, in no particular order, to spur your thinking:

Must one choose between young-earth creationism and an atheistic interpretation of evolution by natural selection? Is there a middle ground?
Is your view consistent with as-yet-undiscovered scientific facts? In other words, is your argument based on a current unknown remaining unknown?
Supposing that the modern understanding of evolution is true, life, even sentient life, may have developed some other place in the universe. How does your theology handle this possibility?
What is the definition of science? How does one determine if darwinian evolution or intelligent design (or any other theory) is a valid scientific theory?
What is to be done about the science upon which evolution is based (chiefly geology and paleontology)?
How does your viewpoint account for the problem of evil in the world?

Update: Part 2 and Part 3

Update: Due to migrating my blog to a commentless system, original comments on this discussion are reproduced below.

Jon Norvell

I’ve tried to answer your questions as best I could on my blog. I needed something to jumpstart my own writing/thinking.

myself

I will be pretty busy for the next few days, but I will try to respond as soon as possible. I admit that question 3 is a little biased, sorry about that. If one takes the premise that sentient life only exists as a special act of God here on earth, the question makes no sense. As a quick follow-up question (until I have time to post more), by which method do you see science progressing: inductive reasoning or deductive reasoning?

Lydia Norvell

Before I read Jon’s answers, I’ll give my very short answers here:

I do not believe there is a middle ground. I would not have answered this way even 6 months ago, but I can’t take pieces out of the Bible and still believe in the rest of the Bible; therefore I must believe in young-earth creationism. I do believe in evolution within a species, but not cross-species, which is what Darwin proposes.
No. God can work miracles without and within science.
Hm…I think that God has a redeeming plan for all, and if that includes Martians, then so be it.
Science needs to be replicable. However, both theories are as improbable as the other in regards to replication. So, a theory needs to be supported by scientific facts.
I’d question the presuppositions that these sciences have. Science is a lot more a belief system than we’d like to admit.
Unlike Darwinianism, the Christian worldview actually considers morality. The problem of evil is very real, and God redeems us individually, and will redeem the world at the end.

Justin Allison

I think that there is a sort of middle ground. For me, the creation stories came about as a response to several different things. First, I think that they express the truth that God is the creator. My view of inspiration allows for God to give this truth to humankind and allow them to write this in their own fashion. In Genesis this takes the shape of 2 creation stories, both of which I believe are polemical against the Babylonian understanding of the time. They are also highly stylized writings, which show a unique thought and way of expressing that thought. So, I believe the creation stories because I believe that their point is that God created the world (universe) and everything in it.

I don’t really understand this question… I think that creation stories are an un-scientific way of expressing something that has been divinely revealed, namely that God is the creator of what we see/experience. Scientific explanations utilize another form of speech/rhetoric and even worldview which results in its own story of origins based upon it’s implied ideology.
My theology would not be changed by a discovery of life somewhere else in the universe. As stated, I believe God created all that exists, and has created life in general. So, any form of life or matter was created by God.
I think science has to do with observing things which happen repeatedly and making deductive conclusions about that behavior. As such, I believe science to be limited, and even inaccurate in regard to it’s theories of the past. After all, theories of the past are not repeatable or deductive. They are instead conjecture and inductive – which is not the realm of science in my understanding.
Based upon my answer to #4, you can see where I’m going. “science of the past” is much less authoritative than something like physics. It involves a lot of inductive reasoning, which can only prove plausibility at best. Geology, Archeology, etc., can only recommend scenarios of the past. They cannot PROVE something without doubt. For more info on this view check out the beginning section Ian Provan’s book: A Biblical History of Israel. The beginning is the best part.
I’m still working on a reasoned judgment of evil in the world. I don’t want to incriminate myself by making a statement I’ll regret later or merely give an opinion. “I don’t know” is the best I can do.

Tim Dolch

John: Margie saw your blog and suggested I post a response. I’ve tried to limit it, and I would be open to further discussion of my claims. You say you paper is on Darwin and the church, but your questions concern science, theology and philosophy, not church history. Even if you’ve already written your paper, I hope my answers are helpful to you.

One can be a theistic evolutionist. Some Christians, especially Catholics, link evolution to the neoplatontic and stoic ideas incorporated into theology by Augustine, Aquinas and others. (God as the One, or highest being, from whom the world emanates and develops according to the potencies of being.) However, I think Biblical Christianity and evolution are inconsistent in at least 3 major ways.

First, the uniquely Christian doctrine of creation ex nihilo, and its corollary, the distinction between God and the world, conflict with evolution’s requirement that natural forces can explain the origin of the universe, life from nonliving matter, intelligence from unintelligent life, etc.

Second, the Bible presents the creation and flood narratives as history. Many people read the Biblical narratives as teaching general ideas rather than historical facts as such (i.e. Gen. 1-2 teaches that God created, but not how). I think it can be shown that that such readings a) are basically Gnostic, b) prevent us from seeing many elements of the texts, or the richness of Biblical theology, c) post-date evolutionary theory and thus seem to be merely easy ways to avoid dealing with the conflict of histories rather than serious attempts at exegesis, and d) actually undermine the general ideas that are supposed to be the real teachings. James Jordan’s In Six Days is excellent on these issues.

Third, the doctrines of the fall of man and the atonement conflict with evolutionary history. The Bible presents the death of man and the higher (breathing) animals as the result of Adam’s sin, and the death of the sinless Christ as bearing the consequences of sin, so that believers gain eternal life because Christ has died for them. Evolution, on the other hand, treats death as a consequence of a struggle for existence and as the instrument of natural selection, so that new forms evolve in part through the death of inferior forms. If evolutionary history is true, then the death of Christ loses its meaning because then death is not an enemy and not the consequence of sin. Thomas Huxley’s essay, “Lights of the Church and the Light of Science,” presses this point nicely.
There are two kinds of unknowns in science: causes and facts. In regard to causes, unknowns are sometimes covered by “gap” arguments. “God of the gaps” arguments propose God as a supernatural cause when the efficient cause is unknown. “God of the gaps” thinking is not only unscientific, in that it tempts us to give up on seeking natural explanations, but it is also bad theology, because it severely limits God’s role in the universe, and makes God less and less significant as science progresses. I take the view of the Westminster Confession of Faith 3.1, that “God from all eternity did…freely, and unchangeably ordain whatsoever comes to pass: yet so, as thereby…the liberty or contingency of second causes [is not] taken away, but rather established.” In other words, God co-causes everything, and even establishes all of the natural laws that science investigates. “God did it” is not an alternative to scientific explanation, but rather is a different level of explanation.

There is also “nature of the gaps” thinking, where a label or story is applied to a phenomena as if it explains it, when in fact the label explains nothing or the story fails to establish its own possibility in terms of known mechanisms. The strong nuclear force in physics, abiogenesis, and Aristotelian natures are labels that explain nothing. Most macro-evolutionary theory (e.g. birds from dinosaurs) is simply imaginative storytelling, no less unscientific than “God did it.”

By unknown facts I mean things a theory predicts will not exist or happen. “Undiscoverables,” if you will. Undiscoverables based on known causes and patterns, are of the essence of science, because understanding what is not possible is the flip-side of understanding how something works. Consider conservation laws, or any equation for that matter: they are informative precisely insofar as they limit what we can expect to observe.

Young-earth creation predicts we won’t find transitional forms or constructive mutations, based to some extent on Genesis, but more technically on what we know about the fossil record, genetics and information theory. By contrast, Darwin put forward his theory in spite of the “imperfection of the geologic record” and ignorance of genetics. A century after Darwin the evidence still had not materialized, and so “punctuated equilibrium” was proposed to explain why we shouldn’t expect to find the evidence. Evolutionary theory has also predicted undiscoverables, e.g. “living fossils,” rapid formation of geological and paleontological artifacts (strata, fossils, petroleum), and functions for vestigial organs and “junk DNA”—all of which have been discovered. Evolution is so broad that it can change to accommodate these unexpected finds, but this is not a virtue, because the more a concept explains the less useful it is scientifically. Repeatedly, belief in evolution has led to inaccurate assumptions, futile paths of inquiry, and counterproductive medical practice. Biblical creation and evolution are both large-scale frameworks for science and both involve unknowns and “undiscoverables,” but I think creation, because it has a category for the omnipotent supernatural, disposes us to more sober expectations and assessment of the natural.
I think intelligent alien life is another “undiscoverable” predicted by Christianity, so I suppose I would give up Christianity if it were discovered. It is worth noting that the search for intelligent alien life presupposes that we will be able to recognize the effects of intelligence when we find them. ID is certainly not any less scientific than the search for intelligent alien life.
I think science involves 2 things: theoretical explanation in terms of efficient causes, and a rigorous method. Ideally the method is controlled experimentation, but where this is not possible, rigorous induction can approximate the certainty provided by controlled experiment. Historical sciences like geology, where theories cannot be directly tested, rely on induction.

Are ID or evolution valid scientific theories? Sort of. Both are historical in nature. ID uses forensic methods to infer design; molecules-to-man evolution theorizes concerning historical relationships of descent. ID is sort of scientific because it uses an established, inductive method; evolution is sort of scientific because it pursues explanation in terms of efficient causes. Neither, it seems to me, contributes much to scientific progress in terms of laboratory results. Evolution in the sense of the common descent of all living forms from non-living matter has nothing to do with laboratory progress in biochemistry or physiology, or even with practical work in population biology.

Science implies a degree of functional naturalism, because we search for natural (efficient) causes and in experiments we can only control natural causes. However, experimental design and interpretation requires speculation or presupposition as to what natural causes exist and how they might function. So science always operates in a super-scientific framework. The claim that evolution is central to science because evolution is naturalistic is wrong, because the functional naturalism of experimentation neither presupposes nor implies real, universal naturalism. Functional naturalism fits better in the framework of Christian theism, as I mentioned at the end of answer #2.
Creation scientists have done a lot of interesting work in geology, e.g. John Baumgardner’s modeling of catastrophic plate techtonics, Steve Austin’s work on the Western U.S. including Mt. St. Helens and Grand Canyon, Michael Oard’s An Ice Age Caused by the Genesis Flood, and the Radioisotopes and the Age of the Earth (RATE) project. But as to the evidence typically claimed for evolution:

First, geology and paleontology really point to the world-wide flood more than to long periods of time: there are billions of dead things buried in rock layers laid down by water all over the earth. Some strata are continuous for hundreds of square miles. Only something on scale of the global flood and its aftermath can explain that.

Second, Darwin’s theory relied on Lyell’s uniformitarian geology, with its principle that all geological formations can be explained by the slow, gradual operation of present phenomena over long ages. From 20th century observations of the power of floods and volcanoes to produce rapid, catastrophic geologic changes we know that long ages are neither necessary nor likely explanations of geologic phenomena.

Third, there is a problem of circular reason between these fields. Rock strata are dated by the fossils in them, and fossils are dated by the rock strata in which they are found. There is an established outline, the evolutionary geologic column, into which all new discoveries are fit by means of this circular reasoning without any serious questioning of the outline—despite the fact that the established column cannot be found complete anywhere in the world.

Fourth, there are several problems with radioactive dating methods: they require a) assuming the amount of starting material, and b) assuming the lack of any external influences (e.g. loss or gain of starting material or products) over the supposed thousands or millions of years. Also, c) in practice they give conflicting results and they conflict with other dating methods.

Fifth, fame and grant money (particularly in the hominid subfield) have driven some paleontologists to propose interpretations that exaggerate the evolutionary significance of their empirical findings, or even to create frauds. Over time some of these things have been cleaned up—almost none of the evidence presented at the Scopes Trial is still accepted today—but some dubious interpretations remain part of the popular understanding of evolution. Creationists argue, for instance, that Cro-Magnon man, Neanderthal, and most homo erectus are humans with anatomical variations that are within the known range today’s population.
Evil is a moral category because God is good, God defines what is good, and God judges between good and its opposite, evil. Evil originated in the rebellion of Satan, and it entered the world and human experience through Adam’s fall. It is because of Adam’s fall that we experience death, disease and suffering, and see these things in the natural world. Evil is an enemy; it does not in itself ultimately serve a good purpose. But evil is not omnipotent; God makes all evil serve good purposes for those who love him, as seen most clearly in Christ’s atoning death on the cross. Through the redemption of man God is working to eradicate evil from the universe.

In Darwin’s God Cornelius Hunter shows that evolutionists from Darwin to the present have always relied on the argument that a good God would not make the world as it is, therefore evolution is a better explanation than divine design for how the world came to be. This Darwinist argument is based on the radically benevolent God (RBG) of Paley and other natural theologians, whose goodness is supposed to be evident throughout the natural world. But the RBG is not the God of the Bible who judges his enemies, and natural theology does not reckon with Paul’s description of the fallenness of the natural world in Romans 1 and 8. Therefore, the “God wouldn’t have made things this way” argument of evolutionists is a straw man argument against Biblical creation.

The Moral Mind (continued)

2008-09-23T00:00:00+00:00

Today I came across an article by the same guy giving the talk in my last post (Jonathan Haidt). Although it says mostly the same thing as the Ted talk, it goes in to much more detail about why Americans vote the way they do, and, specifically, what Democrats can do to attract more people to their side. The article is entitled “What Makes People Vote Republican?”.

The moral mind

2008-09-18T00:00:00+00:00

I came across an interesting video today about the difference between conservatives and liberals and the underpinnings of the moral mind. Take twenty minutes and watch it.

http://www.ted.com/index.php/talks/jonathan_haidt_on_the_moral_mind.html

A good thought

2008-09-17T00:00:00+00:00

I came across a good quote today:

“Easy” is a word you use to describe other people’s jobs.

I found it in an article about 37signals.

Friday fun at work

2008-09-12T00:00:00+00:00

Ok, ok, my work is probably not fun to most people, but on Fridays I like to do something a little different to keep things interesting. Today I made passive ethernet tap to use with intrusion detection software on the office network. Aside from striping a bunch of Cat5 (anyone know a good way to strip the middle part of a wire?), it wasn’t too hard to put together. It reminded me a lot of my days working for the cable company. And I was pretty happy when it worked the first time I tried it.

Dropbox has been released

2008-09-11T00:00:00+00:00

Dropbox, a great little tool to synchronize or share or publish files between multiple computers was released today. I’ve been using it for a few months now. Lydia uses it at work. It’s one of the slickest tools I’ve seen in a while. If you haven’t heard of it before, head over to http://getdropbox.com and check it out (the video is short and quite informative).

Now with OpenID goodness

2008-07-16T00:00:00+00:00

This blog now supports OpenID for all your authentication needs.

My iTunes

2008-07-10T00:00:00+00:00

iTunes now lets you create a little widget that will show various things about the music you have. The widget I have added to the sidebar shows the artists I have purchased most from the iTunes music store my most recent iTunes purchases.

Things to do with Ian

2008-06-24T00:00:00+00:00

A list I made up a few months ago.

Things that we should to with Ian while he is young (less than 10):

watch a trial
build something
go to the zoo
go to the park
go to a lake
go to the mountains
go to the ocean
ride a train
ride a taxi
ride a bus
ride a ferry
ride a plane
visit a construction site

Ian is here

2008-02-25T00:00:00+00:00

February 21, 2008

7 pounds 13 ounces

20.5 inches

New look, new links

2008-02-05T00:00:00+00:00

Getting a decent theme that does everything I want while still looking good seems to be a bit of a challenge. Maybe someday I’ll get ambitious and make my own.

I’ve added a few new links to the sidebar. Project Euler.is a site that has a few math problems on it that can be solved quickly with a little programming skill. For example, find the 10001^st prime number (104743, by the way). It’s a great site to go if you like mind puzzles or want to brush up on some math or learn some simple programming.

I also added a music section that has iTunes links to artists and songs I have recently come across and liked. Your taste in music may not be the same as mine, but, hey, that’s what makes the world interesting, right?

Frist Ps0t

2008-01-29T00:00:00+00:00

w00t

Programmer Thoughts

Swift Tech Overview

Distributed

Proxy

Storage Servers

Consistency Servers

Fault-tolerant

Eventually Consistent

Example Request Flow

Client Data Designs

Summary

Swift State of the Project Spring 2012

Where We Are

Where We’re Going

Storage Systems Overview

Types of Storage

Block Storage

File Storage

Object Storage

CAP Theorem

Object Storage Use Cases

Choose the Right System

Democratization of Data

Swift State of the Project Fall 2011

Setting Permissions (ACLs) on Openstack Swift containers

Quickly Uploading Data To Cloud Files, Part 2

A Vision of the Future of Storage

The Story of an Openstack Feature

Simple script to upload content to Cloud Files

Features I Would Like to See in Swift

Swift (OpenStack Object Storage) Overview

git vs. bzr

Intro to VCS and Workflows

Server-side Object Copy in OpenStack storage

I'm a dad twice over

Compressed File Reader

Deleted File Recovery

Cloud Files CDN Stats

Nested folders in Cloud Files

Quickly uploading data to Cloud Files

Cloud Files Object Copy

When building a starship...

South with the Sun to San Antonio

PyGTK Chart widget beta release

OpenSolaris upgrade

PyGTK Chart Widget

Useful Tools

Friday fun at work, part 2

A little puzzle

Network Shares

More adventures of home tech support

Three tier python imports

Setting Up Zfs

Power supplies, hard drives, and network drivers, oh my!

Testing And Hardware

File server update

Finding a good storage solution

Wordle tag cloud

Darwin and Church History, Part 3

Housekeeping

Darwin and Church History, part 2

Darwin and Church History

The Moral Mind (continued)

The moral mind

A good thought

Friday fun at work

Dropbox has been released

Now with OpenID goodness

My iTunes

Things to do with Ian

More Ian links

Ian is here

New look, new links

Frist Ps0t