Archive for February, 2009

Three tier python imports

Feb 28th

Posted by John in programming

1 comment

Suppose you want to write a python module, and, like a good software developer, you want to keep a separation between your development, model/testing, and production versions of your code. This is a simple task with a few tricks in __init__.py.

First, our tester code:

1
2
3

import module
x = module.test.Klass()
print x.status()

module/_dev/test.py is similarly simple (the model and prod versions respectively return “model” and “prod” instead of “dev”):

1
2
3

class Klass(object):
    def status(self):
        return '%s in dev' % str(self)

module/__init__.py:

1
2
3
4
5
6
7
8
9
10

import os
 
env = os.environ.get('ENV','dev').lower()
 
if env == 'prod':
    from _prod import *
elif env == 'model':
    from _model import *
else:
    from _dev import *

__init__.py in _dev, _model, and _prod:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import os
 
def _load_code():
    parent_dir = os.path.dirname(__file__)
    base = os.path.basename(os.path.dirname(parent_dir))
    modules = []
    for root, dirs, files in os.walk(parent_dir):
    for name in files:
        if name.endswith('.py') and \
          not name.startswith('__') and \
          not name.startswith('.'):
            module_name = '%s.%s.%s' % (base,
                        os.path.basename(parent_dir),
                        name.rsplit('.', 1)[0].replace(os.sep, '.'))
            __import__(module_name)
            modules.append(module_name)
    return modules
 
__all__ = []
 
def update__all__():
    global __all__
    __all__ = [x.rsplit('.',1)[-1] for x in _load_code()]
 
update__all__() # intial setup

When tester.py is run, it checks the environment variable ENV and imports the proper version of the file.

$ ENV=dev python ./tester.py
<module._dev.test.Klass object at 0x9d66e2c> in dev
$ ENV=model python ./tester.py
<module._model.test.Klass object at 0x9435e2c> in model
$ ENV=prod python ./tester.py
<module._prod.test.Klass object at 0x9f12e2c> in prod

Full source code.

, ,

Power supplies, hard drives, and network drivers, oh my!

Feb 17th

Posted by John in home storage

3 comments

I think I finally have a working file server.

The replacement power supply is working great. I’m not sure what happened to the first one. Once I had the server up and running, I started doing some tests of copying data over. My initial tests seemed successful, but I started getting some errors reported from one of the hard drives. A quick replacement from Seagate, and things seem to be going well. My zpool is resilvering the new drive now.

After my initial tests, I tried copying over some larger data sets (my 153GB iTunes library). Initially, things seemed to go fine, but, after a while, the network connection dropped. After quite a bit of searching online, I ran across some fantastic info about this exact problem. I tried his solution and things, so far, are working well. I have copied nearly 200GB of data to the file server so far. If things keep working reliably for the next week or so, I’ll start moving the data to the file server and removing it from my computer.

I’ve had to abandon the OpenSAN iSCSI initiator that I was hoping to use for Time Machine backups. That initiator seems to be buggy in OS 10.5, and it kept causing my computer to hang. Although I like the idea of using iSCSI for backups, there is one big drawback to that method. The OS X install CD does not have an iSCSI initiator, so it would be tricky at best to restore from Time Machine backups stored on an iSCSI target. My next task will be to get Time Machine to work over NFS.

, , , , , , , ,

File server update

Feb 6th

Posted by John in home storage

No comments

Unfortunately, I haven’t made much progress on my new file server. I got OpenSolaris installed last weekend. After I figured out that I had accidentally turned off the SATA controllers in the BIOS, the install went flawlessly.

I didn’t set up the storage pool right after I installed OpenSolaris because I didn’t have all of my hard drives. When I first ordered my four hard drives for the Drobo, one of them was bad, so I sent it back for a replacement. The replacement came on Monday. I opened up the case, screwed in the hard drive, plugged in the cables, neatly arranged everything, and closed the case back up. I pushed the power button, and…nothing. No beeps, no clicks, no anything. Quite disappointed, I opened up the case again and took a look. Nothing smelled bad, and I couldn’t see anything that looked burned, so I assumed the magic blue smoke that makes all electronics work was still inside. I pulled the power supply (no easy task) and tried it in a computer at work. No change. I boxed up the supply and sent it back to get replaced. UPS says it was delivered today, so I hope to get a new one next week.

Karen and I finished watching season two of Battlestar Galactica last night. Unfortunately, without a working file server, I don’t have enough hard drive space on my computer to download season three. I hope to get the replacement power supply soon.

I’ve added a page that gives more specific details on setting up my home file server, including parts, screenshots, and example commands.

, , ,