My Software Development Life Cycle

SDLC Document

Python str with custom truth values

I've run into multiple instances where I get a string from some external service like a config file or a database.  There are values that I want to treat as true and others as false for the purposes of logic in my code.  For example let's say I have a status string with a few possible values, some false and some true.

1. "Yes", "on", "true"
2. "No", "off", "false"

I can map the values to True or False in Python.  Now I just need a string I can configure what it's truth values are.

class BooleanString(str):
    def __nonzero__(self):
        return self.lower() in ('on', 'yes', 'true') 

bool(BooleanString("YES")) # return True
bool(BooleanString("some other value")) # returns False

The thing is I don't want to hard code the truth values in the class definition.  I want to pass them in to the creation of the derived class.

def mkboolstr(truth):
    def __nonzero__(self):
        return self.lower() in truth
    return type('BooleanString', (str, ), dict(__nonzero__=__nonzero__))

BooleanString = mkboolstr(('on', 'yes', 'true'))

bool(BooleanString("TRUE")) # return True
bool(BooleanString("some other value")) # returns False

And now we have a function which creates a BooleanString class using a parameterized string as the comparable truth.

§

Python Expression Idiom: Merging Dictionaries

It's common in Python programming to need to merge 2 or more dictionaries together. 

The first idiom is using the dict constructor.  This idiom has it's limitations, however it will always work fine as long as the keys are all strings. Trying this with non-string keys will fail in Python 3.2 and later, and also fails in alternate Python implementations. The idiom itself is frowned upon, although I think for the wrong reasons.

d1 = dict(a=1, b=2, c=3)
d2 = dict(c=4, d=5, e=6)

# merging 2 dicts with the dict constructor
merged_dict = dict(d1, **d2)

# merging n dicts with the dict constructor
merged_dict = reduce(lambda a, b: dict(a, **b), (d1, d1))

There are also other choices which will work with any type of key. Unfortunately they require a tad more code.

# merging n dicts with a generator comprehension
merged_dict = dict(i for iterator in (d1, d2) for i in iterator.iteritems())

# merging n dicts with dict comprehension
merged_dict = {k:v for iterator in (d1, d2) for k, v in iterator.iteritems()}

UPDATE:

I have left out dict.update because it is only usable in a statement, not an expression. It also modifies a dictionary which may not be what I want to do. You can compare:

def fn1(d1, d2):
   d3 = d1.copy()
   d3.update(d2)
   return d3

# vs

return dict(d1, **d2)

# or

return {k:v for d in (d1, d2) for k, v in d.iteritems()}

I prefer the expressions.

§

Python Expression Idiom: Dict Slicing

As anyone who comes from Perl can tell you hash slicing is useful.  Python dicts do not natively support slicing.  One of the issues is slicing in Python seems limited to defined ranges, rather than an ad-hoc collection of values.

Take heart! There is an expression that has the same effect in Python as Perl's hash slicing:

from operator import itemgetter

d = dict(a=1, b=2, c=3, d=4)
h, i, j, k = itemgetter('a', 'c', 'a', 'd')(d)
# h, i, j, k => 1, 3, 1, 4

Alternate solutions are usually more complicated expressions.

d = dict(a=1, b=2, c=3, d=4)

# conceptually more complicated expressions
h, i, j, k = (d[i] for i in ('a', 'c', 'a', 'd'))
h, i, j, k = map(d.get, ('a', 'c', 'a', 'd'))

itemgetter provides a more simple expression for extracting multiple ad-hoc values from a dictionary.

§

Python: Encapsulating Exceptions with Context Managers

Try/except can often interrupt the flow of logic making code harder to read. Take for example the following piece of code:

import sys

class Car(object):
   def create(self, color, stereo):
      try:
         vin = self._factory.make_car(color)
      except FactoryColorError, err:
         stacktrace = sys.exc_info()[2]
         raise ValidationError(err.message), None, stacktrace

      try:
         self._customizer.update_car(vin, stereo)
      except CustomizerError, err:
         self._factory.remove_car(vin)
         stacktrace = sys.exc_info()[2]
         raise ValidationError(err.message), None, stacktrace

      return vin

   def update(self, vin, color, stereo):
      try:
         self._factory.update_car_color(vin, color)
      except FactoryColorError, err:
         stacktrace = sys.exc_info()[2]
         raise ValidationError(err.message), None, stacktrace
 
      try:
         self._customizer.update_car(vin, stereo)
      except CustomizerError, err:
         stacktrace = sys.exc_info()[2]
         raise ValidationError(err.message), None, stacktrace

      return

A somewhat typical piece of code where the create and update are composed of multiple calls on the backend.  So the logic is as follows:

• exception logic that changes third party exceptions into ValidationError
• rollback exception logic in create that removes the car if the stereo option is invalid

The code is cluttered and the exception handling obscures the flow of both create and update.  The more exception logic we have the less readable the code is.  We also have some duplication of code. Enter context managers.

from contextlib import contextmanager
import sys

class Car(object):
   @contextmanager
   def _factory_error_handler(self):
      try:
         yield
      except FactoryColorError, err:
         stacktrace = sys.exc_info()[2]
         raise ValidationError(err.message, None, stacktrace

   @contextmanager
   def _create_customizer_error_handler(self, vin):
      try:
         yield
      except CustomizerError, err:
         self._factory.remove_car(vin)
         stacktrace = sys.exc_info()[2]
         raise ValidationError(err.message), None, stacktrace

   @contextmanager
   def _update_customizer_error_handler(self, vin):
      try:
         yield
      except CustomizerError, err:
         stacktrace = sys.exc_info()[2]
         raise ValidationError(err.message), None, stacktrace

   def create(self, color, stereo):
      with self._factory_error_handler():
         vin = self._factory.make_car(color)

      with self._create_customizer_error_handler(vin):
         self._customizer.update_car(vin, stereo)

      return vin

   def update(self, vin, color, stereo):
      with self._factory_error_handler():
         self._factory.update_car_color(vin, color)
 
      with self._update_customizer_error_handler(vin):
         self._customizer.update_car(vin, stereo)

      return

Refactoring the exception code to encapsulate the logic in it's own namespace in order to make the code more readable.  The core logic is no longer obscured by the exception handling, in fact the handlers make both the core and the exception logic easy to read and understand. We are also able to de-dupe some of the exception logic through reuse of the _factory_error_handler.  However, we can do better.

from contextlib import contextmanager
import sys

@contextmanager
def cleanup_error_handler(cleanup):
   try:
      yield
   except Exception:
      cleanup()
      raise

def make_error_handler(catch, throw):
   @contextmanager
   def handler():
      try:
         yield
      except catch, e:
         stacktrace = sys.exc_info()[2]
         raise throw(e.message), None, stacktrace
   return handler

factory_error_handler = make_error_handler(FactoryError, ValidationError)
customizer_error_handler = make_error_handler(CustomizerError, ValidationError)

class Car(object):
   def create(self, color, stereo):
      with factory_error_handler():
         vin = self._factory.make_car(color)

      cleanup = partial(self._factory.remove_car, vin)
      with cleanup_error_handler(cleanup), \
           customizer_error_handler():
         self._customizer.update_car(vin, stereo)

      return vin

   def update(self, vin, color, stereo):
      with factory_error_handler():
         self._factory.update_car_color(vin, color)
 
      with customizer_error_handler():
         self._customizer.update_car(vin, stereo)

      return

We retained the readability of using the handlers, but extracted them completely from the class.  The exception logic is now both encapsulated and decoupled from the specific class allowing reuse throughout the rest of the codebase.

§

Python Metaprogramming: Dynamically Adding Methods to Classes

Dynamic addition of methods and attributes to classes and instances.

Dynamic Class Method Addition

Occasionally you may need dynamically add a function as a method to a class.  This is easily accomplished by assigning the function as an attribute of the class.

def fn(self):
  return id(self), self, type(self)

# Traditional Class Definition
class A_Class(object):
  def method_a(self):
    return id(self), self, type(self)

instance = A_Class()

# Modify the class and add fn as a method
setattr(A_Class, 'method_b', fn)

# Call the traditionally defined method
instance.method_a()
# Call the dynamically added method
instance.method_b()

Dynamic Instance Method Addition

When you add the method to a class all instances can access it.  If you only want a particular instance to have a method do this:

from types import MethodType

instance2 = A_Class()
setattr(instance, fn.__name__, MethodType(fn, instance))

# Calls the fn method attached to the instance
instance.fn()

# Throws an exception
instance2.fn()

§

Configing Emacs: TRAMP with Putty

I've already written how to configure TRAMP in Emacs, but it only works on Mac and Linux. Unfortunately there are some who continue to use Windows as a desktop platform.   While they may be second class netizens, they do not need to settle for second class remote editing.

Emacs is available for Windows.  For secure remote editing you need ssh.  Enter Putty, an excellent implementation of the SSH protocol for Windows.  Putty comes with plink.exe which is the command line executable.   Once you've installed Emacs and Putty on Windows you'll want to update the Path environment variable.

Once the Putty path has been appended, update your emacs config.  Set your default tramp method to be plink and the tramp auto save directory to your username's temp directory.

(require 'tramp)
(set-default 'tramp-auto-save-directory "C:\Users\<username>\AppData\Local\Temp")
(set-default 'tramp-default-method "plink")

This all you need to start remote editing with emacs using ssh. You can setup password-less logins to make it even easier.  And if you have to go through an intermediary ssh server you can use transparent multihop with Putty.

§

Python Metaprogramming: Dynamic Class and Metaclass Creation

Python has great metaprogramming capabilities.


Python classes are typically created using the class definition.  Classes are used in much the same other classes in other OOP languages are used.  They are used to define an instantiable object with associated methods and attributes.  Classes are also one of the primary namespacing mechanisms in Python. The terms class and type are often used interchangeably in Python. Occasionaly you may need to dynamically create a class.

There are use cases for dynamically creating a class or type.  Often it's a static definition doesn't meet your needs such as a type factory that needs to work on incoming types.  You may need to create a type dynamically for testing purposes, or because it's required by a framework or library you need to use.

NewClass = type('NewClass', (object, ), {})

Metaclasses are a another kind of class.  They are created in the same way as other classes except they derive from `type`.  You can programmaticlly create a a metaclass as follows:

def new(cls, name, bases, dct):
    print "metaclass new", name
    return type.__new__(cls, name, bases, dct)

def init(cls, name, bases, dct):
    print "metclass init", name
    super(type(cls), cls).__init__(name, bases, dct)

DynamicMetaClass = type('DynamicMetaClass',(type, ),dict(__init__=init, __new__=new))
X = DynamicMetaClass('X',(), dict(foo=lambda self:'foo'))
# returns metaclass new X
#         metclass init X

§

Python Metaprogramming: Dynamic Module Creation

Python has great metaprogramming capabilities.

Python modules are implicitly created with a file.  It is one of the primary namespacing mechanisms in Python.  Occasionally you may wan to dynamically create one.

There are a few use cases for dynamic module creation.  The use case that comes up most frequently is modules created in the service of automated tests.  Sometimes a module is needed as input and sometimes it's needed as a mock.  Other use cases exist, but are more rare in my experience.  One I have done recently is automatic module creation to create the appropriate modules a framework or library expects. There have been other instances as well, although infrequent.  Regardless, it's a useful tool to have in your toolbox should the need ever arise.

from types import ModuleType
import sys

def make_module(new_module, doc="", scope=locals()):
    """
    make_module('a.b.c.d', doc="", scope=locals()]) -> 

    * creates module (and submodules as needed)
    * adds module (and submodules) to sys.modules
    * correctly nests submodules as needed
    """
    module_name = []
    for name in new_module.split('.'):
        m = ModuleType(name, doc)
        parent_module_name = '.'.join(module_name)
        if parent_module_name:
            setattr(sys.modules[parent_module_name], name, m)
        else:
            scope[name] = m
        module_name.append(name)
        sys.modules['.'.join(module_name)] = m
    return sys.modules['.'.join(module_name)]

§

Python Class Decorator: Logging

Python class decorators provide powerful metaprogramming capabilities.  Instead of a function, a class decorator takes a class and returns a class.  The decorator can manipulate the incoming class, or simply create a new class and return it.

The following piece of code is an example of leveraging a decorator to do Aspect Oriented Programming.  The classic example in AOP is the addition of logging to methods and functions.  Rather than writing a function decorator, and adding the decorator to each function, you can choose to create a class decorator and logging by class.

import logging
log = logging.getLogger(name)


from functools import wraps
from types import MethodType

def class_logging(klass):
    """
    Class decorator that adds logging to all "public" class methods.
    """
    method_p = lambda m: not m.startswith('_') and \
                         isinstance(getattr(klass, m), MethodType)
    public_methods = filter(method_p, dir(klass))

    for method_name in public_methods:
        class_method = getattr(klass, method_name)

        def helper(mname, method):
            @wraps(method)
            def wrapper(*a, **kw):
                msg = '{0}({1}, {2})'.format(mname, a, kw)
                log.debug(msg)
                try:
                    response = method(*a, **kw)
                except Exception, e:
                    error_message = 'no additional information'
                    if hasattr(e, 'message'):
                        error_message = e.message
                    msg = '{0} raised {1}, {2}'.format(mname, type(e), error_message)
                    log.debug(msg)
                    raise
                else:
                    msg = '{0} returned {1}'.format(mname, response)
                    log.debug(msg)

                return response
            return wrapper
        
        fn = MethodType(helper(method_name, class_method), None, klass)
        setattr(klass, method_name, fn)
    return klass

§