I blog when I go abroad, and occasionally when I do stuff in the UK too. There's a nicer interface over here.

Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Friday, June 12, 2009

Doing shit offline

I was just thinking it was cool that I'm already the 3rd hit, at the time of writing, for "do shit offline" when searching on Google. But then I snapped out of it. Yes, my generation game post yesterday was about websites, but not all software engineering is about the web. The stuff I bang on about needn't spit out HTML or PHP or anything of the sort. I have an mp3-fixer-upper (mentioned below) which spits out a shell script. That's because these are just software engineering techniques (patterns, if you will) for generating lots of similarly structured output from minimal input: lots of unique data, but comparatively few templates.

I work as a software engineer whose career has mostly led him to working with websites; I do not work as a web developer. And despite currently being on the dole, I figured I might as well pimp some software that helps with doing shit offline, huh. These are 2 projects I've been intimately involved with as both developer and user.

r3

Yahoo!'s r3 is ostensibly an internationalisation/localisation tool, but to my mind its real power comes from the fairly complex, at first glance, inheritance path concept. This brings object-oriented techniques to file generation, and there's the key word: r3 is first and foremost a file generation tool. In go templates, out come files.

I was one of the core engineers on the team which developed and maintained r3's predecessors, which were internal CMS tools at Yahoo!. I had fairly heavy involvement in some of the architectural and design discussions and decisions made during r3's genesis, and was the sole internal customer representative at the team's first "next steps" planning etc session 18 months later. I'm quite a fan, even if the public docs aren't quite up to scratch.

pork.py

This is something I knocked up in Python in the last couple of months. I even blogged about it before. So much for "don't repeat yourself", huh? It's a simple script which marries YAML to a template, and creates some output. The output can be STDOUT or a file, and 4 template engines (of sorts) are supported. It's meant to be standalone, but deliberately usable as the central pivot of a get-some-data, produce-some-output, put-it-somewhere pipeline. In fact, when used this way you don't even need YAML - just a couple of python dictionaries. See my mp3-fixer-upper for an example of how.

Monday, April 27, 2009

simple django+yaml file generator

EDITED to change its name.
EDITED to change its name again! What was I thinking? It should always have been called pork.py.

I was trying to write a website for PORK, but I fail massively at HTML, CSS, design, all of that nonsense. And I'm bored, a bit. Watching The Business on Channel 4 HD despite having seen it about 5 times before (I love it). So, rather than do anything useful, I've done a stereotypically daft, over-engineered, off-on-a-tangent avoiding-the-real-problems thing and knocked up a tool -- which countless other people (me included) have already implemented in the past -- to generate what I need... though of course I still need to write the source templates, so it hasn't actually done me any good at all.

Yes, it's another simple static file generator.

This one uses django+yaml to do its stuff. After some cursory investigation it seems there are already a couple of "lightweight static file generators" that do similar things, but frankly none of them seem as lightweight as mine. It's one file, 100-odd lines but ~75% docs, comments and whitespace. It does the job for me, maybe it does the job for someone else. I called it pork.py. Mmm.

I don't go for the github way of doing things, and nor do I tend to go back to something once it reaches a usefulness limit to me personally -- so if by some miracle you do like this and want it to do more, just take it, extend it, publish it, put your name on it, I really don't care.

Monday, September 29, 2008

A new type of django relationship: Generic Intermediaries

Bloody hell, it's a second technical post in the space of a week. I was really bored last night (when I wrote most of it, as the publish date suggests); had seen both of the Family Guy episodes on FX several times before, and similarly I've seen Die Hard enough times for it to not really require another viewing. Now, if it had been in HD... anyway, the upshot was that out came OmniGraffle, before I knew it I'd created a diagram and then, well, a picture needs a thousand words of explanation. So, after the lozenge, here they are.

NB this stuff is also included in the django-slots wiki; I thought it would be sensible to post it somewhere that might have an audience, as well as this blog.


Generic intermediaries: relationships with characteristics

Introduction

This document describes the GenericIntermediary django model and IntermediaryKey, a key-like object. Together these two classes provide a mechanism for giving characteristics to relationships between models.

Existing relationships in django

fixed relationships

Django already provides relationships between models. These allow you to link single or multiple instances of models to one another. Their existence is reflected in the database schema behind those models, be it generated when using syncdb or defined explicitly with dmigrations. I'm calling these relationships fixed because the model on either side of the relationship is explicitly specified in the code.

generic relationships

The content types application (django.contrib.contenttypes) ships with django and is in INSTALLED_APPS by default. As well as providing a unique identifier to all model instances in your project through an app/model/id triplet, you also get the ability to specify a generic foreign key and/or generic relation. This lets you genericise one side of a foreign key relationship: that is, specify that your model can be attached to any other model. This relationship is specified by using two fields: a ForeignKey to ContentType, and a regular field used to store the ID of an instance of that type. As with the fixed relationships, therefore, this requires columns in your schema, to reflect the fact that the model is related to something else.

Generic intermediaries

Generic intermediaries are a way of specifying that a relationship exists between two model types separately from the instances of those models. The relationship is then given characteristics through a new model, in which the fields containing the instance IDs are also stored. This model can then be used to create a mixin, a Manager-style object or Key-style object, to give new attributes to existing models without requiring schema changes. This is how django-slots is implemented.

Diagram

This is a diagram of how django-slots is implemented, including the slots_demo app which provides the Page and Style models.



Explanation

Page and Style are django models, implemented as normal, with whatever attributes they require.

Between them is GerenicIntermediary In concrete terms this is a model with just two attributes, each of them a ForeignKey on ContentType and a unique_together constraint ensuring only one relationship between two types -- in one direction -- can exist. The direction is important: as with the diagram, the two keys represent the models on the _left_ and _right_ hand side. The left-hand model is that which the right-hand types are _against_; in django-slots therefore Page is on the left.

Slot is a django model which has a ForeignKey on GenericIntermediary This is, in effect, a declaration that Slot implements characteristics of a relationship. Missing from the diagram (bolded to remind the author to remedy this!) are the attributes which contain the IDs of the instances which are related, that is, the ID of the Page objects and that of the Style objects.

Left at this, scheduling would be possible. You would create a slot like this:


# assume we have Page and Style objects called page
# and style respectively; we also have two datetime
# objects, start_time and end_time
cp = ContentType.objects.get_for_model(Page)
cs = ContentType.objects.get_for_model(Style)
gi = GenericIntermediary.objects.get(left=cp, right=cs)
slot = Slot(relationship=gi, against_object_id = page.id,
slotted_object_id = style.id,
start_time = start_time, end_time = end_time)


and retrieve it so:


# same assumptions as above; also same cp, cs,
# and gi assignments
now = datetime.datetime.now()
# look for a slot that now falls inside,
# against our page
try:
current_style_slot = Slot.objects.get(
relationship=gi, start_time__gte=now,
end_time__lte=now,
against_object_id = page.id)
except Slot.DoesNotExist:
current_style_slot = None
else:
current_style = cs.get_object_for_this_type(
id=current_style_slot.slotted_object_id)


This is horribly verbose and inconvenient. It's also not required.

Intermediary keys

Also missing from the diagram above is IntermediaryKey. As the name suggests this is a key-like object which relates to the GenericIntermediary. Informed heavily by the GenericForeignKey API, IntermediaryKey works by specifying which two fields together point to the instances on either side of the relationship. The first argument denotes both the relationship field (the foreign key on GenericIntermediary) and the side of the relationship, using normal django key__attr syntax; attr will always be one of left or right.

By having an IntermediaryKey the model gets an attribute which, like the fixed relationships, returns the actual instance of the related model.

This is how Slot uses IntermediaryKey


against = IntermediaryKey('relationship__left',
'against_object_id')
slotted = IntermediaryKey('relationship__right',
'slotted_object_id')


all this really gives us is the ability to use .against and .slotted as shortcuts to the instances of Page and Style in a relationship. The only improvement we can make to the previous examples is to shorten the current_style assignment:


current_style = current_style_slot.slotted


Still horrible, though.

Usage by django-slots

All the verbosity can be reduced (to taste) by the implementation of a class to define characteristics of the relationships, and the use of techniques to attach these classes to existing models.

django-slots' Slot model/class is the first such relationship (because GenericIntermediary and IntermediaryKey were invented for this project!); ScheduleMixin is the technique which attaches them to existing models.

The introductory blog post explains at a high-level what this means, in that it shows the API of django-slots. To fully understand the way to get from the above code to provision of attributes and methods, read up on mixin classes and see ScheduleMixin in models.py

Conclusion

GenericIntermediary and IntermediaryKey are not replacements for fixed relationships, nor generic relationships. Instead they are a way of representing the fact that a relationship exists between two arbitrary classes separately from the instances of those classes in the relationship. This is useful where:

  • the relationship between two models has characteristics itself;
  • one model's relationship with another is not, or need not be, an attribute of either;
  • a model wants to declare which other models are related to it, rather than the other way round; or
  • there is a need for another model to key on your own, when you cannot change its schema (eg in 3rd party apps you don't want to fork)

The mixin technique currently employed by django-slots demonstrates the first three of these use cases:

  • the relationship exists between two times
  • Style and Page are separate models with no explicit fixed relationships
  • Page declares that it would like Style to be attached to it; Style does not declare itself as tied to Page -- or anything at all

other random thoughts

I don't believe time is the only characteristic that could use this technique, which is why I've written such verbose documentation. I'm struggling to come up with proper use cases for, say, geographic foreign keys (where instead of start_time and end_time you might declare a bounding box, or latlong + radius?), but I have a gut feeling it could be useful.

Tuesday, September 23, 2008

Introducing django-slots

Sigh. Was it inevitable? I don't think it was, but it's happened anyway: I'm putting a purely technical post on my blog. Sorry and all that. Those of you who couldn't give a toss about python, django, coding, my job, and so forth can turn away now. Normal lack of service will resume shortly.

django-slots

This post introduces django-slots, a system for scheduling relationships between django models. It's an open-source (head-above-parapet) project which allows django developers to include time-based foreign keys in their applications/projects.

At the time of writing django-slots should be considered pretty nascent. Some reasons for this are detailed near the end. Nonetheless I believe even its current state provides enough useful functionality to justify its release.

Background and rationale

The first iteration of django-slots was a weekend pet project of mine, inspired by two things. Firstly, the team to which I belong at work were busy implementing several different solutions to what I considered a single problem: making a relationship between two objects occur for a period of time. Secondly, I believe that as a software engineer my job is to make my job easier; and as a software engineer on a CMS this mostly means that my job is to make everyone else's job easier too. This comes down to two things:
  1. Engineers should not be required to make changes happen at a particular time (and this means doing deployments etc).
  2. Users should not be using my software at times when I could really do without them calling me up saying it's broken (ie weekends, midnight, etc)
Both problems are solved by writing software which allows the future state of the data in my CMS to be scheduled, and previewed, in advance.

What django-slots is not

  • django-slots is not a system for making things appear and disappear, or exist and not exist
  • django-slots is not a tool to explicitly make something happen. It is not a replacement or wrapper for cron; nothing is ever triggered.
  • django-slots is not a replacement for foreign keys, or other normal relationships between entities
  • django-slots is not perfect or finished. By a long way.

What django-slots is

django-slots aims to provide developers with a way to satisfy the generic requirement of scheduling changes to relationships, designed with websites in mind. The most common concrete and specific example is probably to schedule a particular ad/sponsor/promotion to appear on a site between two times.

django-slots allows developers the freedom to define what "something" is through an intermediary mechanism. Unlike a normal ForeignKey, a relationship between two models exists separately from the instances of those models; the instance-instance relationship is bound to a period of time, known as a slot.

With this approach django-slots also provides a platform on which developers can build other tools to report, audit, preview, and more. A timeline of relationships means you can see the state of your data in the past, present, and future.

Furthermore, django-slots decouples models from one another, allowing them to exist and develop independently. No changes are required to the models which are scheduled, and no schema changes are involved in declaring the schedules attribute. django-slots is designed to be simple.

Finally, the mechanisms in use to implement the relationships inside django-slots are available for use by other applications. Specifically this means the definition of generic relationships between arbitrary model types on both sides (as opposed to the one-sided relationship already possible with GenericForeignKey). Where django-slots is concerned only with time, I envisage other applications in areas where similar concepts (universally identifiable points, etc) apply, eg geography.

Code

django-slots is hosted on Google Code, and there is a minimal installation guide on the wiki system which it provides.

http://code.google.com/p/django-slots/wiki/QuickstartGuide

Description

django-slots is used by telling your models to use a provided mixin class, and declaring a schedules attribute. This attribute should be a tuple of other class objects, which must be other django models.

By setting up your model like this you are declaring that a relationship can exist between it and those in the tuple. Your model is extended with properties and methods for querying and managing instances of these relationships. You can then schedule a relationship to exist, retrieve the current relationship or that for a given time, and retrieve a timeline of all relationships between your instance and instances of the other models.

API/usage


# models.py
from django.db import models
from slots import ScheduleMixin

class Style(models.Model):
# define your style model here
...

class Page(ScheduleMixin, models.Model):
# Style is the foreign key which varies according to time.
# NB. you don't need a default it it makes no sense to have one
...
default_style = models.ForeignKey(Style)
schedules = (Style,)

# views.py
def detail(request,...):
page = Page.objects.all()[0]
# the Style scheduled for right now, if there is one
style = page.current_for_model('Style')
if style is None:
style = page.default_style
# do stuff with style
...

# properties
# dictionary of schedules keyed by model,
# each entry is an array of slots ordered by time
page.schedule
# dictionary of all objects (or None) currently
# scheduled, keyed by model name.
page.current
# returns next scheduled objects (ie, where start time is
# later than right now) in same format as current
page.next
# returns last scheduled objects (ie, where end time is
# earlier than right now) in same format as current
# per-type query methods
page.last
# just the array of slots for Style
page.schedule_for_model('Style')
# the Style object currently scheduled, or None
page.current_for_model('Style')
# the Style object scheduled next, or None
page.next_for_model('Style')
# the Style object which most recently finished, or None
page.last_for_model('Style')
# finding what's scheduled at a particular time.
# NB this only works on a per-relationship basis;
# you cannot pass a datetime object to page.current()
page.current_for_model('Style',jan_1st)
# scheduling an object
page.add_to_schedule(style_object, start_datetime, end_datetime,
notes)
# a signal catches this and deletes all relevant slots
style_object.delete()

What's missing

As mentioned above django-slots is by no means complete. To my mind there are a few fairly crucial missing pieces right now:
  1. Removal (descheduling) of individual slots
  2. An admin interface.
  3. A test suite.
And there are bound to be far, far more. Hopefully such holes will be filled; better yet, hopefully others will (help) fill them.

Colophon

django-slots should work in any out of the box django installation, though it was written alongside django 1.0. The only configuration requirement is that django.contrib.contenttypes is in INSTALLED_APPS (this is the default).

Friday, August 29, 2008

Capital Radio and me

When I were a lad I used to listen to Capital Radio a lot. I have fairly vivid memories of Saturday mornings filled with it, on the occasions that my brother and I would stay at home rather than go down to stay with our maternal grandparents, as happened every weekend until 1988. At Christmas there was always the top 500 songs, played virtually back to back (maybe just between 9am and 5pm? was radio even 24hr back then?) over the course of several days, almost always culminating in Me and Mrs Jones, Layla, and Hey Jude. I remember listening to Pat and Mick's individual shows, not just their single(s), and to the commercial chart show which differed in some way to the one on Radio 1, though I can't remember how. I even remember some of the ads. Well, two of them: Harry Enfield in Stavros mode advertising a new newspaper, the Independent, with the strapline of "It flippin' is or are you what"; and new train route (now defunct!) Thameslink, singing "Thames-link! Thames-link! The train that takes you // straight through London // without changing stations // yeah!".

Well, now I work at Capital Radio. Actually I work for Global Radio (I think), who just bought the company I joined in March (GCap Media), and who own Capital Radio amongst a whole host of other stations across the country. Odd that we're named Global really, but meh. I'm not really a fan of Capital these days, but it is ace to work at a place I've known all my life. Still getting the hang of working for a company that isn't a pure internet company (this being my first such job!), but there are actually quite a few benefits to that. And best of all I get to keep working in central London, and bumping into (literally) DJs. In the last few months I have managed to stumble past David 'Kid' Jensen, Pat Sharp (uttering the word 'cunt', though not at me), Paul Gambaccini, and Henry Kelly in the corridors or at security. What a list! I'm sure I've also been in close proximity to various people who weren't broadcasting in some way in the 80s too, but I couldn't care less about them (actually I know full well that I've seen Alex Zane an awful lot).

Aaanyway, the real point of this post is just to puff my chest out a little bit. Capital launched their new website yesterday. It's not just a redesign -- it's a complete rewrite from the ground up by the team I belong to, a wholesale move from the outsourced version which existed before to an entirely in-house solution. I'm pretty proud of it for a number of reasons. Every part played is a big part given the small size of the development team here, but I'm (hopefully not unreasonably) particularly proud of my own contribution because of the technology we're using. As if this post wasn't boring enough already...

Capital's site, and the CMS which powers it, is built using Django, a framework written in Python. Nothing majorly special about that, but before March this year I had never coded with either, having never even heard of the former. Leaving Yahoo! after 8.5 years was a gamble for me, and likewise this place took a punt on me, believing my "I'll have no trouble picking it all up" spiel having been presented with a CV that said I could only code in Perl. I'm double pleased that I've repaid their faith, and proved (to myself, even) that I wasn't just boasting.

Though maybe this django and python lark is just really, really easy ;-)