The Generation Game

Let me tell you a story (or "lie", if you prefer). In 3 acts.

Act 1

I live on a roundabout. Like that bloke who lived on a roundabout, or the fella from Parliament Square. And I'm famous. People come up to the roundabout from one of the 5 roads, and before they make their way round it they ask me my name. But I'm mute, so I have to write it down on a sign for them and hold it up. They read it, and go on their way, so I throw the sign away. But then someone else turns up on another road, and I have to get round to them and do the same thing.

It's really very tiring, writing my name on signs over and over again all day. The worst thing about it is the traffic around this roundabout flows very slowly, and people get annoyed at how long it takes to drive round me. All they want is to know my name, but they have to wait 'til I've told loads of other people before they even get to ask. I've even heard that a lot of people just aren't even bothering to come my way any more.

Act 2

I still live on a roundabout. People still drive up to ask my name. But y'know what? I've got a couple of mates now. No-one's interested in them, which is good, 'cos it means I can just get them to write the signs for me. They write about 25 and then have a rest. Now, when people drive up to ask me my name I just take one of the signs out and show it to them, then throw it away.

It's much better. I get to tell so many people my name now, way more than before. It's still not perfect though -- when one or both of me mates is ill I have to scramble around as before. And once 25 drivers have come and gone, the 26th one has to wait ages while a sign gets written for them. But still. I'm loads more popular!

Too popular.

The council gave me a second roundabout. And a third. I'm a tourist attraction. So now there are people driving up to three roundabouts, and I've had to draft in a lot of introvert mates. And every time 25 people have gone past one of the roundabouts my mates have to draw another 25 signs. They're getting burnt out. I'm going to have to start paying them (more).

Act 3

I haven't got many mates any more. Just one, in fact. Got 10 roundabouts though, and well prepared for any more. I had a bit of a brainwave, see.

Me and my mate made 50 signs. And we attached them to posts, facing the roads at each roundabout. So now, when people come up, they can ask my name but it's already there. Right in front of them. Sometimes there's an accident and I have to replace one, but in general it works a treat.

People don't even need to ask me the question any more, and if I change my name, well, I only need to tell my mate. He can make some new signs. And he doesn't take much looking after.

The analogy explained (in case it wasn't obvious)

I used to be a fully dynamic website. Database queries on every request. I had a bit of traffic. Not much though.

Caches were my friends. They're databases, and I still had to do queries, but smaller and faster ones. But I had to maintain the databases which just duplicated the data I already had. And they kept emptying and coming back to ask the main database the same query as they asked half hour ago. They needed a bit of looking after.

I'm now a damn fast website. I get through a lot of traffic. My one companion is a generator/publisher. It prepares stuff in advance which doesn't change, and just places it there, right in front of all the traffic. And if I get twice the traffic? He can probably cope. Five times the traffic? Maybe then I'll need another one.

The moral

Seriously, just do shit offline. The Q in SQL is for "query". Like question. Why would you want to ask the same question multiple times if you know the answer doesn't change, or changes rarely -- and, crucially, if you know when it changes? Why even use a cache if it's only going to expire, and you need to maintain both the databases and the code which populates them? Moreover, the code and database are there to do nothing but store exactly the same information as you already have. What happened to "don't repeat yourself"? Is that only for code?

The master/controlling source of a website's data (eg a CMS for a media site), knows when data changes, so just push it out. Not to some intermediary - again, why bother? Just push it out all the way to the front. Generate it. Build it. Publish it.

I'm not arguing for no dynamism in websites. I'm just saying look, make the dynamism appropriate. Base it on the user, their behaviour, their request, or whatever -- but not on the things you already control. Changed your name? Don't wait for someone to ask you, get it out there before they even ask.

Want to know a query whose answer rarely changes? "What's the main body of the content at URL /2009/06/fail/?" One which changes regularly? "What's the most recent bit of content on the site?" Just think. about which parts of your site, and which parts of each page, actually change based on some kind of external input, be it the time and date, some facet of the current user, or some other unique facet of the individual request. You can do more of those, and provide more functionality, if you generate/build/publish the stuff that you control.

Do shit offline. Capiche?