Major DB changes? Migrate, Duplicate or Deprecate

So you have a shiny new database technology, or it's time to make a major upgrade, or you just need to refactor. For whatever reason you want to move a large chunk of data into db_new_table. How do you cope with this? On a live system??

I see 3 options: Migrate, Duplicate or Deprecate.

Migrate

Migration is the act of coding up a script to dump any data from your legacy database table(s) into your new one(s) Migration, done properly, will give you the least grief in the long term as there will be no legacy tables to maintain in the future.

It may even present an interesting programming challenge.

This method is loved by developers due to the fun scripting problem and removal of legacy chod. It's also hated by the bean counters because of the  large risk of the migration script coder not knowing about some fringe case or other. What happens if it works on your Beta system but all goes bad when released to the wild? Migration is difficult to back out once you start using the new tables.

Duplicate

Duplication is the act of running the old and new databases side by side. Both are written to, and when you come to read, you check the old one first, then the new one. At some point, when you're happy that you are storing the same data in both places, you stop writing to the old one and you start to read from the new location first.

Duplication is a relatively safe way of introducing a major database change with just one slight concern: that you may never be able to retire the old storage method. This technique works well for "temporary" data (who really cares about a deleted email from 2 years ago?)

And if you are storing "temporary" data, this may lead on to deprecation.

Deprecate

To deprecate is to cleanse. Switching off the old system entirely and rolling with just the new one.

If you have a table that is cleaned out on a daily, weekly or monthly basis (such as one used in fast profit and loss calculations) deprecation is as easy as disabling the old interface and enabling the new one. A single config bit flip.

Most often you'll need a combination of these techniques though, such as before deprecation you may migrate some of yesterday's data into your new profit and loss table to check the numbers still add up correctly. Or you may want to migrate your data and then write to both systems - duplicate - until you have confidence. This will ultimately lead to deprecation.

Whichever method you chose will depend on the risk of it all going wrong, how much time you have to do it, and the nature of the data stored.

Pair Programming with Google Wave Part II

It seems that my "Pair Programming with Google Wave" prophecy has come true.

It was such an obvious idea that it was only a matter of time - even if it wasn't how I imagined the technology to be used.

KISS Vs DRY

In my last post I talked about the expert programmer knowing when not to implement a feature or widget, so as to keep the product and codebase as maintainable as possible.

This idealogy stems from two of the core principles of good software development practice: KISS - Keep it simple, stupid. And DRY - Don't Repeat Yourself.

But it turns out that these two fundamental practices are occasionally locked in a violent and bloody conflict.

DRY often requires you to enhance the complexity of the software, violating KISS.

Consider the case where you could either implement a new module - and with it a new interface - or attempt to re-use an existing module with a fixed interface.

The first option would be the easiest to code as you have free reign over the interface definition, sticking to KISS. But it would violate DRY.

The second option may require quite a bit of work in coding up a wrapper and violating KISS. But it would conform to DRY.

In this example, which method you choose depends on a number of factors and although the second option is normally the correct one, it won't always be.

How to be a Guru

To be a true master - a Guru - in the art of software development you do not need to know what feature must be added.

Recognise instead what should be left out.

Go on, ask.

I think we've all been there, spending a couple of days battling with an SDK to get it to do something slightly different, something slightly leftfield. Something slightly risqué.

We keep battling away, Googling and re-Googling, heading further into the SDK's proverbial dusty corners, not knowing if the classes and methods that we're searching for will even do the job.

And then finally we crack and ask someone for help.

What do we expect to get from asking?

Someone to tell us that we just need to call the Foo::Bar() method?

If we're lucky this is exactly what will happen.

If we're less lucky we'll be told to explore another dusty crevice requiring further Google action.

But most likely, we'll be given some advice that will help us to look at the problem in a different way:
Perhaps we don't need the gui to do that funky trick, and it would actually be far move user friendly if it behaved in another, more standard way.
Perhaps we shouldn't try and make that library function totally generic and rewriting it to solve a specific issue is perfectly acceptable.

Whatever happens, we shouldn't see asking for help as a weakness, but rather as an opportunity to better ourselves by learning from our peers.

Startup Incubator is Looking for you

Red Gate Software are prepared to help out start-ups by offering good people a desk, money, accommodation and even free food.

All they want in return after the initial 10-week development period is a chunk of equity.

But the point is they're prepared to take a gamble and invest in good developers, giving them everything they need to make great software - but with no risk.

GDP is not very good at Measuring Global Growth

It's time for a new measure of growth because, in a world with globalisation - where much of the money you spend goes to another country - GDP is incompetent.

GDP increasing year on year is supposed to be a good thing, but why do we expect this to happen at all?

Where does all the extra money come from?

It comes from the planets limited supply of raw materials such as its coal, gas, oil, gold, iron and timber.
They're either being mined faster or becoming more precious because of their increased rarity.

It comes from the planets renewable materials, for example making water drinkable or generating electricity from wind and solar power.
(And, by the way it makes sense - if for no other reason than long term financial gain - for governments to invest in renewable power.)

It comes from farmed goods such as wheat, rice, fish, livestock and timber.

And it comes from the fact that, due to population increase, we are slowly using up all of the inhabitable space on the planet and that's driving
up land value. If you own a chunk of planet, its value is likely to be proportional to the global population.

And that's it, everything else is just inflation and national debt.

If I do a service for you, no value is created, dollars are simply exchanged. No new money enters the international mega-pot.

Governments can't just "make" money without inflation or currency devaluation. And devaluation is just another form of inflation when you consider the global economy as a whole.

Please tell me if I'm totally missing the point of GDP here! Because it seems to me to be a pointless measure of a global border-less macro-economy!

10 Questions to ask before Changing your Software

It's been said that modern day software engineering is like gardening in that you're not normally creating something new but maintaining an existing system.
So when gardening - when making changes to your existing codebase - there are several questions to ask:

1. Why am I Making the Change?

The best code change is no code change at all - because every code change carries the potential work of a bug fix waiting to happen.
The smaller the code or configuration change, the lower the risk. If it ain't broke, don't fix it.

2. Who is the Customer?

Are you making the change because your boss thinks it would be a cool new feature? In which case he's your internal customer (this is often the easiest case as it matters less when the requirements aren't completely defined up front).
Or is it that there's a real, paying, external customer who wants something added to their bespoke software package?

3. What are the Use Cases?

Once you know who it is you're developing for, you'll need to know exactly how they plan to interact with the system...

4. What are the Detailed Customer Requirements?

...and what those new UI components should do under the hood.

5. How will the Architecture be Affected?

Will the change affect your software's architecture? Are you just adding a new widget to your GUI, or are you going to rewrite half of the existing code?

6. Which Existing Modules need to Change?

You do have a modular codebase right? Yeah? So you'll need to think about which ones need to change, and how.

7. What new Modules are Needed?

Will that new widget require its own module, or does the functionality sit naturally in an existing module?
If new modules are needed, what's their public interface and how do they interact with everything else?

8. How will Errors be Handled?

How does the software cope when the inevitable happens and the user does something stupid?
Or the robot that you're driving decides to crash into itself?
What happens when the web service that you query falls over because a pop superstar passes away and there's too much web traffic?

9. How do we test it all?

As this is a change to existing system, you need two types of test:
1. Regression tests to confirm that the existing system still works.
2. New tests to test the new stuff.
If the software already has an automated test suite you've covered 9.1 already and covering 9.2 should be a doddle.

This is also the place to look our for deficiencies and commonly experienced problems, for instance, if you have a highly multithreaded codebase, how do you make sure that the delta hasn't introduced a deadlock?

10. What are the alternatives?

Double check that the change you've decided to make really is the best way before touching any code.

What do you think? Is there anything that you disagree with? Or that should be added to this list of gardeners' questions?

Clustering keywords for meaning

After a couple of occasions when Google has totally let me down, I've been thinking about the problem of improving search engines. And particularly about the problem of meaning in a search time.

For example, when you Google the term Ultralight you are presented with, amongst others, ultralight outdoor gear and ultralight aircraft as Google has no idea which you are interested in.

Flickr gets around this problem by clustering results together - compare their ultralight cluster.

Perhaps this is more of a problem with using short, relatively meaning less search terms, something we've been conditioned to do because search engines just don't understand natural language. Yet.

Pair programming with Google Wave??

Google Wave was announced the other week and seems to be making ripples (excuse the pun) amongst technology bloggers. I recommend readers who know nothing about it to watch the video as I'm only going to talk about one aspect:

Pair programming with a Wave Extension

Wave is all about collaboration. Collaboration and extension.

And it won't be long before extensions exist to compile your favourite language. Then, its real power will be presented to developers.

Imagine coding in C++, pair programming with a legend like Herb Sutter or Scott Meyers?

(I'm sure that all they'd require is a convenient way of charging ;)

How to hit the Front Page of DZone. But why it Doesn't Matter.

I've been lucky enough to hit the front page of DZone on a number of different occasions and I believe that there are several tricks that can be employed to reach the holy grail, the front page.

Captivating or Controversial Title

DZone readers spin through hundreds of new links everyday trying to find something new or interesting to read. As a blogger or link submitter, you have a dozen words or so to captivate their attention and get their valuable click.

The title you choose should give them measurable facts, because what coder can resist indisputable fact? E.g. 6 Free Tools that every Windows Programmer should Install

Or, the title should provoke controversy, such as Should Coders be Allowed to Wear Headphones at Work? Make it a subject that most programmers have an opinion on and they'll click on the link just so that they can add their comment.

Quality Content

You can't find a post on blogging that doesn't mention quality content. So, I shall to!

Seriously though, people will only give it the DZone thumbs up if the article is well written and delivers what the title promises. Obvious really.

When to Submit

I've not yet hit the front page from a Monday submission. I think people must be just too busy on Mondays to promote links. Tuesday to Friday is best.

But why it Doesn't Matter

Chris Guillebeau over at the Art of Nonconformity argues in his excellent manifesto, 279 Days to Overnight Success, that you don't need to hit Digg (or indeed DZone's) front page to make a living from blogging.

He says that if you're trying to make money from Adsense - and he reckons that this is quite a difficult task - then you need that constant stream of visitors that Digg (or indeed DZone) can provide. But if you're trying to build a fan base interested in your compelling story then those sites won't help.

Multi-threaded JavaScript Makes Stream Processing a Reality

The future is finally here:

Multi-threading in a JavaScript app.

Wow!

Do you realise what this means? It means websites could use a form of stream computing to get your computer to do stuff without you even knowing.

It means that a website sending out processing requests to every browser that hits it - without said browser becoming unresponsive - is now a reality.

And as JavaScript engines become more efficient, this will be even more of an option.

The web-based super computer is amongst us, if a site where users tend to stick around for a while, such as Facebook or Myspace, wants it to be.

Let me just explain that last point - Facebook et al (or, indeed any web site) could send packets of data to your machine for it to process in the JavaScript enabled web page without you even knowing it. The answer to the processor-intensive question would then be sent back, and the web site gets its result.

It could even sell this processing power!

Is it legal, I think so.

Is it ethical, well, that's for another debate.

Motivating Geeks... And non-Geeks Part II

In How to motivate Geeks... And non-Geeks I talked about not de-motivating by getting the basics wrong.

So, time has passed and you've got the basics nailed, your hygiene factors are suitably hygienic, where do you go now?

Well, now comes the motivators: Respect, Truth, Openness, Trust, Responsibility, Recognition, Achievement and Growth.

I learnt some of these the hard way, like the time when I used to make computer games and I was technical lead on a new PS3 development, I told a developer who'd been programming since the ZX Spectrum days how he should go about finding and fixing a bug.

He sturnly replied, "Yes, I know!"

My comment - at the very least - broke the Resposibility and Recognition rules. And probably the Respect rule too.

There's a common misconception that salary is a motivator. This is wrong. A decent salary is required to stop people worrying, and you won't be motivated unless you have enough money to pay the bills, but an increase in salary is not equal to an increase in motivation.

Be honest with your staff, give them responsibility and then trust in them to do a good job. Share with them how the company is doing, ask them how they're doing. Tell them the truth.

Then, and only then, will they really excel.

Motivating Geeks... And non-Geeks

Maslow's hierarchy of needs and Herzberg's two factor theory both say the same thing:

You need to get the basics right before you can expect your staff to be motivated.

If workers are scared of losing their job or frightened of not being able to buy food for their children, they aren't going to be motivated.

When people get annoyed at their unresponsive PC and tediously slow compile times they aren't going to be motivated.

It won't help if they're in pain because their desk or seating area is wrong. Or if the temperature is sweaty-forehead-hot or freezing-nose-cold.

When there is excessive noise they won't concentrate, and won't care.

If bullying occurs or workers feel undervalued they'll just not bother.

Pay attention to the little things. Ask your staff what they need and what they want, and you'll be well on your way to a motivated workforce.

Motivating Geeks... And non-Geeks Part II

Is Programming a Creative Pasttime? Discuss.

Martin Amis missed a trick when he called his book The War against Cliché. The War on Cliché would have been far more apt.
"What has a book on literary criticism got to do with software development?" I hear you question.

43/365 - That's So Cliche! FGR
Originally uploaded by AMooseinBrighton

Well, programming is often said to be a creative pasttime. And when I was developing video games I would have totally agreed with that statement.

But now, with some years of professional development in non-entertainment-software circles, I'm not so sure that it is.

Beautiful software is built from well structured reusable components that adhere to design patterns.

The design pattern is the programmer's cliché, in that the world has seen it all before.

So when you're thinking about being creative and coming up with some fancy new algorithm, just remember that good ol' design pattern clichés are good.

What do you think? Is programming creative?

Business Balls for Leadership and Management

I just quickly need to mention BusinessBalls.com as it's an excellent repository of business and management articles, reference sheets and explanations.

If you're wondering why that staff member might not be motivated, what it takes to be a fantastic leader, how you should conduct that interview, or even if you're looking for some team building day icebreakers, this site has it. And some.

Risk in Software Development.

I've been listening to some talks on financial markets and it's got me thinking about risk in software development.



Risk is everywhere in the software creation process.

For example, should you invest resource in writing formal user requirements specifications, functional specifications and acceptance test plans. Or do you risk the project overrunning when it doesn't quite do what the customer expects and you have to make a number of late changes?

Should you take time at the start of a small project to write unit tests before any other code. Or do you risk a drawn out testing phase later on?

In financial markets risk is measured using a number of statistical methods but how would you go about measuring the risk in the examples mentioned above?

If you know the customer that you are delivering the product to, you may know how forgiving (or not!) that they are. If you have a good relationship with the client you may be able to get away with a number of severity 3 or 4 bugs.

When dealing with an unknown customer assume the risk is high. And the larger the development, the larger the risk. With a high risk project you should aim to employ as many good software development practices as possible.

(Thanks to Esko for the excellent photo)

6 Free Tools that every Windows Programmer should Install

1. Unlocker

I've talked about unlocker before, it's basically a fabulous little utility that tells you which app is locking your file (and it normally says it's the TortoiseSVN cache which brings me neatly onto my next utility...)

2. TortoiseSVN

The code you produce at work is version controlled, so why isn't the code your make at home? TortoiseSVN is the best way of managing a home version control system as it's a really easy Windows Explorer plugin.

3. Dependency Walker

So you're developing a large application - and by large I'm talking tens or even hundreds of dlls.

You change the public interface on one of the non-dynmically loaded ones and, because you're in a hurry to get a build finished, only rebuild that dll.

Suddenly you can't start you app up any longer.

Dependency walker can tell you which other dlls need to be rebuilt so that they can interface with the changed dll.

4. Notepad++

You're not still using notepad to open .txt files are you?

Notepad++ is far better as it supports custom plugins and has macro recording, yet it still starts up in an acceptable time.

Of all of the freeware editors out there, this is my favourite as it behaves the most like Visual Studio when the Ctrl key is held down and your flicking from word to word.

Other alternatives to the Big Bad Pad are: PSPad or Programmer's file editor, but I've tried them both all I'm sticking with Notepad++.

5. Process Explorer

Want to know what's eating all your processor time? Download Microsoft's pimped-up task manager, Process Explorer.

6. AutoHotkey

There is generally thought to be a 10 fold difference in productivity between poor programmers and expert ones. I believe that this is in no small part down to the experts' use of keyboard shortcuts. Auto Hotkey allows you to set up hotkey scripts to control virtually anything.

Gold Code

I'd previously written that working code is worth it's weight it gold. That is true, but it's only partially true.

Excellent code doesn't just work - proven through unit tests and structured manual testing - it is also written in an obvious way. Any old code monkey can take ownership of the code and quickly understand what it's meant to do and how it's meant to do it.

If they can't just pick it up and run with it then it's not gold standard software, It's probably tin, maybe bronze. Silver at a push. But certainly not gold.

Gold standard code uses no magic numbers, instead named constants are used. If magic number just have to be used, comments are mandatory.

Gold code is consistantly formatted throughout, commented where appropriate and has variable names that follow the 5 golden rules.

Golden code works and can easily be changed by anyone.

Gedanken Experiment - Removing Debt

This is the first post in a series of off-topic thought experiments inspired by the credit crunch.

Imagine a world with no debt.

I'm not talking about elimination of third world debt and Make Poverty History that Bono and his pals quite rightly support. I'm talking about the removal of debt as a concept. That is, no person or company ever owing any other any money.

But would it work?

How could it work?

The banks

Would banks be able pay interest on your deposits if they couldn't invest them?
Would banks even be allowed to take deposits off you, as one man's deposit is another man's debt - the banks will be in debt to you after taking your money!

So if banks can't take or lend any money, why would they even exist?

Bricks and Mortar

How would people buy houses if mortgages don't exist? Would people have to save until they can afford that first home? And how long would that take - 10 years? 20? It will certainly be less than the normal 25 years because no interest will be paid.

So where do they live for those 10 years? In rented accommodation, government provided accommodation, or in the family home. And, if they do rent, how does their landlord go about buying the property to then let it out again?

And where are all these savings kept until the house is bought? Without banks the only option would be to buy real objects: gold, oil, commodities that appreciate in price and can be sold on.

Shares

Company shares aren't really real are they? They're just a way of a company borrowing money off a bunch of investors. And by borrowing off a number of different investors they mitigate against the risk of one investor suddenly wanting all their money back. But there's that word again, "borrowing", so shares are out and so is the FTSE.

So what do you think? Here are just a couple of the problems with this radical idea, but could it really work?