Survive a Ground-Up Rewrite Without Losing Your Sanity, thanks to OnStartups.com

(Nice, I especially like "Worship at the Altar of Incrementalism"  :-)  I find the ideas relevant to smaller projects as well.)

How To Survive a Ground-Up Rewrite Without Losing Your Sanity

52
inShare

aka: Screw you Joel Spolsky, We're Rewriting It From Scratch!

This is a guest post by Dan Milstein (@danmil), co-founder of Hut 8 Labs.

Disclosure: Joel Spolsky is a friend and I'm an investor in his company, Stack Exchange(which powers the awesome Stack Overflow) -Dharmesh

So, you know Joel Spolsky's essay Things You Should Never Do, Part I? In which he urgently recommends that, no matter what, please god listen to me, don't rewrite your product from scratch? And lists a bunch of dramatic failures when companies have tried to do so?

First off, he's totally right. Developers tend to spectacularly underestimate the effort involved in such a rewrite (more on that below), and spectacularly overestimate the value generated (more on that below, as well).

But sometimes, on certain rare occasions, you're going to be justified in rewriting a major part of your product (you'll notice I've shifted to saying you're merely rewriting a part, instead of the whole product. Please do that. If you really are committed to just rewriting the entire thing from scratch, I don't know what to tell you).

If you're considering launching a major rewrite, or find yourself as the tech lead on such a project in flight, or are merely toiling in the trenches of such a project, hoping against hope that it will someday end... this post is for you.

Hello, My Name is Dan, and I've Done Some Rewrites

A few years back, I joined a rapidly growing startup named HubSpot, where I ended up working for a good solid while (which was a marvelous experience, btw -- you should all have Yoav Shapira as a boss at some point). In my first year there, I was one of the tech leads on a small team that rewrote the Marketing Analytics system (one of the key features of the HubSpot product), totally from scratch. We rewrote the back end (moving from storing raw hit data in SQLServer to processing hits with Hadoop and storing aggregate reports in MySQL); we rewrote the front end (moving from C#/ASP.Net to Java/Tomcat); we got into the guts of a dozen applications which had come to rely on that store of every-hit-ever, and found a way to make them work with the data that was now available. (Note: HubSpot is now primarily powered by MySQL/Hadoop/HBase. Check out the HubSpot dev blog).

It took a loooong time. Much, much longer than we expected.

But it generated a ton of value for HubSpot. Very Important People were, ultimately, very happy about that project. After it wrapped up, 'Analytics 2.0', as it was known, somehow went from 'that project that was dragging on forever', to 'that major rewrite that worked out really well'.

Then, after the Analytics Rewrite wrapped up, in my role as 5 Whys Facilitator, I led the post-mortem on another ambitious rewrite which hadn't fared quite so well. I'll call it The Unhappy Rewrite.

From all that, some fairly clear lessons emerged.

First, I'm going to talk about why these projects are so tricky. Then I'll pass on some of those hard-won lessons on how to survive.

Prepare Yourself For This Project To Never Fucking End

The first, absolutely critical thing to understand about launching a major rewrite is that it's going to take insanely longer than you expect. Even when you try to discount for the usual developer optimism. Here's why:

• Migrating the data sucks beyond all belief

I'm assuming your existing system has a bunch of valuable data locked up in it (if it doesn't, congrats, but I just never, ever run into this situation). You think, we're going to set up a new db structure (or move it all to some NoSQL store, or whatever), and we'll, I dunno, write some scripts to copy the data over, no problem.

Problem 1: there's this endless series of weird crap encoded in the data in surprising ways. E.g. "The use_conf field is 1 if we should use the auto-generated configs... but only if the spec_version field is greater than 3. Oh, and for a few months, there was this bug, and use_conf was left blank. It's almost always safe to assume it should be 1 when it's blank. Except for customers who bought the Express product, then we should treat it as 2". You have to migrate all your data over, checksum the living hell out of it, display it back to your users, and then figure out why it's not what they expect. You end up poring over commit histories, email exchanges with developers who have long since left the company, and line after line of cryptic legacy code. (In prep for writing this, when I mentioned this problem to developers, every single time they cut me off to eagerly explain some specific, awful experience they've had on this front -- it's really that bad)

Problem 2: But, wait, it gets worse: because you have a lot of data, it often takes days to migrate it all. So, as you struggle to figure out each of the above weird, persnickety issues with converting the data over, you end up waiting for days to see if your fixes work. And then to find the next issue and start over again. I have vivid, painful memories of watching my friend Stephen (a prototypical Smart Young Engineer), who was a tech lead on the Unhappy Rewrite, working, like, hour 70 of an 80 hour week, babysitting a slow-moving data export/import as it failed over and over and over again. I really can't communicate how long this takes.

• It's brutally hard to reduce scope

With a greenfield (non-rewrite) project, there is always (always) a severe reduction in scope as you get closer to launch. You start off, expecting to do A, B, C & D, but when you launch, you do part of A. But, often, people are thrilled. (And, crucially, they forget that they had once considered all the other imagined features as absolutely necessary)

With a rewrite, that fails. People are really unhappy if you tell them: hey, we rewrote your favorite part of the product, the code is a lot cleaner now, but we took away half the functionality.

You'll end up spending this awful series of months implementing all these odd edge cases that you didn't realize even existed. And backfilling support for features that you've been told no one uses any more, but you find out at the last minute some Important Person or Customer does. And, and, and...

• There turn out to be these other system that use "your" data

You always think: oh, yeah, there are these four screens, I see how to serve those from the new system. But then it turns out that a half-dozen cron jobs read data directly from "your" db. And there's an initialization step for new customers where something is stored in that db and read back later. And some other screen makes a side call to obtain a count of your data. Etc, etc. Basically, you try turning off the old system briefly, and a flurry of bug reports show up on your desk, for features written a long time ago, by people who have left the company, but which customers still depend on. This takes forever all over again to fix.

Okay, I'm Sufficiently Scared Now, What Should I Do?

You you have to totally own the business value.

First off, before you start, you must define the business value of this rewrite. I mean, you should always understand the big picture value of what you do (see: Rands Test). But with rewrites, it's often the tech lead, or the developers in general, who are pushing for the rewrite -- and then it's absolutely critical that you understand the value. Because you're going to discover unexpected problems, and have to make compromises, and the whole thing is going to drag on forever. And if, at the end of all that, the Important People who sign your checks don't see much value, it's not going to be a happy day for you.

One thing: be very, very careful if the primary business value is some (possibly disguised) version of "The new system will be much easier for developers to work on." I'm not saying that's not a nice bit of value, but if that's your only or main value... you're going to be trying to explain to your CEO in six months why nothing seems to have gotten done in development in the last half year.

The key to fixing the "developers will cry less" thing is to identify, specifically, what the current, crappy system is holding you back from doing. E.g. are you not able to pass a security audit? Does the website routinely fall over in a way that customers notice? Is there some sexy new feature you just can't add because the system is too hard to work with? Identifying that kind of specific problem both means you're talking about something observable by the rest of the business, and also that you're in a position to make smart tradeoffs when things blow up (as they will).

As an example, for our big Analytics rewrite, the developers involved sat down with Dan Dunn, the (truly excellent) product guy on our team, and worked out a list of business-visible wins we hoped to achieve. In rough priority order, those were:

• Cut cost of storing each hit by an order of magnitude
• Create new reports that weren't possible in the old system
• Serve all reports faster
• Serve near-real-time (instead of cached daily) reports

And you should know: that first one loomed really, really large. HubSpot was growing very quickly, and storing all that hit data as individual rows in SQLServer had all sorts of extra costs. The experts on Windows ops were constantly trying to get new SQLServer clusters set up ahead of demand (which was risky and complex and ended up touching a lot of the rest of the codebase). Sales people were told to not sell to prospects with really high traffic, because if they installed our tracking code, it might knock over those key databases (and that restriction injected friction into the sales process). Etc, etc.

Solving the "no more hits in SQLServer" problem is the Hard kind for a rewrite -- you only get the value when every single trace of the old system is gone. The other ones, lower down the list, those you'd see some value as individual reports were moved over. That's a crucial distinction to understand. If at all possible, you want to make sure that you're not only solving that kind of Hard Problem -- find some wins on the way.

For the Unhappy Rewrite, the biz value wasn't perfectly clear. And, thus, as often happens in that case, everyone assumed that, in the bright, shiny world of the New System, all their own personal pet peeves would be addressed. The new system would be faster! It would scale better! The front end would be beautiful and clever and new! It would bring our customers coffee in bed and read them the paper.

As the developers involved slogged through all the unexpected issues which arose, and had to keep pushing out their release date, they gradually realized how disappointed everyone was going to be when they saw the actual results (because all the awesome, dreamed-of stuff had gotten thrown overboard to try to get the damn thing out the door). This a crappy, crappy place to be -- stressed because people are hounding you to get something long-overdue finished, and equally stressed because you know that thing is a mess.

Okay, so how do you avoid getting trapped in this particular hell?

Worship at the Altar of Incrementalism

Over my career, I've come to place a really strong value on figuring out how to break big changes into small, safe, value-generating pieces. It's a sort of meta-design -- designing the process of gradual, safe change.

Kent Beck calls this Succession, and describes it as:

"Design changes are usually most efficiently implemented as a series of safe steps. Succession is the art of taking a single conceptual change, breaking it into safe steps, and then finding an order for those steps that optimizes safety, feedback, and efficiency."

I love that he calls it an "art" -- that feels exactly right to me. It doesn't happen by accident. You have to consciously work at it, talk out alternatives with your team, get some sort of product owner or manager involved to make sure the early value you're surfacing matters to customers. It's a creative act.

And now, let me say, in an angry Old Testament prophet voice: Beware the false incrementalism!

False incrementalism is breaking a large change up into a set of small steps, but where none of those steps generate any value on their own. E.g. you first write an entire new back end (but don't hook it up to anything), and then write an entire new front end (but don't launch it, because the back end doesn't have the legacy data yet), and then migrate all the legacy data. It's only after all of those steps are finished that you have anything of any value at all.

Fortunately, there's a very simple test to determine if you're falling prey to the False Incrementalism: if after each increment, an Important Person were to ask your team to drop the project right at that moment, would the business have seen some value? That is the gold standard.

Going back to my running example: our existing analytics system supported a few thousand customers, and served something like a half dozen key reports. We made an early decision to: a) rewrite all the existing reports before writing new ones, and b) rewrite each report completely, push it through to production, migrate any existing data for that report, and switch all our customers over. And only then move on to the next report.

Here's how that completely saved us: 3 months into a rewrite which we had estimated would take 3-5 months, we had completely converted a single report. Because we had focused on getting all the way through to production, and on migrating all the old data, we had been forced to face up to how complex the overall process was going to be. We sat down, and produced a new estimate: it would take more like 8 months to finish everything up, and get fully off SQLServer.

At this point, Dan Dunn, who is a Truly Excellent Product Guy because he is unafraid to face a hard tradeoff, said, "I'd like to shift our priorities -- I want to build the Sexy New Reports now, and not wait until we're fully off SQLServer." We said, "Even if it makes the overall rewrite take longer, and we won't get off SQLServer this year, and we'll have to build that one new cluster we were hoping to avoid having to set up?" And he said "Yes." And we said, "Okay, then."

That's the kind of choice you want to offer the rest of your larger team. An economic tradeoff where they can chose between options of what they see when. You really, really don't want to say: we don't have anything yet, we're not sure when we will, your only choices are to keep waiting, or to cancel this project and kiss your sunk costs goodbye.

Side note: Dan made 100% the right call (see: Excellent). The Sexy New Reports were a huge, runaway hit. Getting them out sooner than later made a big economic impact on the business. Which was good, because the project dragged on past the one year mark before we could finally kill off SQLServer and fully retire the old system.

For you product dev flow geeks out there, one interesting piece of value we generated early was simply a better understanding of how long the project was going to take. I believe that is what Beck means by "feedback". It's real value to the business. If we hadn't pushed a single report all the way through, we would likely have had, 3-4 months in, a whole bunch of data (for all reports) in some partially built new system, and no better understanding of the full challenge of cutting even one report over. You can see the value the feedback gave us--it let Dan make a much better economic choice. I will make my once-per-blog-post pitch that you should go read Donald Reinertsen's Principles of Product Development Flow to learn more about how reducing uncertainty generates value for a business.

For the Unhappy Rewrite, they didn't work out a careful plan for this kind of incremental delivery. Some Totally Awesome Things would happen/be possible when they finished. But they kept on not finishing, and not finishing, and then discovering more ways that the various pieces they were building didn't quite fit together. In the Post-Mortem, someone summarized it as: "We somehow turned this into a Waterfall project, without ever meaning to."

But, I Have to Cut Over All at Once, Because the Data is Always Changing

One of the reasons people bail on incrementalism is that they realize that, to make it work, there's going to be an extended period where every update to a piece of data has to go to both systems (old and new). And that's going to be a major pain in the ass to engineer. People will think (and even say out loud), "We can't do that, it'll add a month to the project to insert a dual-write layer. It wil slow us down too much."

Here's what I'm going to say: always insert that dual-write layer. Always. It's a minor, generally somewhat fixed cost that buys you an incredible amount of insurance. It allows you, as we did above, to gradually switch over from one system to another. It allows you to back out at any time if you discover major problems with the way the data was migrated (which you will, over and over again). It means your migration of data can take a week, and that's not a problem, because you don't have to freeze writes to both systems during that time. And, as a bonus, it surfaces a bunch of those weird situations where "other" systems are writing directly to your old database.

Again, I'll quote Kent Beck, writing about how they do this at Facebook:

"We frequently migrate large amounts of data from one data store to another, to improve performance or reliability. These migrations are an example of succession, because there is no safe way to wave a wand and migrate the data in an instant. The succession we use is:

Convert data fetching and mutating to a DataType, an abstraction that hides where the data is stored.

Modify the DataType to begin writing the data to the new store as well as the old store.

Bulk migrate existing data.

Modify the DataType to read from both stores, checking that the same data is fetched and logging any differences.

When the results match closely enough, return data from the new store and eliminate the old store.

You could theoretically do this faster as a single step, but it would never work. There is just too much hidden coupling in our system. Something would go wrong with one of the steps, leading to a potentially disastrous situation of lost or corrupted data."

Abandoning the Project Should Always Be on the Table

If a 3-month rewrite is economically rational, but a 13-month one is a giant loss, you'll generate a lot value by realizing which of those two you're actually facing. Unfortunately, the longer you solider on, the harder it is for people to avoid the Fallacy of Sunk Costs. The solution: if you have any uncertainty about how long it's going to take, sequence your work to reduce that uncertainty right away, and give people some "finished" thing that will let them walk away. One month in, you can still say: we've decided to only rewrite the front end. Or: we're just going to insert an API layer for now. Or, even: this turned out to be a bad idea, we're walking away. Six months in, with no end in sight, that's incredibly hard to do (even if it's still the right choice, economically).

Some Specific Tactics

Shrink Ray FTW

This is an excellent idea, courtesy of Kellan Elliot-McCrea, CTO of Etsy. He describes it as follows:

"We have a pattern we call shrink ray. It's a graph of how much the old system is still in place. Most of these run as cron jobs that grep the codebase for a key signature. Sometimes usage is from wire monitoring of a component. Sometimes there are leaderboards. There is always a party when it goes to zero. A big party.

Gives a good sense of progress and scope, especially as the project is rolling, and a good historical record of how long this shit takes. '''

I've just started using Shrink Ray on a rewrite I'm tackling right now, and I will say: it's fairly awesome. Not only does it give you the wins above, but, it also forces you to have an early discussion about what you are shrinking, and who in the business cares. If you make the right graph, Important People will be excited to see it moving down. This is crazy valuable.

Engineer The Living Hell Out Of Your Migration Scripts

It's very easy to think of the code that moves data from the old system to the new as a collection of one-off scripts. You write them quickly, don't comment them too carefully, don't write unit tests, etc. All of which are generally valid tradeoffs for code which you're only going to run once.

But, see above, you're going to run your migrations over and over to get them right. Plus, you're converting and summing up and copying over data, so you really, really want some unit tests to find any errors you can early on (because "data" is, to a first approximation, "a bunch of opaque numbers which don't mean anything to you, but which people will be super pissed off about if they're wrong"). And this thing is going to happen, where someone will accidentally hit ctrl-c, and kill your 36 hour migration at hour 34. Thus, taking the extra time to make the entire process strongly idempotent will pay off over and over (by strongly idempotent, I mean, e.g. you can restart after a failed partial run and it will pick up most of the existing work).

Basically, treat your migration code as a first class citizen. It will save you a lot of time in the long run.

If Your Data Doesn't Look Weird, You're Not Looking Hard Enough

What's best is if you can get yourself to think about the problem of building confidence in your data as a real, exciting engineering challenge. Put one of your very best devs to work attacking both the old and the new data, writing tools to analyze it all, discover interesting invariants and checksums.

A good rule of thumb for migrating and checksumming data: until you've found a half-dozen bizarre inconsistencies in the old data, you're not done. For the Analytics Rewrite, we created a page on our internal wiki called "Data Infelicities". It got to be really, really long.

With Great Incrementalism Comes Great Power

I want to wrap up by flipping this all around -- if you learn to approach your rewrites with this kind of ferocious, incremental discipline, you can tackle incredibly hard problems without fear. Which is a tremendous capability to offer your business. You can gradually rewrite that unbelievably horky system that the whole company depends on. You can move huge chunks of data to new data stores. You can pick up messy, half-functional open source projects and gradually build new products around them.

It's a great feeling.

---

What's your take?  Care to share any lessons learned from an epic rewrite?

Like this article? You can now find more popular articles as part of the LinkedIn Influencers program.

How To Survive a Ground-Up Rewrite Without Losing Your Sanity

'via Blog this'

out of Memory Error in Hadoop on ubuntu - 128m limit - really? - thanks to Stack Overflow

out of Memory Error in Hadoop Q: I tried installing Hadoop following thishttp://hadoop.apache.org/common/docs/stable/single_node_setup.html document. When I tried executing this bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+' I am getting the following Exception java.lang.OutOfMemoryError: Java heap space Please suggest a solution so that i can try out the example. The entire Exception is listed below. I am new to Hadoop I might have done something dump . Any suggestion will be highly appreciated. .... A: For anyone using RPM or DEB packages, the documentation and common advice is misleading. These packages install hadoop configuration files into /etc/hadoop. These will take priority over other settings. The /etc/hadoop/hadoop-env.sh sets the maximum java heap memory for Hadoop, by Default it is: export HADOOP_CLIENT_OPTS="-Xmx128m $HADOOP_CLIENT_OPTS" This Xmx setting is too low, simply change it to this and rerun export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS" share|improve this answer edited Jun 20 '12 at 23:41 ILikeFood 9117 answered Mar 16 '12 at 15:49 Zach Garner 816185

that fixes the issue... – polerto Apr 7 at 22:36 I just had the exact same problem as the OP, and I was using the RPM package. This fixed the problem. Upvoted. – Aaron Aug 12 at 20:13

[ed: so in Connie's ubuntu install, this worked (don't try on a real network with automounts, just funny little standalone ubuntu systems, where you don't know where hadoop got installed):

find / -name hadoop-env.sh -print 2>/dev/null|xargs grep -l Xmx128m|sudo xargs sed -i 's/Xmx128m/Xmx2048m/'

sudo `which hadoop` jar ../*examp* grep input output 'dfs[a-z.]+'
# That was the example given in the docs, which now runs for me without error...

Why modify files by hand when you can use find and sed -i...   :-) ]

java - out of Memory Error in Hadoop - Stack Overflow

'via Blog this'

Test-driven development - from Wikipedia, the free encyclopedia

Test-driven development

From Wikipedia, the free encyclopedia

Software development process
A software engineer programming at work
Activities and steps
Requirements   Specification   Architecture Design   Implementation   Testing   Debugging Deployment   Maintenance
Methodologies
Waterfall   Prototype model   Incremental Iterative   V-Model   Spiral   Scrum   Cleanroom RAD   DSDM   RUP   XP   Agile   Lean Dual Vee Model   TDD   FDD
Supporting disciplines
Configuration management   Documentation Quality assurance (SQA)   Project management User experience design
Tools
Compiler   Debugger   Profiler   GUI designer IDE   Build automation
v   t   e

Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: first the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test, and finally refactors the new code to acceptable standards. Kent Beck, who is credited with having developed or 'rediscovered' the technique, stated in 2003 that TDD encourages simple designs and inspires confidence.^[1]

Test-driven development is related to the test-first programming concepts of extreme programming, begun in 1999,^[2] but more recently has created more general interest in its own right.^[3]

Programmers also apply the concept to improving and debugging legacy codedeveloped with older techniques.^[4]

Contents   [hide]  1 Requirements 1.1 xUnit frameworks 2 Test-driven development cycle 2.1 Add a test 2.2 Run all tests and see if the new one fails 2.3 Write some code 2.4 Run the automated tests and see them succeed 2.5 Refactor code 2.6 Repeat 3 Development style 4 Best practices 4.1 Test structure 4.2 Individual best practices 4.3 Practices to avoid, or "anti-patterns" 5 Benefits 6 Shortcomings 7 Code visibility 8 Fakes, mocks and integration tests 9 TDD for complex systems 9.1 Architecting for testability 9.2 Managing tests for large teams 10 See also 11 References 12 External links

[edit]Requirements

In test-driven development, the developers create automated unit tests to define code requirements, then immediately write the code themselves. The tests contain assertions that are either true or false. Passing the tests confirms correct behavior as developers evolve and refactor the code.

[edit]xUnit frameworks

Developers may use computer-assisted testing frameworks, such as xUnit, to create and automatically run sets of test cases. Xunit frameworks provide assertion-style test validation capabilities and result reporting. These capabilities are critical for automation as they move the burden of execution validation from an independent post-processing activity to one that is included in the test execution. Many popular xUnit frameworks are openly available:

• JUnit
• cppUnit
• UnitTest++
• Unity (for C)
• NUnit
• PHPUnit

This concept of built-in test oracles helps to reduce the unit test maintenance burden by requiring maintenance on only one artifact and eliminating the independent validation of often complex and fragile output. Additionally, the execution framework provided by these test frameworks allows for the automatic execution of all system test cases or various subsets along with other features.^[5]

[edit]Test-driven development cycle

A graphical representation of the development cycle, using a basicflowchart

The following sequence is based on the book Test-Driven Development by Example.^[1]

[edit]Add a test

In test-driven development, each new feature begins with writing a test. This test must inevitably fail because it is written before the feature has been implemented. (If it does not fail, then either the proposed "new" feature already exists or the test is defective.) To write a test, the developer must clearly understand the feature's specification and requirements. The developer can accomplish this through use cases and user stories to cover the requirements and exception conditions. This could also imply a variant, or modification of an existing test. This is a differentiating feature of test-driven development versus writing unit tests after the code is written: it makes the developer focus on the requirements beforewriting the code, a subtle but important difference.

[edit]Run all tests and see if the new one fails

This validates that the test harness is working correctly and that the new test does not mistakenly pass without requiring any new code. This step also tests the test itself, in the negative: it rules out the possibility that the new test will always pass, and therefore be worthless. The new test should also fail for the expected reason. This increases confidence (although it does not entirely guarantee) that it is testing the right thing, and will pass only in intended cases.

This additional step in test-first programming is the main difference between it and invariant-based programming.

[edit]Write some code

The next step is to write some code that will cause the test to pass. The new code written at this stage will not be perfect and may, for example, pass the test in an inelegant way. That is acceptable because later steps will improve and hone it.

It is important that the code written is only designed to pass the test; no further (and therefore untested) functionality should be predicted and 'allowed for' at any stage.

[edit]Run the automated tests and see them succeed

If all test cases now pass, the programmer can be confident that the code meets all the tested requirements. This is a good point from which to begin the final step of the cycle.

[edit]Refactor code

Now the code can be cleaned up as necessary. By re-running the test cases, the developer can be confident that code refactoring is not damaging any existing functionality. The concept of removing duplication is an important aspect of any software design. In this case, however, it also applies to removing any duplication between the test code and the production code — for example magic numbers or strings that were repeated in both, in order to make the test pass in step 3.

[edit]Repeat

Starting with another new test, the cycle is then repeated to push forward the functionality. The size of the steps should always be small, with as few as 1 to 10 edits between each test run. If new code does not rapidly satisfy a new test, or other tests fail unexpectedly, the programmer should undo or revert in preference to excessive debugging. Continuous integration helps by providing revertible checkpoints. When using external libraries it is important not to make increments that are so small as to be effectively merely testing the library itself,^[3] unless there is some reason to believe that the library is buggy or is not sufficiently feature-complete to serve all the needs of the main program being written.

[edit]Development style

There are various aspects to using test-driven development, for example the principles of "keep it simple stupid" (KISS) and "You aren't gonna need it" (YAGNI). By focusing on writing only the code necessary to pass tests, designs can be cleaner and clearer than is often achieved by other methods.^[1] In Test-Driven Development by Example, Kent Beck also suggests the principle "Fake it till you make it".

To achieve some advanced design concept (such as a design pattern), tests are written that will generate that design. The code may remain simpler than the target pattern, but still pass all required tests. This can be unsettling at first but it allows the developer to focus only on what is important.

Write the tests first. The tests should be written before the functionality that is being tested. This has been claimed to have many benefits. It helps ensure that the application is written for testability, as the developers must consider how to test the application from the outset, rather than worrying about it later. It also ensures that tests for every feature will be written. Additionally, writing the tests first drives a deeper and earlier understanding of the product requirements, ensures the effectiveness of the test code, and maintains a continual focus on the quality of the product.^[6] When writing feature-first code, there is a tendency by developers and the development organisations to push the developer on to the next feature, neglecting testing entirely. The first test might not even compile, at first, because all of the classes and methods it requires may not yet exist. Nevertheless, that first test functions as an executable specification.^[7]

First fail the test cases. The idea is to ensure that the test really works and can catch an error. Once this is shown, the underlying functionality can be implemented. This has been coined the "test-driven development mantra", known as red/green/refactor where red means fail and green is pass.

Test-driven development constantly repeats the steps of adding test cases that fail, passing them, and refactoring. Receiving the expected test results at each stage reinforces the programmer's mental model of the code, boosts confidence and increases productivity.

Keep the unit small. For TDD, a unit is most commonly defined as a class or group of related functions, often called a module. Keeping units relatively small is claimed to provide critical benefits, including:

• Reduced Debugging Effort – When test failures are detected, having smaller units aids in tracking down errors.
• Self-Documenting Tests – Small test cases have improved readability and facilitate rapid understandability.^[6]

Advanced practices of test-driven development can lead to Acceptance Test-driven development (ATDD) where the criteria specified by the customer are automated into acceptance tests, which then drive the traditional unit test-driven development (UTDD) process.^[8] This process ensures the customer has an automated mechanism to decide whether the software meets their requirements. With ATDD, the development team now has a specific target to satisfy, the acceptance tests, which keeps them continuously focused on what the customer really wants from that user story.

[edit]Best practices

[edit]Test structure

Effective layout of a test case ensures all required actions are completed, improves the readability of the test case, and smooths the flow of execution. Consistent structure helps in building a self-documenting test case. A commonly applied structure for test cases has (1) setup, (2) execution, (3) validation, and (4) cleanup.

• Setup: Put the Unit Under Test (UUT) or the overall test system in the state needed to run the test.
• Execution: Trigger/drive the UUT to perform the target behavior and capture all output, such as return values and output parameters. This step is usually very simple.
• Validation: Ensure the results of the test are correct. These results may include explicit outputs captured during Execution or state changes in the UUT.
• Cleanup: Restore the UUT or the overall test system to the pre-test state. This restoration permits another test to execute immediately after this one.^[6]

[edit]Individual best practices

• Separate common set up and teardown logic into test support services utilized by the appropriate test cases.
• Keep each test oracle focused on only the results necessary to validate its test.
• Design time-related tests to allow tolerance for execution in non-real time operating systems. The common practice of allowing a 5-10 percent margin for late execution reduces the potential number of false negatives in test execution.
• Treat your test code with the same respect as your production code. It also must work correctly for both positive and negative cases, last a long time, and be readable and maintainable.
• Get together with your team and review your tests and test practices to share effective techniques and catch bad habits. It may be helpful to review this section during your discussion.^[9]

[edit]Practices to avoid, or "anti-patterns"

• Do not have test cases depend on system state manipulated from previously executed test cases.
• A test suite where test cases are dependent upon each other is brittle and complex. Execution order has to be specifiable and/or constant. Basic refactoring of the initial test cases or structure of the UUT causes a spiral of increasingly pervasive impacts in associated tests.
• Interdependent tests can cause cascading false negatives. A failure in an early test case breaks a later test case even if no actual fault exists in the UUT, increasing defect analysis and debug efforts.
• Do not test precise execution behavior timing or performance.
• Do not try to build “all-knowing oracles.” An oracle that inspects more than necessary is more expensive and brittle over time than it needs to be. This very common error is dangerous because it causes a subtle but pervasive time sink across the complex project.^[9]

[edit]Benefits

A 2005 study found that using TDD meant writing more tests and, in turn, programmers who wrote more tests tended to be more productive.^[10] Hypotheses relating to code quality and a more direct correlation between TDD and productivity were inconclusive.^[11]

Programmers using pure TDD on new ("greenfield") projects reported they only rarely felt the need to invoke a debugger. Used in conjunction with a version control system, when tests fail unexpectedly, reverting the code to the last version that passed all tests may often be more productive than debugging.^[12]

Test-driven development offers more than just simple validation of correctness, but can also drive the design of a program.^{[citation needed]} By focusing on the test cases first, one must imagine how the functionality will be used by clients (in the first case, the test cases). So, the programmer is concerned with the interface before the implementation. This benefit is complementary to Design by Contract as it approaches code through test cases rather than through mathematical assertions or preconceptions.

Test-driven development offers the ability to take small steps when required. It allows a programmer to focus on the task at hand as the first goal is to make the test pass. Exceptional cases and error handling are not considered initially, and tests to create these extraneous circumstances are implemented separately. Test-driven development ensures in this way that all written code is covered by at least one test. This gives the programming team, and subsequent users, a greater level of confidence in the code.

While it is true that more code is required with TDD than without TDD because of the unit test code, total code implementation time is typically shorter.^[13] Large numbers of tests help to limit the number of defects in the code. The early and frequent nature of the testing helps to catch defects early in the development cycle, preventing them from becoming endemic and expensive problems. Eliminating defects early in the process usually avoids lengthy and tedious debugging later in the project.

TDD can lead to more modularized, flexible, and extensible code. This effect often comes about because the methodology requires that the developers think of the software in terms of small units that can be written and tested independently and integrated together later. This leads to smaller, more focused classes, looser coupling, and cleaner interfaces. The use of the mock object design pattern also contributes to the overall modularization of the code because this pattern requires that the code be written so that modules can be switched easily between mock versions for unit testing and "real" versions for deployment.

Because no more code is written than necessary to pass a failing test case, automated tests tend to cover every code path. For example, in order for a TDD developer to add an else branch to an existing if statement, the developer would first have to write a failing test case that motivates the branch. As a result, the automated tests resulting from TDD tend to be very thorough: they will detect any unexpected changes in the code's behaviour. This detects problems that can arise where a change later in the development cycle unexpectedly alters other functionality.

[edit]Shortcomings

• Test-driven development is difficult to use in situations where full functional tests are required to determine success or failure. Examples of these are user interfaces, programs that work with databases, and some that depend on specific networkconfigurations. TDD encourages developers to put the minimum amount of code into such modules and to maximize the logic that is in testable library code, using fakes and mocks to represent the outside world.
• Management support is essential. Without the entire organization believing that test-driven development is going to improve the product, management may feel that time spent writing tests is wasted.^[14]
• Unit tests created in a test-driven development environment are typically created by the developer who will also write the code that is being tested. The tests may therefore share the same blind spots with the code: If, for example, a developer does not realize that certain input parameters must be checked, most likely neither the test nor the code will verify these input parameters. If the developer misinterprets the requirements specification for the module being developed, both the tests and the code will be wrong, as giving a false sense of correctness.
• The high number of passing unit tests may bring a false sense of security, resulting in fewer additional software testing activities, such as integration testing and compliance testing.
• The tests themselves become part of the maintenance overhead of a project. Badly written tests, for example ones that include hard-coded error strings or which are themselves prone to failure, are expensive to maintain. This is especially the case with Fragile Tests.^[15] There is a risk that tests that regularly generate false failures will be ignored, so that when a real failure occurs, it may not be detected. It is possible to write tests for low and easy maintenance, for example by the reuse of error strings, and this should be a goal during the code refactoring phase described above.
• Overtesting can consume time both to write the excessive tests, and later, to rewrite the tests when requirements change. Also, more-flexible modules (with limited tests) might accept new requirements without the need for changing the tests. For those reasons, testing for only extreme conditions, or a small sample of data, can be easier to adjust than a set of highly detailed tests. However, developers could be warned about overtesting to avoid the excessive work, but it might require advanced skills insampling or factor analysis.
• The level of coverage and testing detail achieved during repeated TDD cycles cannot easily be re-created at a later date. Therefore these original, or early, tests become increasingly precious as time goes by. The tactic is to fix it early. Also, if a poor architecture, a poor design, or a poor testing strategy leads to a late change that makes dozens of existing tests fail, then it is important that they are individually fixed. Merely deleting, disabling or rashly altering them can lead to undetectable holes in the test coverage.

[edit]Code visibility

Test suite code clearly has to be able to access the code it is testing. On the other hand, normal design criteria such as information hiding, encapsulation and the separation of concerns should not be compromised. Therefore unit test code for TDD is usually written within the same project or module as the code being tested.

In object oriented design this still does not provide access to private data and methods. Therefore, extra work may be necessary for unit tests. In Java and other languages, a developer can use reflection to access fields that are marked private.^[16]Alternatively, an inner class can be used to hold the unit tests so they will have visibility of the enclosing class's members and attributes. In the .NET Framework and some other programming languages, partial classes may be used to expose private methods and data for the tests to access.

It is important that such testing hacks do not remain in the production code. In C and other languages, compiler directives such as#if DEBUG ... #endif can be placed around such additional classes and indeed all other test-related code to prevent them being compiled into the released code. This then means that the released code is not exactly the same as that which is unit tested. The regular running of fewer but more comprehensive, end-to-end, integration tests on the final release build can then ensure (among other things) that no production code exists that subtly relies on aspects of the test harness.

There is some debate among practitioners of TDD, documented in their blogs and other writings, as to whether it is wise to test private methods and data anyway. Some argue that private members are a mere implementation detail that may change, and should be allowed to do so without breaking numbers of tests. Thus it should be sufficient to test any class through its public interface or through its subclass interface, which some languages call the "protected" interface.^[17] Others say that crucial aspects of functionality may be implemented in private methods, and that developing this while testing it indirectly via the public interface only obscures the issue: unit testing is about testing the smallest unit of functionality possible.^[18][19]

[edit]Fakes, mocks and integration tests

Unit tests are so named because they each test one unit of code. A complex module may have a thousand unit tests and a simple module may have only ten. The tests used for TDD should never cross process boundaries in a program, let alone network connections. Doing so introduces delays that make tests run slowly and discourage developers from running the whole suite. Introducing dependencies on external modules or data also turns unit tests into integration tests. If one module misbehaves in a chain of interrelated modules, it is not so immediately clear where to look for the cause of the failure.

When code under development relies on a database, a web service, or any other external process or service, enforcing a unit-testable separation is also an opportunity and a driving force to design more modular, more testable and more reusable code.^[20] Two steps are necessary:

1. Whenever external access is going to be needed in the final design, an interface should be defined that describes the access that will be available. See the dependency inversion principle for a discussion of the benefits of doing this regardless of TDD.
2. The interface should be implemented in two ways, one of which really accesses the external process, and the other of which is a fake or mock. Fake objects need do little more than add a message such as “Person object saved” to a trace log, against which a test assertion can be run to verify correct behaviour. Mock objects differ in that they themselves contain test assertions that can make the test fail, for example, if the person's name and other data are not as expected.

Fake and mock object methods that return data, ostensibly from a data store or user, can help the test process by always returning the same, realistic data that tests can rely upon. They can also be set into predefined fault modes so that error-handling routines can be developed and reliably tested. In a fault mode, a method may return an invalid, incomplete or null response, or may throw anexception. Fake services other than data stores may also be useful in TDD: A fake encryption service may not, in fact, encrypt the data passed; a fake random number service may always return 1. Fake or mock implementations are examples of dependency injection.

A Test Double is a test-specific capability that substitutes for a system capability, typically a class or function, that the UUT depends on. There are two times at which test doubles can be introduced into a system: link and execution. Link time substitution is when the test double is compiled into the load module, which is executed to validate testing. This approach is typically used when running in an environment other than the target environment that requires doubles for the hardware level code for compilation. The alternative to linker substitution is run-time substitution in which the real functionality is replaced during the execution of a test cases. This substitution is typically done through the reassignment of known function pointers or object replacement.

Test doubles are of a number of different types and varying complexities:

• Dummy – A dummy is the simplest form of a test double. It facilitates linker time substitution by providing a default return value where required.
• Stub – A stub adds simplistic logic to a dummy, providing different outputs.
• Spy – A spy captures and makes available parameter and state information, publishing accessors to test code for private information allowing for more advanced state validation.
• Mock – A mock is specified by an individual test case to validate test-specific behavior, checking parameter values and call sequencing.
• Simulator – A simulator is a comprehensive component providing a higher-fidelity approximation of the target capability (the thing being doubled). A simulator typically requires significant additional development effort.^[6]

A corollary of such dependency injection is that the actual database or other external-access code is never tested by the TDD process itself. To avoid errors that may arise from this, other tests are needed that instantiate the test-driven code with the "real" implementations of the interfaces discussed above. These are integration tests and are quite separate from the TDD unit tests. There will be fewer of them, and they need to be run less often than the unit tests. They can nonetheless be implemented using the same testing framework, such as xUnit.

Integration tests that alter any persistent store or database should always be designed carefully with consideration of the initial and final state of the files or database, even if any test fails. This is often achieved using some combination of the following techniques:

• The TearDown method, which is integral to many test frameworks.
• try...catch...finally exception handling structures where available.
• Database transactions where a transaction atomically includes perhaps a write, a read and a matching delete operation.
• Taking a "snapshot" of the database before running any tests and rolling back to the snapshot after each test run. This may be automated using a framework such as Ant or NAnt or a continuous integration system such as CruiseControl.
• Initialising the database to a clean state before tests, rather than cleaning up after them. This may be relevant where cleaning up may make it difficult to diagnose test failures by deleting the final state of the database before detailed diagnosis can be performed.

[edit]TDD for complex systems

Exercising TDD on large, challenging systems requires:

• A modular architecture.
• Well-defined components with published interfaces.
• Disciplined system layering with maximization of platform independence.

These proven practices yield increased testability and facilitate the application of build and test automation.^[6]

[edit]Architecting for testability

Complex systems require an architecture that meets a range of requirements. A key subset of these requirements includes support for the complete and effective testing of the system. Effective modular design yields components that share traits essential for effective TDD.

• High Cohesion ensures each unit provides a set of related capabilities and makes the tests of those capabilities easier to maintain.
• Low Coupling allows each unit to be effectively tested in isolation.
• Published Interfaces restrict Component access and serve as contact points for tests, facilitating test creation and ensuring the highest fidelity between test and production unit configuration.

A key technique for building effective modular architecture is Scenario Modeling where a set of sequence chart is constructed, each one focusing on a single system-level execution scenario. The Scenario Model provides an excellent vehicle for creating the strategy of interactions between components in response to a specific stimulus. Each of these Scenario Models serves as a rich set of requirements for the services or functions that a component must provide, and it also dictates the order that these components and services will interact together. Scenario modeling can greatly facilitate the construction of TDD tests for a complex system.^[6]

[edit]Managing tests for large teams

In a larger system the impact of poor component quality is magnified by the complexity of interactions. This magnification makes the benefits of TDD accrue even faster in the context of larger projects. However, the complexity of the total population of tests can become a problem in itself, eroding potential gains. It sounds simple, but a key initial step is to recognize that test code is also important software and should be produced and maintained with the same rigor as the production code.

Creating and managing the architecture of test software within a complex system is just as important as the core product architecture. Test drivers interact with the UUT, test doubles and the unit test framework.^[6]

Test-driven development - Wikipedia, the free encyclopedia

'via Blog this'