I’ve been mulling over my dissatisfaction with BDD, and I’m finally in a position to write down my thoughts.
In short, BDD doesn’t actually seem to be a coherent thing. There are many different perspectives on it, and many different tools with slightly different focusses. The only consistent thing is that the word ’should’ is required to be used anywhere that you would traditionally have used ‘assert’.
To be clear, my argument isn’t with the tools that have been developed: I’m not sure I have a need for ScalaTest or easyb or RSpec or Cucumber, but I think they’re interesting and I hope that future development will lead to something I find compelling. My problem is with the definition of BDD itself.
BDD started off being touted as a replacement for TDD. TDD was too hard, people confused TDD with a testing methodology, people didn’t understand TDD was about design, tests were too hard to read: so the BDD inventors and proponents suggested. Some of this may have been true, there are many people who fail to grasp that TDD was primarily, but not exclusively, about design. Dan North, Dave Astels and others decided that the way to reduce confusion was to change the name; which always reminded me of the way Borland decided to improve sales by changing their name to Inprise.
So here we have it, BDD is a replacement for TDD, now it uses specs and the word ’should’ rather than tests and the word ‘assert’. I thought it was a bad idea at the time because you don’t make things easier for people to understand by having more than one way to say things.
assertEquals(oneWayToDoThings(), "good"); many_ways_to_do_something shouldBe 'bad'
I have no doubt that the creators of BDD and RSpec had good intentions. I have no desire to challenge their integrity or their creativity.
As time has gone on, BDD has morphed into a creature that is neither fish nor fowl. BDD specs have morphed, in many cases, to be ‘executable requirements specifications’ which is a laudable goal, but leaves a number of holes.
If a spec is an executable requirement then it is written in customer language. If it is written in customer language then it is of no use as a tool for driving design of classes, methods, objects and functions. So, if this is the home of BDD then it isn’t a replacement for TDD it is something that lives alongside TDD. Mention this to a BDD proponent, though, and you get an argument. You can write specs that cover the same ground as unit tests. Yes you can, but what are the advantages of doing so? You’ll have wordy, obfuscated tests carrying the baggage added to make BDD more palatable for customers and, you still miss out on many of the benefits of TDD.
TDD, as a design methodology, forces you to write code that is used by two different clients: the tests and the final application. This is one of the ways that it helps promote flexibility. It gives you an early look at ‘how exactly am I going to use this class/object’. DBB frameworks don’t look or work like the code that will be calling the final code, so there really is no benefit here. Take easyb as an example, it doesn’t even use the same language as the code you’re writing (assuming the common use case of using easyb as a BDD framework for Java development).
TDD highlights pain points when writing tests, if it is hard to write a test for, your code probably needs a little work. BDD frameworks such as Cucumber, can quite happily support specs written in terms of ‘the system’ or ‘the application’ and you have to write grotesque code to make it work under the covers. This doesn’t make Cucumber a bad tool, just the wrong tool for the job of designing code.
So what is the output of BDD: User Acceptance Tests? Functional Tests? Unit Tests? System Tests? A system design? The design of individual units? It depends on who you talk to.
I’ve seen specs that are written from a customer point of view, useful as user acceptance tests. I’ve seen these mashed until they talk about an application in terms of controllers and pages and objects, making them useless as user acceptance tests because they’re not what the user agreed to and they no longer understand them well enough to re-agree to them. They’re also sub-optimal as drivers for design because they’re based on the implementation assumptions of the end user and they’re constrained to be about the subject matter that the user was originally talking about. This leaves lots of areas undefined where you need design but don’t have explicit user requirements because they just want it to work. The users will not tell you that they want to ensure that you have properly nested resource de-allocation blocks in your JDBC code because that’s not their domain. Do you need to have it? Yes, absolutely. Do you want it to work every time. Yes, absolutely. Will a badly designed solution make your code a pain to work with forever, yes absolutely.
So what happens? Developers have to write more specs, for things that users actively don’t want to know about.
I’m not convinced by BDD as a means of writing functional specifications either ( this doesn’t include User Acceptance Tests ). The best functional specs I’ve seen were elegant documents that used text, screenshots, charts, tables, diagrams, flowcharts and even embedded sound files to describe as concisely and accurately as possible the users expectations. Constraining these documents to a structured textual format is, at the moment, a bad idea. We’re reinventing Knuth’s Literate Programming ideas of the seventies, only we’re trying to impose it on clients and analysts too. There may be a future here, but I don’t think we’re here yet.
Particularly egregious offenders here are those cases where the functional specification is written in embedded strings in a document that is otherwise code. Conflating two completely different documents, and levels of abstraction into a single chimera-like whole that serves nobody well.
Ten years ago, I worked for a company where all development was done in UML and we forward generated the code from our diagrams. This didn’t work well, it was slow and painful but ‘technically’ it was possible so those in charge kept pushing the idea. Writing functional specifications using BDD frameworks is ‘technically’ possible but that doesn’t make it a good idea. Not here, not now.
We’ve learned new ways of building tools, and we’re looking for ways to leverage them. BDD isn’t AN answer, so it certainly can’t be THE answer.
May 27th, 2009 at 10:10 pm
I think you’ve put your finger very close to what *I* have found unsatisfying about the notion. I know a lot of folks, tho, who believe that they’re doing something both different and better. Not convinced yet. Thanks for adding to my perspective!
Hill
May 27th, 2009 at 10:40 pm
I have to strongly disagree, I’m afraid. The combination of Cucumber acceptance tests and RSpec unit tests is easily the most satisfying I have ever used for test-driven development.
Using the domain language of the customer in your acceptance tests, whether you’re writing them in Fit, Cucumber or straight-up xUnit, is an un-disputably good habit. BDD just makes this a first-class practice, and provides the tooling to make it easy.
I think this is the key but subtle point that people who dismiss BDD miss out on. Forcing yourself to wear the customer’s hat to write an automated test before you write any code can save a lot of headaches later by driving out any misunderstandings and uncertainty early on. Who wouldn’t want that?
Rspec should probably not be called a BDD framework - it’s really just a nice way to write unit tests, and I think you’d struggle to find anyone who uses RSpec and Cucumber regularly who would claim much differently. I’m surprised you’ve found that ‘BDD frameworks don’t look or work like the code that will be calling the final code’ - this is exactly how I use RSpec every day.
Thanks for sharing your thoughts though - it’s an interesting debate
May 27th, 2009 at 11:23 pm
Interesting post….
what do you make of the fact that easyb has won the 19th annual Jolt award! in the Testing Tools category?
May 28th, 2009 at 3:09 am
Not all BDD’s spec is written for customer - yes, cucumber may belongs to this category, but rspec doesn’t.
I think BDD is strong related with TDD - I like the Borland metaphor
So if TDD isn’t an answer, then BDD either; but if TDD is an answer, BDD should be an answer too.
May 28th, 2009 at 5:38 am
In case your interested… Here are my thoughts regarding your argument on the rspec-users mailing list:
http://rubyforge.org/pipermail/rspec-users/2009-May/014704.html
May 28th, 2009 at 6:08 am
I wrote too much, so here’s a link to my response:
http://www.continuousthinking.com/2009/5/28/bdd-not-so-much-really
May 28th, 2009 at 6:15 am
Hi Matt,
I’m glad you’re happy with your tools. If you’ve found a pattern where they work really well for you.. fantastic. My point was never about tools, it was always about the methods by which one uses them.
Your point seems to be that you use Cucumber for acceptance tests and RSpec for TDD. And together this makes BDD. But then you say that RSpec probably isn’t a DBB framework.
What does that leave? BDD == Cucumber == acceptance testing? I’m not buying that.
If RSpec isn’t a BDD framework then easyb and ScalaTest aren’t either. Can you point me at some other things that you’d call a BDD tool?
I have no problem with acceptance testing. I am a strong advocate of having acceptance tests before you start writing code - otherwise how do you really know when you’re done. I’m sure we agree 100% here.
But this isn’t new, it already has a name. A good name. Acceptance tests. Adding a non-descriptive name, a name highly polluted by previous usage doesn’t help. If I were pairing with you used BDD when you meant acceptance tests I’d ask for the keyboard.
As for the section about BDD code that doesn’t look like actual code, I hoped I was clear that I was talking about tools such as easyb where the testing isn’t even done in the same language.
May 28th, 2009 at 6:20 am
Hi Philip,
I make nothing at all of the award that easyb won an award. I always said I had no problems with the tools, only the philosophy behind their use in BDD.
Interestingly easyb won as a testing tool, not as a design tool.
Beyond that, my belief in awards as a ‘Good Thing’ has been jaded by too many years of watching self selecting groups skew polls. The number of years that VisualStudio won IDE of the Year awards when it was nothing more than notepad with highlighting and code completion when IDEA and Eclipse and JBuilder and TogetherJ were 1000 times better was depressing. Rational Rose won countless awards as a design tool and as a productivity tool.. and I never found that to be useful.
So, the awaard doesn’t make me feel one way, or another. It is what it is. I do hope that the developers of easyb were pleased with it. I’m sure they worked hard and any recognition must be nice.
May 28th, 2009 at 6:25 am
Hi Jan,
Your definition of BDD seems to be different from Matt who commented before you. Matt, as I understand it from the limited context of a single comment, feels that BDD lies aligned more with functional testing than TDD. This kind of makes my point. Without agreement about what it is, it isn’t really a useful term, yet.
May 28th, 2009 at 7:46 am
I need to be less long-winded, another follow-up:
http://continuousthinking.com/2009/5/28/bdd-not-so-much-really-part-ii
Not to shamelessly keep posting links, but I feel horrible for writing a multi-page comment on your blog.I think there’s a confusion that BDD == toolset (ie. Cucumber, Cucumber+RSpec, rspec). That needs to stop because its wrong and that means we’re going to end up with x number of definitions at least where x is the number of unique combinations between tools that can be used in a BDD style manner.
May 28th, 2009 at 1:51 pm
I think you are missing the point of where BDD has evolved to. It is no longer a ‘replacement for TDD,’ but, rather, it is a workflow for creating a system that more closely resembles what the client is looking for. Cucumber as a system behavior specification tool provides a set of executable specifications that can be used to show what the system currently does, as well as provide a framework for discussing what desired behavior is missing. The Cucumber features drive the requirements that are then implemented using a lower-level spec-ing tool like RSpec. It isn’t Cucumber that drives the design, it is RSpec. Just like it isn’t RSpec that drives the system specifications, it is Cucumber.
Focusing on and critiquing individual portions of the process outside of their context confuses the issue. I’m going to make an assumption here, but from your confusion as to what BDD is and is used for, it sounds as though you have not spent a significant enough amount of time working with the process/tools to get a full understanding of their value.
Statements like ‘BDD isn’t AN answer, so it certainly can’t be THE answer’ show a bit too much ‘I know everything’ attitude. There are people who say the same sorts of things about , but, in general, most descriptive processes are THE answer for someone. Is it the BEST answer for that person, probably not, but the important thing is that there is something.
It is important to realize that there are no THE answers that work across the board, but there do exist answers for people. BDD fills the need for a lot of people. The fact that you don’t see its benefit does not mean it can’t work.