Rob Lally : Robert Lally

Estimation: ““

(From the genius that is xkcd.com.)

I was talking with my friend Clarke Ching today about Kanban and pull systems in general. I came up with a metaphor that we both liked and thought I would share it.

Problem : Imagine you want to get a ball of string ( your features ) through to the other side of a funnel ( your development team ).

Traditional methods involved pushing REALLY, REALLY HARD until it goes through.

Agile and other iterative methods realise that trying to force a whole ball of string through a funnel at once is madness, so they chop the string up into smaller pieces which are easier to push through.

Pull systems, such as Kanban, change the dynamic, rather than pushing the string through, they feed the tip of the string through and then pull it as fast as they can.

Which of these is best?

Well, traditional methods often cheat by mashing some subset of the string through and declaring victory. Or by making the team ( the funnel ) enormous so that you can fit a fairly large ball of string through the hole without effort. Or by minimising the features so that they’re dealing with a tiny ball of string. Or some combination of the above. I’m not sure this is ever a good idea.

Agile works pretty well, but the chopping takes effort ( iteration planning ) and consumes time. But it does break down into nice, neat iterations which provide clear points for releases, milestones, measurement and feedback.

Kanban works pretty well too, it might be slightly more efficient than an iterative Agile approach, but the milestone and release points need to be superimposed onto it since there are no natural break points.

Lean and ToC have their place here too - tools, techniques and strategies to make the hole in the funnel wider and to make sure that the string that gets through is the right string, used as efficiently as possible. Stretching the funnel as far as possible, just as I have stretched this metaphor.

I was reading through the principles page on the Agile Manifesto site today and I realised that I’ve never really been very good at one of the items.

The best architectures, requirements, and designs emerge from self-organizing teams.

I’ve managed a lot of teams, and for the most part, I’ve been deemed pretty successful. My management style was always to try to blend in with the team, to be one of them, to get people to move forwards by setting an example or by encouraging them. I never introduce or describe anyone as working for me, I always say I’m working with them or we’re on the same team. Of course, there are management activities that need to be performed (reviews, promotions, demotions, bonus awards etc), and I’m happy doing those things. I’ve never felt that knowing people, or treating them as the equals (which, of course, they are) hampered me in any way. The more closely you work with someone, the more of the real them you see. You get a better idea of their potential, their skills, their interests, ways in which they could contribute more to the organisation.

I’m actually lying when I say I’ve never introduced anyone who worked for me as a subordinate. My friend and mentor, Gerry Martin, who was instrumental in my decision to return to college and retrain in software, eventually came to work for me. Every time I got to introduce him as my minion, I got a kick out of it. But, hey, if you can’t poke fun at your best-friend, who can you poke fun at. (Sorry Gerry, you know I love you.)

My attempt at egalitarian leadership didn’t always work out as I intended. One of the things I didn’t consider, is that because I could ignore differences in organisational authority, it didn’t mean that others felt the same. This led to one incident where I upset someone through an ill-considered joke that left them feeling angry, impotent and embarrassed.

I feel that, for the most part, the people on my teams were given authority and autonomy, everyone was given the opportunity to contribute as much as they wanted or were able to. There was opportunity for decision making, direction setting, personal contribution, group collaboration and individual growth. But, were my teams self-organising? Not really. I was always there trying to guide people in the direction I felt we should be moving. Not by edict, decree or demand, but by persuasion, example and leadership. But, in the end, however gently the boss tries to guide you, you’re still aware that he’s the boss and his opinion carries more weight than merit alone. It wasn’t a democracy or a meritocracy: even if everyone disagreed with me silently or publicly, it wouldn’t have led to my replacement.

Is this a problem? Perhaps. I’m not sure what the alternative is. I firmly believe, that leadership is vital to the success of any organisation. By leadership, I don’t just mean “somebody has to be in charge”, I mean that someone has to accept responsibility and authority in equal measure and then use that authority judiciously, sparingly and only for the good of the organisation and all of its stakeholders.

Is a truly self-organising team possible in a commercial organisation?

I’ve been mulling over my dissatisfaction with BDD, and I’m finally in a position to write down my thoughts.

In short, BDD doesn’t actually seem to be a coherent thing. There are many different perspectives on it, and many different tools with slightly different focusses. The only consistent thing is that the word ’should’ is required to be used anywhere that you would traditionally have used ‘assert’.

To be clear, my argument isn’t with the tools that have been developed: I’m not sure I have a need for ScalaTest or easyb or RSpec or Cucumber, but I think they’re interesting and I hope that future development will lead to something I find compelling. My problem is with the definition of BDD itself.

BDD started off being touted as a replacement for TDD. TDD was too hard, people confused TDD with a testing methodology, people didn’t understand TDD was about design, tests were too hard to read: so the BDD inventors and proponents suggested. Some of this may have been true, there are many people who fail to grasp that TDD was primarily, but not exclusively, about design. Dan North, Dave Astels and others decided that the way to reduce confusion was to change the name; which always reminded me of the way Borland decided to improve sales by changing their name to Inprise.

So here we have it, BDD is a replacement for TDD, now it uses specs and the word ’should’ rather than tests and the word ‘assert’. I thought it was a bad idea at the time because you don’t make things easier for people to understand by having more than one way to say things.

	assertEquals(oneWayToDoThings(), "good");
	many_ways_to_do_something shouldBe 'bad'

I have no doubt that the creators of BDD and RSpec had good intentions. I have no desire to challenge their integrity or their creativity.

As time has gone on, BDD has morphed into a creature that is neither fish nor fowl. BDD specs have morphed, in many cases, to be ‘executable requirements specifications’ which is a laudable goal, but leaves a number of holes.

If a spec is an executable requirement then it is written in customer language. If it is written in customer language then it is of no use as a tool for driving design of classes, methods, objects and functions. So, if this is the home of BDD then it isn’t a replacement for TDD it is something that lives alongside TDD. Mention this to a BDD proponent, though, and you get an argument. You can write specs that cover the same ground as unit tests. Yes you can, but what are the advantages of doing so? You’ll have wordy, obfuscated tests carrying the baggage added to make BDD more palatable for customers and, you still miss out on many of the benefits of TDD.

TDD, as a design methodology, forces you to write code that is used by two different clients: the tests and the final application. This is one of the ways that it helps promote flexibility. It gives you an early look at ‘how exactly am I going to use this class/object’. DBB frameworks don’t look or work like the code that will be calling the final code, so there really is no benefit here. Take easyb as an example, it doesn’t even use the same language as the code you’re writing (assuming the common use case of using easyb as a BDD framework for Java development).

TDD highlights pain points when writing tests, if it is hard to write a test for, your code probably needs a little work. BDD frameworks such as Cucumber, can quite happily support specs written in terms of ‘the system’ or ‘the application’ and you have to write grotesque code to make it work under the covers. This doesn’t make Cucumber a bad tool, just the wrong tool for the job of designing code.

So what is the output of BDD: User Acceptance Tests? Functional Tests? Unit Tests? System Tests? A system design? The design of individual units? It depends on who you talk to.

I’ve seen specs that are written from a customer point of view, useful as user acceptance tests. I’ve seen these mashed until they talk about an application in terms of controllers and pages and objects, making them useless as user acceptance tests because they’re not what the user agreed to and they no longer understand them well enough to re-agree to them. They’re also sub-optimal as drivers for design because they’re based on the implementation assumptions of the end user and they’re constrained to be about the subject matter that the user was originally talking about. This leaves lots of areas undefined where you need design but don’t have explicit user requirements because they just want it to work. The users will not tell you that they want to ensure that you have properly nested resource de-allocation blocks in your JDBC code because that’s not their domain. Do you need to have it? Yes, absolutely. Do you want it to work every time. Yes, absolutely. Will a badly designed solution make your code a pain to work with forever, yes absolutely.

So what happens? Developers have to write more specs, for things that users actively don’t want to know about.

I’m not convinced by BDD as a means of writing functional specifications either ( this doesn’t include User Acceptance Tests ). The best functional specs I’ve seen were elegant documents that used text, screenshots, charts, tables, diagrams, flowcharts and even embedded sound files to describe as concisely and accurately as possible the users expectations. Constraining these documents to a structured textual format is, at the moment, a bad idea. We’re reinventing Knuth’s Literate Programming ideas of the seventies, only we’re trying to impose it on clients and analysts too. There may be a future here, but I don’t think we’re here yet.

Particularly egregious offenders here are those cases where the functional specification is written in embedded strings in a document that is otherwise code. Conflating two completely different documents, and levels of abstraction into a single chimera-like whole that serves nobody well.

Ten years ago, I worked for a company where all development was done in UML and we forward generated the code from our diagrams. This didn’t work well, it was slow and painful but ‘technically’ it was possible so those in charge kept pushing the idea. Writing functional specifications using BDD frameworks is ‘technically’ possible but that doesn’t make it a good idea. Not here, not now.

We’ve learned new ways of building tools, and we’re looking for ways to leverage them. BDD isn’t AN answer, so it certainly can’t be THE answer.

I’ve been reading W.E. Pete Peterson’s excellent book Almost Perfect (available to read on line here) and he has some interesting things to say about his attempts to add copy protection to WordPerfect. At the time he was running WordPerfect Corporation and was one of only three shareholders, so his perspective isn’t an academic one:

“It was simply not fair to make the honest, paying customers put up with an inconvenience that had been made necessary by the dishonest ones. In the end, what was good for the legal customers was also good for our bottom line.”

The book has been a most entertaining and educational read. It’s great to read the story of an ethical, decent man who built by success without sacrificing his principles. Particularly heartening from a technologists perspective is that he attributes the companies early success to the quality of their engineering staff and their ability to hang on to smart individuals and focussed teams.

Sadly the book has been out of print for a very long time, but second hand copies are available on Amazon, that’s where I picked up my copy.

David Anderson’s InfoQ talk on Kanban and software development is fascinating. In particular I liked the clarity of the numbers he produced to prove the successes of his methods. I also liked the photographs of the tracking boards that different groups were using, David’s analysis of them and the way they changed over time as groups came into contact with each other.

All in all, a thought provoking presentation that backs ideas up with concrete numbers.

I like the teams I run to use pair programming, because the end product is better. I’ve also heard lots of people say that they don’t want to do it.

I have a theory that pair programming is the brussels sprout of the technology world. Lots of people don’t eat them because, not because they think they’re bad for you, but just because they don’t like them

I have no children. When I do, they’ll eat brussels sprouts. I won’t do it because I want them to be sad, or to make those little sicky faces that children pull when they put something unfamiliar in their mouth. I’ll do it because I want them to be healthy, productive and to live a long time.

I’ve been working on my CV recently and finding it hard to quantify what I do and am. I’ve worked in almost every capacity in technology:

• Developer
• Process Engineer
• Architect
• Team Leader
• Project Manager
• Analyst
• Product Manager
• Programme Manager
• Service Delivery Manager
• Support Engineer
• Agile Coach
• RUP Mentor
• Technology Evangelist and Trainer
• Trouble-shooter

And a whole bunch of other roles that don’t even have good names. I’ve loved every one of these roles, I’m fascinated with every aspect of technology. Beyond that, I like to flatter myself that I’ve acquitted myself well in all of these roles. I love technology, I love managing people. I love guiding and participation of the creation of something out of nothing. Well, nothing more than sweat, tears, inspiration and a compiler.

I’ve always loved the idea of being a polymath, a renaissance man - but sadly my interests have always been restricted to one field. Luckily it is a pretty big field and there are few people that have become masters of it all. I’m not there. Not even close. But I’d like to be.

Everyone needs a goal in life. “Be the best you can be at the thing you love” works for me.

Here’s a rule of thumb I like to use.

Q. How do you know when a document is too long to be useful?

A. Print it out and try to slip it under the door of the person you wrote it for. If it doesn’t fit … it’s too big.

As a an addendum to my test; whilst you’re at the door, open it, go inside, and say hello. Nine times out of ten, that’ll do more good than the document.

Last week I said that I’m using the time I have free to catch up on my reading - getting to all those books I’ve been meaning to get through.

First on my list was Eli Goldratt’s latest book The Choice. It’d been a while since I read his other works, so I decided to re-read them first. In the last few days I’ve re-read The Goal, Necessary But Not Sufficient, Critical Chain and It’s Not Luck. Every time I read these books, I get something new out of them. If you’ve not read Goldratt’s work, I heartily recommend it.

I said that I was re-reading Goldratt’s earlier works in preparation for reading his latest book. I haven’t got to that yet, that’s next on my list. I got distracted when a friend tried to explain something to me by referencing Seth Godin’s Purple Cow theory and, since I haven’t read the book, I didn’t know what he was talking about. So I read ordered a copy from Amazon and read that last-night and this morning.

This is the first book I’ve read that just focusses on the ‘getting the message out’ end of marketing. I’m surprised that I got as much out of it as I did. In retrospect, this is a field I should have spent more time reading about. I need to work to fill in my knowledge here. Anyone with pointers towards the ‘right’ reading material feel free to drop me a line.

For the last couple of days, I’ve been chewing over an experience I had about a decade ago. I’m trying to understand the underlying thought processes that led a problem.

About a decade ago, I worked for a project manager that I’ll call Mr D. He was (and probably still is) intelligent, wise, kind and dedicated. A genuinely decent human being. I learned a great deal by working under him.

I worked for Mr D at a company that was having some problems, Mr D was brought in to try to fix things. One of the problems we had was a slavish dedication to a development process that didn’t work. Mr D saw this, and took my recommendation to bring in an Agile coach and migrate from RUP with all the trimmings to XP. We worked through this process change and things picked up, they got a lot better in fact. But not better enough.

At the time, I was too inexperienced to see all the other mistakes we were making as an organisation. In the end, it didn’t really matter what we did, by the time we started making changes the project was years behind schedule and the writing was on the wall.

Here’s the incident that has bothered me for the last decade.

Our team was following a classic XP process, we were pair-programming most of the time and we decided right at the start to estimate in ‘pair-hours’ rather than in hours to keep things simple. Our velocity fluctuated wildly over the first few iterations and eventually settled to a consistent level. Our estimation unit was hours - that was what we started with, but over time we had come to think of them as arbitrary units, we could have changed the name to ‘arbitrary work units’ but it didn’t seem important. Everyone was working hard all day, distractions had been minimised and people were feeling good about our progress. We knew we could be better, we knew we could improve and we were. Every week were getting a little better.

Mr D, who I realise in retrospect must have been protecting our team from tremendous outside pressure, came in one day and announced that he was changing things again. It was his call, I don’t have a problem with that. It is his reasoning that bothers me. He had totalled up the estimates of work accomplished for the last few weeks, totalled up the available man hours and calculated that we were only fifty percent efficient. He wasn’t even complaining about the pair-programming, he was calculating things in pair-hours rather than hours. When people tried to point out the basis for our estimates wasn’t hours, he became enraged.

In summary:

He knew the basis for our estimates wasn’t hours.

He worked in the same room as everyone else, so he could see with his own eyes that people were working hard all day every day.

But still he came to the conclusion that an efficiency calculation he’d used in the past was telling him something was wrong and that he had to make a change.

Why, despite the evidence of his own eyes, did he come to this conclusion? He didn’t need an excuse to make a change, it was his house and his rules. It don’t even think that he was trying to justify his decision to the team: nobody agreed that his justification made sense, but he kept insisting that we were, at best, only fifty percent efficient.

Was it pressure and panic that made him turn to something he knew, even if he knew it didn’t make sense?

Was it a rejection of the methods we were using, trying to turn the team towards a structure he knew and had more faith in?

Did he really believe that we were only fifty-percent efficient?

I wish I knew.

Over the years, there have been many times that people have made decisions I didn’t agree with, but at least I understood their reasoning. In this case, I don’t disagree with the decisions Mr D made, I just wish I could understand his reasoning. Deep down, the fact that I don’t understand scares me. How can I stop this happening again, if I don’t understand what happened the first time?

Agile development is more culture than process: “Why thinking of agile as culture and not just process explains resistance and difficulty in teaching and learning the approach”

(Via Jeff Patton.)

Jeff Patton presents a well thought out, well articulated article. But one I couldn’t disagree with more.

Jeff presents an argument that Agile isn’t a process, but a culture, and I’d agree that Agile isn’t a process but, despite the many Agile proponents who make it seem that way, Agile isn’t a culture either. Agility is just one quality that an organisation can have. An organisation can have an Agile culture, but that doesn’t mean that that their culture is Agility.

Presenting Agile as a culture is something I’ve seen many times, and it puts more people off. Agile zealots, a club I was a member of a decade ago, try to sell Agile as a culture in a way that reminds me of Mao and his Cultural Revolution; they want to come in to your organisation, declare your culture to be bankrupt and rip it out, replacing it with an idealised regime of their own. (I do not put Jeff Patton in this category, his writing is very sensitive to the existing culture of organisations.) Mao famously said that political change can only come through the barrel of a gun. Mao’s way may have changed china. But it isn’t the way to change organisations in the twenty-first century.

An organisation that wants to undergo an Agile transition is an organisation that wants to acquire a new quality. Adopting this new quality is not easy, that’s why experienced Agile practitioners can be invaluable during this process. Cultural change is not easy, it is painful for organisations and the people that make them up. I think this is why Jeff, and the rest of us who’ve been down this road, have seen push-back and pain when performing transitions.

You must be sensitive to the needs and feelings of an organisation. You must be conscious of the times when existing cultural practices, processes and artifacts are in conflict with the changes you are trying to make. You can’t steam-roll people and tell them that their existing culture is valueless, you have to show them what they have, where they can be and help them make the journey.

After the transition an organisation should have Agile qualities. They shouldn’t be Agile. An organisation that has no quality but Agility can not survive, they have nothing unique to offer the world. As someone performing an agile transition, we should strive to ensure that the organisations we work with should be strong, energetic, diverse, healthy and Agile. They should have their own culture at the end of the process, just as they did at the start. Just a bit better.

Over the last few years, I’ve moved past Agile. Not that I think it is bad, I still embrace it fully, and I still use Agile practices because I believe they work. But Agile isn’t everything, any more. It never was, I just didn’t see all the other things that I and those around me were doing. I think that Agile brought the humanity back to software development, something that had been bleeding out and something that we need to be effective. Agility recognises that it is people who write software and that by remembering that fact, we get the most out of the people.

These days I consider myself to be in favour of Humane Development. Respect people, respect their culture, challenge people to do more, to do better, reward them for successes, help them when they falter, teach them to be the best they can be and expect their best efforts from them in return. Like Agility, organisations need to have Humane as a characteristic if they are to thrive.