Andrew on Agile

During my ScrumMaster certification, Mike Cohn recommended that the size of a cross functional scrum team should be between 5 and 9.  In a recent exercise, we tried splitting a scrum team three ways – 5-6 people in each team - for the next release.

At the end of last year I counted 16 people in a scrum.  For most of the release the team size averaged 12, consisting of a mix of product managers, QA, front-end engineers, software engineers and the ScrumMaster.  16?  It never meant to get that big, but with an addition here and an addition there, it did.  It was too big.

The scrums were starting to take too long, there were too many stories in the sprint for one team to truly keep track and impediments were flying around everywhere.  Risk was starting to creep into each sprint and eventually the release.

I put it to the team that they should split in half for the next release and while some members were ok, others really objected to the idea.  They felt it would generate unnecessary tension between a team that had grown strong together.

Due to the type of work within the first sprint in that next release the team worked as one again, but agreed tentatively to an unequal split (1/3 to 2/3) for the following two (and final) sprints.  It was ok, both teams had a couple of issues mainly down to the fact that they were building three applications which all needed to communicate with one another, but the smaller team certainly preferred being small.

So now they’ve split into 3.

It’s brilliant.  The teams are made up of 1-2 QA, 2-3 Engineers and a Product Manager.  The teams were encouraged to go through the agile scrum motions how they saw fit.  The smaller size enabled intense collaboration during a short planning phase, resulting comprehensive stories.  Poker and sprint planning was a breeze, less big personalities to get in the way of proceedings, and it was so much easier to keep on top of the progress during the sprint.  The software’s of high quality.  It’s done.  It contains a good level of tests and the quality of the code is great.

Now that risk is greatly reduced.  In the sprints before if something started to go wrong it could have affected the whole team.  All members would get distracted by the issue and if it didn’t get resolved and the goal compromised, the team would be left demoralised by all the drama such failed sprints bring.  Oh yeah, and the team members love it.

So if it was going good, then the team gets bigger and you start to have problems, or if you’re trying things for the first time and some members aren’t engaging, break it down.

I’ve been having a series of conversations lately with one of my engineers about learning on the job.   By this I mean that technological problems get figured out during a sprint.  These problems usually arise from implementing technologies that aren’t so well known by the engineering team.  A recent example for us would be finding solutions for complex queries with Fluent NHibernate.  The conversations were sparked by recent events during our last release where a sprint goal was compromised by the introduction of a new Domain Driven Design (DDD) architecture, and the previously mentioned object relational mapper (ORM), Fluent NHibernate.

The engineer’s argument for learning on the job centres on the satisfaction gained by solving technical challenges presented by new technological ideas.  As a manager, engineer’s job satisfaction is very important to me.  In an unsettled economic climate such as the one we’ve just been (and perhaps still are going) through, it becomes even more important to use everything available to keep morale high.  Our favourable stance to adopting new technologies over the past couple of years – we built a production app in .Net MVC right through its preview stages, releasing the app as MVC was released – has kept the engineers interested in their jobs.   That being said, introducing a new technology is accompanied by risk, risk that can turn into a high cost.  Learning on the job during the sprint is risky because it can create impediments that are very difficult to remove.

In order to find the right balance it is important that all the engineers know what they’re doing throughout the sprint.  If a sticking point arises, e.g. a situation where you notice an engineer stuck with their head down for hours on end, a couple of scrums pass where they say they’ve got 1 or 2 hours left and they’ll be finished that day but they never do, then you know you have a problem.  Sometimes this can be solved by pairing, but probably it’s a sign the team hasn’t done enough homework.

We just halted the introduction of a messaging technology into the next release and are mid way through a two week planning phase.  The aim is to take a small sideways step in order to make sure the engineers are technically up to speed with what’s going into it. 

The focus of the first planning week has been on story generation, next week we’ll be exploring those stories further to find potential engineering sticking points. When they are found the whole engineering team will come together to investigate the solution.  This object of this exercise is not to create solutions for all our potential problems up front, as any changes to the backlog during the release could render this a waste of time, but to use the potential problems as case studies to ensure the engineers have enough knowledge about the technologies we have introduced before embarking on the sprints.

This is not something we will do between every release.  A series of weekly, hour long sessions to contextually spike the implementation of the messaging technology using a current production app will get underway as soon as the release starts.  We will only introduce it across our range of production apps when the team feels ready.

This is what I consider learning on the job.  There will inevitably be some learning during the sprint, but the aim should be to control that as much as possible.

I wanted to post this as an update to the original post on how to estimate story points using the poker planning technique and involving your QA team.

In the first post I describe how a poker planning session almost broke down when including the Quality Assurers in estimating the relative size of each of our stories.  Just to get through  that session we ended up producing one or two estimates that didn’t include the testing time.

It’s amazing how adaptable people can be.  On our second attempt, only 2 weeks later, I’m very pleased to say our session went brilliantly.  They key to this success, I feel was down to the ScrumMaster.  They took the initiative, pre-empting another possible disastrous session and clarified some of our own rules.

Last time, because both engineering and QA were unused to thinking about delivering stories to done including each other’s time, we ended up settling on the highest value either the engineers, or QA, or a hybrid could agree on.  There was very little unanimity, nor was anyone really convinced we were providing estimates to the best of our ability. 

This time we started our session by reviewing a very simple process designed for our team that the ScrumMaster had put together.  If consensus could not be reached by the first attempt at laying cards, try again.  If still no consensus after a round of discussion, try again.  On the third attempt if there was still no consensus we would settle on the highest value, as we had in the previous session.  Mike Cohn suggests in his “Agile Estimating and Planning” book to “…continue the process as long as estimates are moving closer together…”, but in the past this hasn’t worked for us hence the 3 time cap.

From there on in, the team set about story pointing.  We had about 9 stories to estimate and in about an hour and a half we were done.  Estimating for each other’s disciplines still didn’t come easy for everyone, but when someone struggled, the team worked with them so they could understand what they were thinking about.  Only once did we not reach a consensus by the third attempt, and that story had such a high estimation it would need to be broken down further.  The mood of the session was very positive leaving all present extremely satisfied with the result.

So, the first stage particular exercise to incorporate QA into our story point estimation sessions has been a success.  Yet another leap forward in our journey.

There will be more to follow on sprint planning with QA and integrating the seating plan of teams who are already co-located.

Good stuff.

We had just finished the first stage of a green field application and missed our sprint goals.  Damn!  When we first saw a problem, should we have pulled some work out of the sprint?

 This is not something anyone should be afraid of doing.

Let’s look at what happened.

The competent team had worked together for 18 months on various projects.  A new style of architecture was to be introduced to improve testability and in preparation for this shift we had invested in several internal group educational sessions, as well as some good debates during sprints.  At the start of the sprint we didn’t commit to as much work as we had capacity to do, however we were pretty close.

After 4 days we hadn’t delivered anything to test.  Complications with the precise approach to the new architecture were holding up the project.  At the end of the 5^th day we still had nothing into QA, so we held a 2^nd daily scrum.  We agreed if we finished engineering a couple of  stories so testing could start by the end of the next day, the engineers would finish the rest of their work during the week and time could be made up by engineers pair testing with QA.  A little ‘waterfalacy’ but it was only 1 sprint so never mind!

Can you guess what happened?  We didn’t meet the sprint goals and were left with about 40 hours worth of work.

I’ve been in several sprints like this where for various reasons the goals haven’t been met.  The end of the sprint is unbearable.  The team gets micro managed; frustration grows and turns into panic – a desperate rush to get things done during the last couple of days – then there’s the realisation at the 11th hour of failure. 

Failure!

I don’t like that word, nor does the team who’ve worked so hard to try to reach their goals.  The effect it has on morale is a real pain to manage and it doesn’t encourage better quality work.  To avoid ‘failure’, teams cut corners and in my experience the tests – so important in modern-day software engineering – go first.

So what should we have done? 

The real focus should have been right at the beginning in sprint planning.  The potential unknowns associated with incorporating a new technical design into a green field project should have been taken into consideration by the team when confirming their commitment.  It’s easy to say in hindsight, but the team should have committed to less in the first place.

That said, if the team are confident that their commitment is achievable, as they were in this instance, and you find yourself in the same place as us, you have three options.  The first is to carry on building, accepting the risk of digging yourself a deeper hole.  The second: abort the sprint.  The third is to pull some stories from the sprint with the approval of the stakeholders.

We know the first option is a BIG risk as you may well end up compromising the quality of your product.  The second, I don’t think the situation was serious enough for an abnormal termination.  I prefer the third option of redressing the expectations and pulling something out of the sprint.  I just think it’s more practical.  There is no need for all the drama the other two options create and the sooner the stakeholders are informed of any problems the better.

I don’t think this solution is right for every situation.  What’s the point of having a goal if you can just move the goal posts when you feel like it?  But sometimes, such as the example Mike Cohn alluded to during my ScrumMaster certification where you have a team new to scrum who are finding their velocity, or this situation where there was a good reason for changing the architecture, then I think it’s the best solution as long as everyone agrees.

Previously in our sprints we included QA time by adding a buffer to the engineer’s task estimates.   From now on, to provide greater visibility of the challenges QA might face whilst testing each story, we have started to include them in both the story point estimating and tasking sessions.

We started out by suggesting that the team (QA and Engineering) collectively estimate a story point value not only for how long it takes to engineer, but also how long it may take to test (relative to another of course). 

This didn’t go as expected. 

Some members got it and were able to take into account the advice or suggestions from their peers and re-evaluate accordingly.  Others found it very difficult to accept how QA and engineering could effectively estimate each other’s time.  

Such reservations were confirmed during the estimation of the second story.  The engineers agreed a 5, QA initially put down a 1.  The engineers then suggested to QA what might be involved in the story, but QA re-evaluated only a 2.  This resulted in a lively debate about what we were doing, how “pointless” this all was, etc, etc and the session pretty much broke down.

The strongest opinion of what to do next was for the engineers to story point their value, QA to story point their value and then add the two together.  For the sake of moving on we tried it.  We went back to the first story, engineers a 2, QA a 1 so we got 3.  Fine.  Next, engineers 5, QA 3 and we have 8.  This is easy!  Story 3: engineers re-evaluate a bit between 13 and 20 but settled on 20, QA 5.  Ok, so we suggest either re-evaluating again collectively to see if we accept 20, or we round up to 40.  Nope.  Some of the engineers wanted to stick with the story point value of 25 and the previous debate flared up again.

Both engineers and QA understand why we story point; that we use the Fibonacci sequence to balance out the estimations.   They’d heard the bucket of water metaphor Mike Cohn used during my ScrumMaster certification ”… you have 25 litres of water, that’s not going to fit in a 20 litre bucket, you should use a 40″.  They want (and agree with the need) to be more integrated, but just struggled with the concept of collectively settling on a figure.  There was alot of confusion over the level of accuracy required.

After a lo————ng debate, to get through the session a consensus was reached.  In order to obtain our team velocity we just needed to be consistent over time.  The team settled on this: QA story point their value, engineer’s their value and to accept the highest.  From that point the session finished quite quickly; everyone was a bit frazzled but content with the result.

What we have done is wrong.  Settling on the highest value is fine if QA vote 1 and engineers vote 13.  But what if engineers vote 13 and QA vote 8?  Or QA vote 20 and engineers vote 13?  The real estimates for those last two examples would surely be nearer 20 and 40?  We are not providing an accurate estimate to the best of our knowledge at the time.  This is important for management to plan a release.  It’s difficult to accurately estimate for a long release period, but we should not knowingly underestimate.  This will need to be addressed again during the next session.

The goal is that the team will naturally start considering each other during estimating sessions.  They will get there.  Even the senior engineer who raised the most questions about the process at one point put down a 40 and justified it by saying “well there’s a lot to test”. 

We have just started with test automation which will help to bring QA and engineering together further.  We might also get there more quickly with smaller teams; yesterday’s was 3 QA and 7 engineers.

The positives from the story pointing session?  We can now clearly see the different challenges that QA face from story to story, which sometimes do not have a direct correlation with engineering time for the story.  This is definitely going to help us with both planning and execution.

Ever wondered how to support your client’s service expectations, or provide enhancements to products you’ve built in the past, while trying to develop a market leading flagship product?  I recently attended a presentation by Nik Silver of Guardian News and Media (GNM) at the Agile Business Conference in London (view, twitter: #agilebc09), where he mentioned that GNM does this under the guise of “Business as Usual”.  Attending that presentation in some way encouraged me to write this as I’ve been assessing the value (ROI) of running such a concept for a while now.

The scope of this article is not to propose a formula for measuring value, instead I want to provide an example of how the efficiency of maintaining legacy applications could be optimized in order to maximize their value.

I work for a large media organization, within an agile scrum digital department which includes two Product Managers, eight Software Engineers (including my role as Engineering Manager) and four Quality Assurers (QA), as well as Project, Creative Service and Client Service teams.  Over the last few years (pre agile) the department has developed several software applications that support various internal business divisions.  Each of these applications, with the exception of one, was engineered by different individuals.  The quality of the code varies and domain knowledge has declined over the years because team members have moved on; to access that domain knowledge, the department is now forced to rely either on myself for most of the domains, or another long standing member of the team for the rest.  Until recently these applications were neglected due to a lack of resource, as in 2009 our entire focus switched to the rebuilding of our flagship product (I’ll refer to this as the “main release” from now on).

Four months ago a colleague from Product and I established an “Ops Rota”.  Its objectives were:

• Alleviate client pressure to support the different applications
• Expose the Product Managers, Engineers and QA to each of those applications
• Avoid costly new version releases in order to fulfill application roadmap expectations

In order to achieve these objectives the Ops Rota would require the diversion of resource away from the continued development of the main release. This would inevitably impact productivity, however, we agreed to allocate one software engineer and one quality assurer full time.

Last month we assessed the Ops Rota’s output against its three objectives and it was considered a success: our clients have visibility on the development of their products and in four months we have reduced the number of defects for each legacy application by 60-70%, Engineers and QA had been working on and therefore exposed to applications other than the main release and even some new features are making it into production.  Additionally, the Ops Rota was managed with our established agile scrum methods and those involved have really enjoyed working on it.  During the last month, since I’ve handed the management of it over to the project team (who now assume all scrum mastering responsibilities), it’s become clear how much it’s still dependent on me.

On closer inspection things don’t look so good.  The process of delivering bug fixes or features is inefficient because the iterations are often disrupted, something that was covered up by my “affectionate” determination to make the Ops Rota work.  Unless we make some changes, achieving the original objectives may not be possible.  Our problems are outlined below:

1. Only occasionally are tasks (sprint backlog items) added to bugs or product backlog items; very rarely are acceptance specifications defined.  The impact of this can be illustrated as follows: One feature of a particular domain is a revenue report. It constantly has bugs, or requires enhancements to parts that have no record of the work that’s been done before.  Over time different engineers have made many assumptions about how revenue should be reported because acceptance specifications for the expected outcome have never been defined.  The code is now in a very bad state, it’s extremely fragile and a nightmare for any engineer to work on.

1. Our Engineer’s exposure to the different domains is not happening quickly enough.  They rotate every two weeks to fit in with the main release’s sprint cycle; an Engineer can spend their whole two weeks within one domain fixing bugs or building new features and when their turn comes around again, Product’s priorities mean they could be working on the same domain.

1. Sprint iterations tend to focus on only one domain.

1. Resource is often removed to compensate for issues arising in a main release’s sprint.  This impacts the capacity of the Ops Rota iteration, compromising the ability to accomplish goals.

1. Releases are infrequent.

1. Feature bottlenecks build up with QA.  After development within a domain, QA testing may not start for several weeks, mostly due to resource availability.  When testing does start, in order to answer any questions, I’ve been picking up an item where the previous engineer left off, as they’ve moved back into the main release and disturbing them is disruptive [see Joel on Software, last three paragraphs].  This disjointed development – QA testing, combined with QA’s preferred method of testing all bugs and features for any particular domain in one go – delays a production release.  We are not using branching effectively, this leads to the “stacking up” of untested domain features on the trunk.  If a problem gets identified in the production application that requires an emergency fix, you guessed it, we can’t release because of all the “stacked up” untested features on the trunk.

1. Builds are still not automated.

It’s surprising how many principles and practices get ignored when the elements of a particular project management methodology break down.  For example, we have invested a great deal of time with the Engineers on “Agile Design” in our main release; TDD, before we got there the SOLID principles, refactoring, paired programming and so on.  We’re using scrum: iterations, daily stand ups, and a task board.  All these principles and practices are part of daily life in the main release but as you can see not in the Ops Rota, the scrums starts are irregular, we ignore the burn down most of the time and the retrospectives and demos aren’t formalized.

To me, this has raised the question: for those problems to manifest in an organization where our implementation of the agile scrum methodology is pretty mature for the main release cycles, are we using the right kind of agile methodology for this particular environment?  Should we invest more time and effort trying to make scrum work or is there something more suitable?

The “Lean Stream”

One of the Ops Rota’s greatest successes has been delivering to the client bug fixes or feature requests that enhance their product.  Whether or not this has translated to an increase in revenue, something that we can use to measure ROI, it’s too early to say but it’s certainly easier to walk past people in the corridor now.  By starting the Ops Rota, we are in a much better position than we were a year ago to react quickly to those software issues in applications that can hold the business back.  How though, especially with the volatility of resource availability, can we become more efficient and have half a chance at fully achieving the original objectives?

Over the last few weeks I have been looking at the Kanban and Lean ways of practicing Agile.  According to David Joyce, Lean Software Engineering’s processes can be described as work in progress limits, queues and control loops.  My summarized interpretation of that is to limit the amount of features under development and manage that limited amount through to the done state, then release.

“As adding features becomes more straightforward we may find that we are in a state akin to production and can take advantage of a Kanban approach to further optimize for reduced cycle time, increased throughput and just in time scheduling” David Draper on agile & design:

By its very nature the Ops Rota is similar to a production line so let’s explore this further.   First, let’s try to simplify the existing problems into areas:

Documenting and Quality

Product, Engineer and QA exposure to applications

Resource availability

Releasing

I’ll slightly alter the Ops Rota’s scrum management process stages from the “Not Done”, “In Progress”, “Ready for Test” & “Done” statuses to a Kanban style “Ready for Engineering (3)”, “In Progress (2)”, “Ready for Test (3)”, “In Test (2)” & “Done” which includes limiting work items on the task board; the numbers after each heading indicate the limit imposed on the amount of items that can be in each column based on the capacity.

Now let’s now address those problem areas:

Documenting & Quality

Instead of operating under scrum methods, story pointing stories or bugs up front and assigning enough to fit into our two week cycle, feature requests and bugs should be assessed as soon as they are raised, prioritized by business value, then if deemed valuable enough, story pointed and investigated fully to define acceptance specifications (include scenarios).  ONLY then can that piece of work be put into a “Ready for Engineering” state.  Sticking to the processes described above, only a limited number of features would be under development.  The controlled flow, or focused management of that feature through development would encourage greater scrutiny and the enforcement of the rule to add tasks (SBIs), helping to document by providing a historical record of the engineering work being done.

Product, Engineer and QA exposure to applications

“A limited amount of features under development”.  I’m starting to like this.  Each feature has a business priority attached to it that will determine when work gets done.  If you were to look at the domain work currently remaining in our Ops Rota system and at the nature of the issues raised from the different domains, I’m sure you’d agree that by switching the focus from only working on one domain during a two week rota, to working on as many different features as the capacity allows, it would mean everyone involved would be exposed to multiple domains very quickly.

Resource Availability

Again, by limiting the amount of work under development, which would also apply to how much is allowed to be tested, bottlenecks can be controlled.  If they did occur there is much greater visibility on what resource would be needed to get something to the done state to release the bottleneck.

Releasing

Kanban says an In Progress “place” should always be available for blockers / emergencies which is fine too.  Branches should be made when releasing versions from now on, so that emergencies can be managed.  Feature or bug work should be engineered on the trunk and then the new release branched when the feature or bug is set to done.  Emergency fixes get engineered on the branch and merged back down onto the trunk.  If a bottleneck forms, there will only be a small amount of features causing the bottleneck therefore resource can be assigned to clear it without having a significant affect on other projects in development.  Features or bugs should be speedily managed through the system and confidently spot/hot fix/minor released into production; the high release rate should force the creation of build scripts to automate the builds.

It seems to me adopting this Lean approach would provide a controlled, feature driven environment for the Ops Rota.  I would expect a rhythm to develop for creating stories and defining feature or bug acceptance specifications, the high release rate would provide great domain exposure for Engineers and QA.  With everyone focused on one feature it should promote the Engineer’s adherence to the agile design principles laid out by Robert C. Martin and Mica Martin.  There is also a solution to our problem of bad quality legacy code suggested in the point about coding to interfaces on the Improving Your TDD Skills session notes from CITCON Paris 2009.  The “pull” nature of the development stream combined with the work item limits means that big backlogs cannot build up.

If this worked I could see it really benefiting our clients by providing them a constant stream of value.  A stream that could be switched off and back on with great ease during periods of resource constraint.

I haven’t mentioned the Kanban cards yet.  To be honest, I haven’t given them as much thought as I have the management of work items.  I could be in danger of becoming a Kanbut but I do not place as much importance in what colour the cards are at the moment as I do in solving the Ops Rota’s problems.  We are trialing a digital task board at the moment and we haven’t investigated the Kanban templates yet.  I also haven’t mentioned automated acceptance testing.  We are not set up in our organization for this yet so that’s also a topic for another day.  One step at a time.

I’ve convinced myself we should convert the current format of the Ops Rota into what I want to name our “Lean Stream”.  Theoretically this would make the job of managing it more effective, or perhaps, as it should, become self managing.  I also reckon everyone else involved will enjoy working on it even more and gain a greater sense of achievement.  I can also see a similar, “sister” stream operating for our flagship product, to cater for bugs and urgent feature requests, enabling the main releases (“curtain lifts” as Nik Silver called them) to just focus on version objectives.  We could call it “Lean Jeanie”!  OK stop now.