CAST 2015 has come to a close; I always have mixed feelings after each one of these conferences:

  1. Glad they are over because I am absolutely exhausted, brain dead, and looking forward to getting back to my family
  2. I WANT MOAR! The amount of awesome experiences, networking, and knowledge at these events always astound me. Every year I come in wondering what new thing I could possible learn about and every year I leave staggering with the amount of stuff I have to think about
  3. How do I keep in contact with everyone and keep the momentum going?
  4. Now what? How do I take all this awesome new stuff I learned about, distill it down to what I can actually take and use at a 20,000 person company, and then start using it?

So what now? Where do I start? Seems to me the best place to start is list out three things that really resonated with me.

  1. The CDT community as a whole seems to be embracing the inevitability that test & programming are slowly becoming one. This doesn’t mean that there won’t be testers or that testing won’t be done, it’s just that the expectation of what a “developer” is is changing. I feel like there is a whole blog post on this one so I will probably expand on it more later but it felt like an unintentional theme of CAST this year was “The Tester role as we know it may not exist in the future, how do we not only survive but thrive during and after this metamorphosis?”
  2. Put your Exploratory Testing charter in the backlog stack so they are pulled off in order just like normal stories. I’ve been struggling to figure out how to get exploratory testing become “just something we do” and I really like the way Pivotal does it.
  3. I want to do more experiential training, both participate in and design/lead. Between the experience I had in the “From Velcro to Velocity” workshop and the number or discussions I had with people at the conference I am really excited about the possibilities with experiential learning!

I have my three takeaways, now I need to figure out how to take action on each of them.

  1. I want to submit proposals and speak at at least 2 developer conferences in the next 12 months. As the testing and programming roles converge it’s important that we start broadening the reach of our message, instead of just “preaching to the choir.” I already have my first proposal title ready to go “So you got rid of testers, now what?”
  2. I’ll be working with my local team to do a trial run of something like this and see where it goes. I would love to see success with individual teams and be able to start expanding it across the company.
  3. I am going to immediately start working with our training department to come up with some new experiential learning classes. The first thing I’m going to do is a trial run of Robert Sabourin’s “From Velcro to Velocity” training for TDD. I also want to try some experiential training for exploratory testing but I don’t quite know where I’m going to start with that yet.

When you look at the bigger picture, I go to these types of conferences to better myself but also to find ideas that can be applied to my day job. Due to the fact that I work at a larger company that isn’t always easy; in fact I pretty miserably to apply anything I learned at CAST 2014 to my day job. I am not going to repeat that same mistake this year. In addition, I really hope to spread some of the stuff I heard using the local Bay Area Software Testers (BAST) meetup group. I’ve really let that stagnate and it’s time to start engaging the 130+ people in that group.

At the end of the day, if I’m not spreading things I learn to other people I’m only being fractionally as impactful and effective as I could be. I want to be a multiplier. I will be a multiplier.

When you are writing product code there is generally a goal to avoid code duplication as much as possible and abstract code out into different routines/functions. When writing unit/functional checks you should focus on “how quick can I tell exactly what this check is trying to do” rather than focusing on reducing code reuse.

The first way to do that is make sure your check method is named appropriately. For example, which of these method names would you prefer to run into while walking through check code:

  1. check1()
  2. checkNullInputReturnsInvalidInputParameter()

The second tells us exactly what this check is trying to do whereas the first tells you absolutely nothing. Use descriptive method names! The next step to ensure readable checks is to organize each check into three areas.

Reuse code all you want, abstract all you want here but make sure that you comment the hell out of what’s going on so we can tell at a glance

Reuse and abstraction is still acceptable here but I try to limit how much I do.

Keep everything 100% in test for this step. If you want to verify content then have a verifyContent() method but keep the actual content trying to be verified in the test itself. Don’t worry if you are duplicating code, in this context duplicated code is better than abstracted code.

In general I should be able to look at any check and know almost immediately what it is checking, what setup steps are being done, what execution is occurring, and exactly what is being asserted. If I can’t figure that out within a few seconds to a minute then the check is either trying to do too much or not readable enough

Last night was the last straw for me; I walked out of Hobby Lobby, leaving my purchases at the register. This wasn’t a protest over the religious beliefs of the founder or around their decision to not cover certain birth control options. This was a protest over the fact that Hobby Lobby is a poorly run business with deceptive practices.

The first time I stepped into a Hobby Lobby I was overwhelmed, this made Michaels and Joann’s look like the local mom & pop craft store. I walked around and had to prevent myself from buying half the store. I took my cart to the front, eager to check out and bring home my new finds. That’s when I noticed the cashier had to find the price tag on each time, manually enter the price into the computer, then look up in a flyer to determine if the item is on sale and apply any applicable discounts. She had to do this because Hobby Lobby doesn’t actually enter their products and/or sales into the POS. They make the cashiers, who are human, try to remember all the different sales at any given point.

Speaking of sales, Hobby Lobby sure does like to use them. Some of their products are on sale all the time, others seem to come and go at a moments notice. For example, furniture is always 30% off the marked price, poster frames are always 50% off the marked price, and often times picture frames (yes, they differentiate between the two) are 50% off the marked price. For the items that are “always” on sale, the signs say something along the lines of “Furniture is always 30% off the marked price, which is what you would pay for equivalent pieces from the competition.” So instead of just marking the prices lower, they mark them at their “competitors rate” and then have signs advertising “Always x% off!”

When they have actual sales, they use the same sign format, the same font; they simply exclude the word “always” from the sign. Unlike most retail stores they will run multiple concurrent sales with different end dates. For example, their wood décor sale may run for the entire month of April but the 50% off frames sale was only for a few days. Remember earlier when I said that none of this is in the POS and the cashiers have to lookup and make sure to apply all discounts correctly? That’s a remarkably hard thing to do when things are constantly changing.

So what made me finally decide to stop shopping there? It’s a combination of everything about how they run their advertisements and checkout system.

  • The signs are ridiculously confusing. As already mentioned, there is very little difference between the “always x% off” and “sale” signs; they simply remove the word “always.” The signs themselves don’t list any exclusions to any sales; “50% off all wall frames” won’t include things like shadowboxes even though they are in the same aisle and are technically wall frames.
    • Another one I noticed the day I walked out: A sign stating “50% off glassware and accessories.” The sign was on an end cap with mason jars and some kits to turn mason jars into lights. Even though the sign was right there, the light kits were not part of the sale.
  • It’s hard to know what’s a sale and what’s an “always” discount. This bit me when I was buying some frames on a Saturday, returned one of them on Tuesday and tried to buy 2 more. I swear the sign I saw had said that wall frames were always 50% off but the cashier didn’t apply the discount, the signs were gone and the manager told me it was a temporary sale only.
  • Cashiers are human and expecting them to remember 100% of the sales going on at any given point, and forcing them to manually enter prices is inefficient and very error prone.

Combine everything talked about above and you get what amounts to a confusing process where customers don’t know the actual price of an item and have to rely on a human cashier to remember all the sales prices. As a customer you have to watch the cashier like a hawk to make sure they apply the right discounts and then be frustrated when it turns out the sale didn’t apply to your specific item when the signs led you to believe otherwise.

I stopped shopping at Hobby Lobby because they either don’t know how to run a retail store or are intentionally trying to deceive their customers. I don’t like supporting incompetence or deceitfulness, so regardless of the reason Hobby Lobby will not getting any more money from me.

DevOps. It’s a word that some people respect, some people hate and only some people truly understand. It’s been a big buzzword in the industry for a few years now and, like any buzzword, it’s gotten a lot of negative attention. However, for all of the negative attention it’s gotten it has also shone the spotlight on several areas that the technology sector hasn’t been very good at. I’m not going to get into details about all of those areas in this post, but I did want to highlight one: collaboration.

Collaboration is so critical in the technology world (or just the world in general) and yet we almost universally suck at it. Hardware designers have to work with hardware builders who have to work with software designers and programmers. The programmers need to work with testers, business analysts, program managers, UI and UX, and a host of other people. In a well oiled machine these interactions are natural and happen without even thinking about it. In the common scenario these interactions don’t happen until a fire drill is already underway and making any changes is incredibly expensive, and not just from a money perspective. Reputation, happiness/morale, time and often quality are also impacted by fire drills.

There is one subset of the disciplines that has made tremendous improvement in how they collaborate over the past few years: Developers and testers/quality engineers. Again, this isn’t universally true but in my observations it’s becoming a lot more common to have good to great collaboration between dev and test than it is to have bad collaboration. That transformation hasn’t been easy and details on how it was accomplished vary widely. Rather than try to explain how I’m going to talk about the what the change looks like and some of the benefits.

Let’s go back in time a few years. A standard development process looked something like this:


Between each step there is this white space that you can think of as a wall. Work would flow in a serial manner from silo to silo and be thrown over the wall to the next group. There was very little collaboration between the groups and almost none of it was in advance of work being thrown over the wall.

Enter “The Agile Methodology(tm)”


Agile methodology attempted to break down the walls between groups, or at the very least shorten the amount of time that it would take to go through a cycle. Unfortunately in many cases the walls between developers and testers still existed. Developers would work on something for a sprint and then throw it over the wall to testing to test it. It didn’t promote collaboration and certainly didn’t promote a quality mindset.

Finally, something ended being the last straw (disclaimer: I have no idea what it was, but in this type of scenario there is almost always something that finally makes somebody step back and say “how can we do this better?”) and the wall came tumbling down.


Okay, the wall is down; what exactly does that mean? How do dev and test collaborate now that’s so different than it used to be?

  • Tester/QA staff are involved in the process from the beginning. They participate in spec reviews, ask questions about the business cases, etc. Testers often times have valuable insight into requirements and usability that are great to have at the beginning of a project
  • Testers can be involved in code reviews, looking for things like testability, test coverage and missed use cases.
  • Testers and developers work together throughout the development process, developers ask testers to review test cases and testers provide a test plan that can be used by developers to double check assumptions and review risks.
  • Pair testing. If developers can pair program why can’t testers and developers pair test. Having testers and developers sit down and do a testing session together is beneficial to both parties: testers can learn about the system as the developer sees it and developers get exposed to use cases and testing methods they may not have thought of.
  • Feedback loops are drastically reduced. Because potential defects are found while code is actively being written the developer still has the full context at the front of his mind and bugs can be fixed before going through a full build and release cycle.
  • And more…

From my personal experience, groups that have good collaboration between development and testing tend to produce higher quality products, work more efficiently and have a higher morale. The value provided to the business in those cases is massive, imagine if this kind of collaboration were applied to all disciplines; that’s the promise of DevOps.

Building quality in

Posted: December 13, 2014 in Uncategorized

“Build quality in” is a term that is becoming more popular in the software world lately so I thought I’d take a stab at explaining it a bit. First, it’s important to define what quality actually means. This is something I indirectly tackled in a previous post while talking about measuring quality. The definition I like to use is:

A system can be considered quality when it meets the needs of the user, is intuitive and provides a better experience than the previous solution.

That definition leads to having quite a large spectrum of what quality is. Some people may very well be happy with a slow, buggy and barely usable product as it meets their needs. Those are most likely the edge cases so when I’m talking about building quality in it’s with the assumption that the user would like the product/feature to be a stable, relatively well designed piece of software.

It’s been proven many times that it is easier to do something right the first time than trying to fix it after the fact. “Building quality in” is just as fancy way of saying “do it right the first time.” What are some of the things we can do to help ensure it’s done right the first time?

1. Design for testability
When I say design for testability I am referencing automated testing (or checking, whichever you’d prefer.) Designing code to be testable has several books written about it, but the high level is you want code that allows your tests meets the following criteria:

  • Repeatable. Tests should be able to run multiple times with the same conditions and return the same result every time
  • Easy to write. Tests should be easy to write and maintain
  • Fast. Unit tests should run in a few seconds. Component/functional should run in a few minutes. This allows for quick feedback and promotes frequent execution
  • Easy to understand. Tests shouldn’t be complicated or difficult to understand. I should be able to see what a test is doing in just a glance.

Designing your code for testability allows frequent and fast feedback loops which helps you isolate defects during development and in any future work done to that code.

2. Write unit tests (and frequently run them)
Whether you decide to follow TDD or not, you still should be writing (and frequently running) lots and lot of unit tests. Unit tests are a great way to  ensure that your code is doing what you intended it to do and getting quick feedback about any changes you’ve made. 

3. Engage other groups early and often (testers, ux, product management, etc)
Don’t write code on an island. The days of waterfall are largely behind us, specs aren’t frozen and signed off before any work is done. Development is done iteratively and should be getting fast feedback on each of the iterations (have you noticed a trend around fast feedback?) User interface designers will be providing new mockups that should be incorporated, user experience groups will be providing suggestions based on focus groups, testers will be reporting defects, product management needs to provide input around scope changes, etc.

Building quality in requires collaboration across many groups, not just the technical teams.

4. Non-functional testing
Ensure that non-functional testing is considered a first class citizen and used in the evaluation of doneness. This means UI, UX, Performance, Security, Compatibility, Accessibility and the list goes on. Functionality is a large part of quality, but it is still only a part; it takes many parts to represent a quality product.

5. Have a well defined and supported CI/Test/Reporting system in place from the beginning
Making Continuous Integration/Testing/Reporting infrastructure a first class citizen not only encourages test development but provides quick feedback, prevents merge hell and provides visibility into certain quality metrics (keeping in mind the details talked about in the Measure Quality post).

6. Utilize production traffic and data
One of the most powerful testing tools available to a company is production testing. When I say production testing I don’t mean “let the users test it!” I mean things like A/B and Shadow testing. Both of these methods allow the use of real production traffic to get feedback about a change, but they do it in different ways.

A/B Testing
A/B testing is the concept of taking a small percentage of users/sessions and directing them to an alternative experience. A is your control group (a majority percentage of users) and B is your treatment group. By allowing some users to use new code you are able to measure and analyze all sorts of data and use it to make decisions. For example, you can test different checkout workflows to see if your conversion rate is affected positively or negatively or you can use it to try out new site designs and see what sort of impact it has.

In most cases your users will be exposed to changes when doing A/B testing

Shadow Testing
Shadow testing happens when a separate, non customer facing instance of your code/infrastructure is running in production and you send a subset of requests/traffic to both the current production code and your shadow box. Just like in A/B testing this allows for collection of data to make decisions. Unlike in A/B testing the customer is never exposed to these changes and you cannot release new user experiences as part of a shadow test. The most common use cases for shadow testing are testing changes that have potential performance impacts and swapping out underlying systems without changing the user experience.

There are other things that can be done as well but I just wanted to get a list started. Let me know in the comments if you have any other ideas (or just general thoughts about the subject!)

Measuring Quality

Posted: December 5, 2014 in Uncategorized

One of the things I’ve struggled with my entire career is how to measure quality in an effective way. I’ve seen several different attempts at measuring quality but they all eventually led back to the same issue: they were collecting data that didn’t provide any actual insight into the quality of the project. What data was being collected? Things like:

  • Lines of code
  • Number of defects found (both at a project level on by an individual person)
  • Number of test cases
  • Percent of test cases automated
  • Code coverage
  • Cyclomatic complexity

I’ve seen these data points used in two different ways:

  1. attempting to measure quality through a formulaic approach, once you reach a certain threshold in the formula your system can be considered quality.
  2. to track trends and correlate with other data.

The second one is a much better use of the data but it still seemed flawed to me so I wanted to try something different. I stepped back and asked myself this question: What does it mean to have a quality system?

The answer I came up with was potentially surprising, but simple:

A system can be considered quality when it meets the needs of the user, is intuitive and provides a better experience than the previous solution.

Notice what isn’t in the definition. There is no mention of bug free (ha!), no mention of performance or reliability and nothing about the percentage of the system that has automated tests. Now look at what it does talk about:

Meets the needs of the user
Why is software written? To meet the needs or desires of some sort of user. That user may be a paying customer, may be an internal organization or may even be another system that depends on your system. If a system goes to production with zero defects (ha!), is lightning fast, and can withstand a nuclear bomb but fails to meet the basic needs of the user then that system simply cannot be considered quality.

I battled with this one a bit. Does a system really have to be intuitive to be considered quality? In the end I decided to leave it in there for this one reason: if a system is so complicated that it takes a 100 page manual and 6 months of training to be able to use, was it really quality? Could things have been done differently to make the system easier to understand and use?

Provides a better experience than the previous solution
Aside from simply meeting the needs of the user, the system should also strive to improve upon the previous experience. Maybe that means cutting steps out of a workflow, providing an automated solution from something that used to be manual, or an entirely different experience than existed before (e.g. television)

If we think about quality in the way I’ve described above, what are we really trying to measure? Does the service meet the needs of the user, is it intuitive and does it a provide a better experience than the previous solution. What are some metrics that help us answer those questions?

  • User satisfaction surveys. Who better to tell you if the user has their needs met than the actual user! Send out periodic user surveys and track the trends over time. Are you getting better about meeting the needs of your user or are you getting worse?
  • Defect Leakage. How many defects is the user filing. If a user takes the time to file a defect that’s an indication that the defect is impacting the service’s ability to meet their needs. Again, track this over time, the raw numbers don’t matter as much as the trend.
  • Performance data. I know I said that a system doesn’t have to be performant to meet a users needs, but in many cases there are performance considerations. If the user has certain performance requirements or expectations you should be monitoring the performance of the system over time, once again keeping an eye on the trends.
  • Error rates. Keep track of the error rates from the system’s logs. Correlate it with usage patterns. High error rates likely mean the user is experiencing a problem and not having their needs met.

Now look back at the metrics mentioned at the beginning of this post. Do any of those metrics help you answer the questions needed to measure quality based on the definition I provided? I don’t think they do. That’s not to say there isn’t a place for those metrics (spoiler: there is) but I would argue that they are not valid measures of quality.

There has been a good amount of chatter dealing with software testing standardization, specifically ISO 29119, over the past few weeks, most of it bad. For those who don’t know, ISO 29119 is an attempt to provide a single standard to be used in all software testing. In their own words:

ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation.

Okay, great, what’s wrong with standards? In many cases, nothing. Standards are great for situations and scenarios that don’t require human ingenuity or creativity. Software testing is not one of those things. James Bach has defined testing as a performance and I tend to agree with him. The act of testing is not something that can be constrained, restricted or defined by a single set of requirements. Testing is exploratory, creative and never the same.  Do we try to standardize the way actors perform or prep for a live show? No, so why are we trying to standardize software testing?

I believe ISO 29119 is going to cause the opposite effect of what the creators were hoping to cause; by putting a standard on software testing all innovation in the craft is going to be stifled. Creative thinking and exploratory testing are going to become much less common, if not completely forbidden. Every single tester will be doing the exact same things in the exact same way meaning that there will be less opportunity for information gathering. Testers will be relying on information provided to them instead of providing valuable information.

The net effect of this will be lower quality software, a worse experience for customers, and potentially, in extreme cases, physical injury. The world runs on software in ways that many people don’t consider: cars, train, planes, infrastructure grids, defense systems, etc. The world is controlled by software so why handicap the people who test it?

If you are opposed to Software Testing standardization I urge you to sign the Stop 29119 petition. The petition takes the stance that there has not been a consensus reached and therefore the standard should have never been formalized (There are over twice as many signatories on the petition than there were in the working group who worked on the petition.) 

In addition to signing the petition I would suggestion you read and sign The Professional Testers Manifesto. The manifesto is a great alternative to standardization and is something I am a firm believer in.

There are many other issues I have with ISO 29119 but those have been covered by other bloggers and extensively on twitter so I won’t rehash them. Take a look at the #stop29119 hashtag and/or follow James Christie, Rob Lambert, and