Quality Gates: Why all the fuss?

Let me start this blog post with a question: Is a gate designed to keep things in or let something out? Depending on your mood, you may have different answers to this question which may not be limited to the options I have provided. However, if I am to be diplomatic and keep my emotions out, I would say a gate is designed to control what is allowed to pass through its threshold. For this discussion, what we are looking at controlling, is quality.

Moreover, it is essential to note that a quality gate is only as strong as the will of the gatekeeper.

Each stage in the software development lifecycle provides us with several opportunities to implement quality gates. At each quality gate, the gatekeeper could be a person, personnel, or a tool. The main goal of the gatekeeper would be to regulate what goes out and what is restricted. This is easier with tools like Sonar, which provides the facility to configure the quality gates. Such tools can monitor the health of the codebase based on several criteria and indicate where quality aspects such as maintainability and dependability are lacking. The situation gets a bit sticky when people are involved, where added timeline pressures will test the will of the gatekeeper. As I have mentioned earlier, a quality gate is only as good as the will of the gatekeeper. It is a symphony of integrity, accountability, bravery, and negotiation skills of the gatekeeper that provide high-quality products and components to the next set of stakeholders.

Now that we’ve discussed the gatekeeper’s mindset (or ruleset), let’s evaluate which quality gates we should use and when we should use them.

Why, What, When, Who, and How. These are the questions you should ask yourself, your audience, and the stakeholders to understand the purpose behind introducing quality gates. If we are to search online to find out the best quality gates to use or which are the most attractive quality gates used in the industry, we will end up with a long list of promising options. However, a report or a dashboard should never be ‘everything-but-the-kitchen-sink’—this approach will most definitely confuse, distract and overwhelm your audience. You need to avoid falling into this trap.

Given below is a simple depiction of how your thought process could best work to achieve a carefully curated yet effective report/dashboard containing quality gates.

Why

What

When

Who

How

How effective is the time spent on requirement gathering?

To check the quality of my requirement specifications

Before the subsequent team starts work on the requirements
(E.g., Development Team)

By: Development & Testing Teams
To: Requirement & Management Teams

1. Requirement Clarity Index
2. Q&A document summary and progress

To check the effectiveness of work done by the development team.

1. To see how good the development codebase is
2. The amount of defective code introduced?
3. The amount of rework required?

Before handing it over to the testing team

By: Development & testing teams
To: Testing & management teams

1. Code quality measurement tools (E.g., Sonar)
2. Code reviews
3. Development unit test pass rate
4. First-time test pass rate
5. Number and criticality of defects identified
6. Defect removal efficiency
7. Defect Aging
8. Defect severity index
9. Defect density

To check the effectiveness of work done by the testing team.

1. To see whether test case coverage has been met
2. To see if there is no risk to meeting project milestones
3. Number of invalid defects recorded

Before handing it over to the client

By: Testing Team

To: Technical, Management & Client Teams

1. Test case reviews
2. Test execution progress
3. Defect to remark ratio
4. Test coverage verification
5. Test case walkthroughs

Now that we have collected all the pieces of the puzzle, it’s time to put them together to create the picture(s). The most exciting part of this stage is creating multiple visuals for different audiences using the same set of quality gates. The mastery is in the selection, benchmarking, representation, and the placement of each KPI related to each quality gate. Also, there is no foolproof way of getting this right the first time. Perfection is a perception that is subjective to the audience.

Below are few examples of quality gates for further clarification:

When

Quality Gate

Benchmark

Accepting a requirement for development

Requirement clarity index

Q&A status

The confidence level should be 4 or more
(in a scale of 1-5, 5 being fully understandable)

Over 90% of the questions should be answered

Accepting a development release for testing

Code review

Development unit test pass rate

Smoke test case pass rate

Completed

Greater than or equal to 80%

Equal to 80%

Completing testing of a release

Test execution status

Number of defects

Defect severity Index

Completed

No Critical/Major/Minor defects

Less than or equal to 1.9
(industrial standard)