Automating Chaos Just Gives You Faster Chaos

Automation can be an awesome productivity booster within testing teams and a quality enhancer for your systems when done correctly or else can be a disaster when done incorrectly. The key is to make sure that it is used the right way, which is the hardest part when starting. Every organization defines a set of measurable benchmarks of quality for their product and it’s their job to make sure that it never falls below them. Those benchmarks of quality can be related to many aspects of the product like usability, performance, functionality, integration, etc.

Why Automate Tests? Tell me the Benefits

Test automation is a program using a tool or language or set of tools or languages that allows the organization to measure the quality of the product in a consistent and timely manner.
Continuity — Running tests 24/7.
Reusability — test scripts are reusable.
Simultaneity — test in multiple environments simultaneously.
Reliability — fewer chances of human error.
Volume — in a given time a big chunk of cases can be tested.
Increase — in throughput or productivity.
Improvement — in the predictability of quality.
Improvement — in the robustness of processes or products.
Increase — in the consistency of output.
Reduction — in direct human labor costs and expenses.

Automation Challenges and their Solutions

Automated tests not only empower the teams to easily prepare test data, execute complex test cases across multiple platforms rapidly and accurately, and catch defects early on in the development cycle but also benefit the organizations in multiple ways; when used correctly, such solutions can boost a test team’s productivity, enhance software quality and accelerate time to market, meanwhile there are several challenges involved too.

A visual depiction of how a software engineer feels when encountering a flaky test. Photo by Jamie Haughton on Unsplash

Ideally, the automated tests should always have the same result when the tested code doesn’t change. A test that fails randomly is commonly called a flaky test. When tests behave as expected, they’re a boon to productivity. But as soon as tests can’t find real problems, their results arrive too slowly or can’t be trusted, it turns into a nightmare. As your codebase, test suites, and dependencies grow, so will the number of flaky tests. You should always try to check on the selectors you use for automation as they are one of the main reasons for flaky tests apart from sync issues in the test scripts.

Try to get a green build and the confidence back by applying the above approach. Also, you can break the tests into categories according to the functionality or by nature of the tests, for example-

  • Smoke test — the tests needed to ensure that critical functionalities(i.e business & technical behaviors) are intact.
  • Regression tests — the tests which cover all workflows that have been developed to date.
  • Functional tests — the tests which cover the newly developed functionalities from the current release cycle.

Unit Tests form the base of the testing pyramid because they run faster than tests at other layers. Push as many tests as possible (without compromising accuracy) to the unit test level. Identifying and pushing tests to a lower level makes tests faster and reliable. Join hands with developers, review the unit tests and add more to the unit levels. By reviews, you can avoid test duplication in multiple layers. Add integration test, if applicable. Ensure only high-level workflows are covered in end-to-end/UI tests.

Create test data smartly — a good automation test is the one that creates and cleans up the needed data after use.

It doesn’t matter how great your tools or resources are — you should always remember that testing can’t be carried out in a vacuum. It serves an objective: improving your product. We need to ensure that the testing/development team, the management, and other stakeholders all agree and understand the desired results of the test automation exercise.

The application under test or your Product under testing undergoes constant changes, and the test scripts have to be adjusted accordingly. This primarily increases the time spent on the creation and maintenance of test automation scripts. As a result, it becomes difficult to calculate ROI while continuously adding/updating/performing the tests.

The initial phase of test automation is usually expensive. It’s necessary to analyze, design, and build a test automation framework, libraries or reusable functions, etc. In some cases, it is required to take into account licensing costs, facilitating and operating costs such as hardware and software costs. Moreover, even though we can use free open-source tools to reduce the licensing costs, we might spend significant efforts on learning, training, and maintaining them. Furthermore, we also take hidden costs (such as meeting, communicating, and collaborating) into consideration.

Successful automation requires the right tools and skilled teams, but to have this winning combination might be a little problematic as:

  • team members lack the expertise to make the most of a specific tool;
  • team members don’t know whether the tool they need exists;
  • no one tool has all the desired features;
  • tools of their choice don’t offer 100% test coverage;
  • the cost of tools exceeds their test budget.

Most of the tools provide a trial license, try it before finalizing. Perform an adequate evaluation of the advantages and disadvantages of using the tool before finalizing the tool for automation. If your team doesn’t know how to use a tool, you can organize a training session with its creator, buy an online course for your testers, or hire a consultant who can help your team master it. Reach out to the testing community if you’re still looking for the right automation tool. Also, you can try searching for multi-tool solutions that cover areas critical to your team. If the tool you found is way out of your budget, just prepare a cost vs. benefit analysis and analyze the expenses coming from bugs that would have been solved if you had the right tool in place is bound to work in your favor.

Don’t just automate everything blindly, it may seem tempting at face value- and it’s easy to get carried away with your shiny new automation tool. But trust me automating everything won’t help. You’ll end up wasting time and money automating processes that are either not needed, or the ones that are better handled by humans.
Plus, attempting to automate everything risks losing the human touch and friendly face of your company. When customers can’t get support from a human, and the automation in place isn’t equipped to help them, they might inevitably feel alienated or devalued.

When all you have is a hammer, everything looks like a nail, isn’t it?
What do you think?

If you like what you read, please give a clap and/or share your thoughts in the comments section below as I learn just as much from you as you do from me.


Quality Enthusiast!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store