Tips for Success in Creating an Test Automation Solution

It is not difficult to imagine the benefits of having automated tests that can help the application testing process because it will make the development process feedback much faster, increase the effectiveness, efficiency and coverage of the test, from frequent and repeated test executions. , that’s a bit of the benefits of automated testing. Over time, existing features on the system will continue to grow, QA will conduct testing starting from the system / features that already exist plus the new system being developed.

An automation testing (test automation) is able to make the computer system run a series of activities that have been designed to compare the results of the system whether in accordance with the desired standards (expected behavior) so as to produce a test report that the system has succeeded or failed.

With this test automation it is expected that QA will be lighter in carrying out the same regression test repeatedly, of course this will save QA time so that it can focus on other things that have a large impact in maintaining good software quality, because that is its role in software development very important and can help achieve the success of the project

If the automatic testing has been made correctly, it should be very beneficial for the development team as a whole, I will try to share tips on making a good and useful test automation, then what things should be avoided when creating this automated test set, because we must be realistic, not all can be in computer automation, there are limits that must remain in mind

Manual vs Automated or Testing vs Validation

Please avoid comparing these two things, between manual and automated testing, because both have different functions and need each other. Some say Testing = Validation + Thinking, so it should be called automated testing instead of being called automated testing / test automation, because it is a fact-based test, but for me it’s more common for other people to say “testing”.

During manual testing, the feeling and reasoning of a tester will be very necessary in finding weak points and potential problems that will arise from the system, and often this process is not written on the test plan (exploratory testing) because the tester adapts to the results of the testing process so need to be improvised in “following” the application.

On the other hand automated testing are human works that also contain instructions so that the computer can do a task (in this case testing the results), well every time this automated test is run, it will do the exact same activity as the instructions written in the code, and it only checks things that are requested to be checked.

Start with the automated regression test

The main purpose of why test automation exists, is because we are tired of running the same test over and over with each new release, Regression testing need to be run in tandem with an evolving system, we need to ensure that after adding new features to the system it will not damage other features that already available.

This series of tests is very time consuming and tedious work for QA to run manually every time, this is a sign of automation needed to maintain consistency of test results, so regression tests are very suitable to start in automation

Design the test before you start coding

A good process is to create user scenarios, test cases, what needs to be expected before starting to automate, because this can be very helpful in finding the risk of problems with the system, then discuss with the team as well

It is not recommended when directly creating the code automation to test positive scenarios and happy flow, rather than thinking about and identifying which scenarios that might occur and can be tested.

Do not also automate things that are actually not very important for the computer to do, like for example we will test the login page, the definition of the person has been successfully logged in does not need to confirm the URL! = “/ Login” but more important to ensure, for example the username in the corner is appropriate with the login credential earlier, we need help from other team members to review our test design if it is sufficient and good in the test scenario and its verification (expectations)

Remove doubts

One thing that becomes the main point of automation is being able to maintain consistent results, so we can know when the test fail means something is wrong and a system anomaly appears

Imagine when the test pass is then run again to become a file, without any changes to the application being tested (SUT), we are never sure when a file is caused by an automation code error or an error in the system.

When there are errors, we must check the causes that cause this error, if there are many inconsistencies found or false positives will make this error checking process long and difficult. Don’t be afraid to delete such unstable tests in our test set, because what we need is consistent and trustworthy results.

It should be noted, the quality of test automation is very dependent on the quality of the person who wrote it, it could be that the test went well, all checks passed, but there is a possibility that errors might arise, but the computer does not care (because it is outside the set expectations) so that it can give a false quality impression.

Do not automate features that aren’t stable yet

As a new feature that is being developed, there are many things that are not smooth, often even so agile it turns out that the feature is no longer relevant and cancelled because the business has changed from the original plan, so the automation becomes useless

When the test automation process has started together with the features started, the test will often be changed to adjust the implementation of the moving features too, and this is a waste of time, although there are also ISQA friends from other start-ups who have succeeded in making automation concurrent test with the features created.

However, I am more inclined to think that the implementation of automation (especially UI) will be better made when the feature is stable and has not changed much

Do not expect miracles from automation

Never expect automation to be able to catch a lot of bugs, in fact with manual testing the number of bugs found is more tablets.

The purpose of making test automation is to help QA in the testing process so that more QA free time and can be used to think about process improvement and run difficult tests and edge cases so that it can give confidence that the application is still running smoothly with the deployment of new features

A series of regression test scenarios that are run automatically will certainly provide faster feedback and can provide a sense of security for the team to deploy to production

Understand the context

The test can be in various layers, from unit tests, APIs, services, and UI, each layer has its own differences.

Unit tests will ensure that the code at the class level runs as it should, according to the expected logic, prioritizing verification rather than validation.

The API test or integration test will ensure that the unit functions and classes can run as expected when combined between classes

UI or end to end test is a series of user travel interactions, here you can say the UI level is more validation than verification in testing user scenarios, tests that run at this layer tend to be slow and error-prone, there can be improvements in appearance, but in usability still the same and does not change so that it can give a UI test arguably an error because it turns out there are elements that change (css class instead of making elements not found).

By understanding the context of each test layer we will be able to set the automation strategy, it can be that for scenarios A, B, C we will test through the API test, then only scenario B is also run on the UI test to make it faster and more stable

Do not bother 100% cover everything

100% test coverage is very difficult to achieve because the combination of possibilities that arise is very diverse (possible permutation), when talking about test automation, adding more tests does not always mean increasing confidence in the test the better. It all depends on how good the test automation is designed.

To make automated scripts require resources and time, when we want to automate everything, it certainly requires a lot of resources and time, which in general cases cannot be fulfilled, even more automated tests will increasingly require more intense code maintenance

Therefore, we better make it based on risk analysis, in order to be able to choose which tests are suitable for automation so that they can greatly benefit in the testing process and in terms of business process scenarios.

Oh yes, please note that not all tests can be automated, there are often cases where the tests are originally very complex, often the results are inconsistent, so this case we better test manually by QA, because we better focus on the most important features / funcionality and crucial for businesses to be automated rather than pursuing full coverage.

Do not intend to replace the manual tester

There are people who think that having an automated test is sufficient and there is no need for manual QA anymore, even though it is clear to make automation, we must understand what results are desired, what needs to be checked so that it is said to be a valid outcome. The testing process requires multi-disciplinary abilities such as investigation, detective, psychological so that we can plan trials as well as direct execution, a good manual tester will always have the ability to think critically to prevent errors from happening again for the user

Don’t lose confidence

This is the most important thing, we must anticipate the side effects of test automation, when we have devoted our time and energy to making a perfect and cool automatic test set, but it turns out to be useless because it is not useful and helpful to the team.

It could be that the team did not feel the benefits, or even did not know, so the visibility of the test automation solution series that we made also needed, how they were used, and then test what was in it so as not to do duplicate regression.

If the automated test is often flaky (intermittent error), slow and not transparent, how can it give an indication of the quality of the system while the automated test alone is not good and stable quality.

This series of automated tests needs to be treated like a living thing, it continues to grow over time, there are times when certain tests are no longer relevant, don’t be afraid to remove tests that often fail or do not provide consistent results in order to maintain the credibility of our test automation solution as an indicator from application / system quality.

By understanding the above limitations, in my opinion the role of humans still plays an important role in the testing process cannot be completely replaced by computer automation, so why do we need to automate application testing? the reason for the short is “repeatability”, more collaboration with the help of computers to run automated testing will help QA to be more free and can be more focused to do exploratory testing to more multiply the risks, and defects.

So, the next time you plan to automate a test, try to think for a moment and think, how often will this test be run? is it feasible enough to be done manually? how to say the test was successful?

Leave a Reply