End-to-End Testing: Best Practices and Pitfalls

End-to-end testing is a testing method that ensures that the behavioral flow of an application, as designed from start to finish, works as expected. It typically considers the most critical, as well as the most likely, business scenarios. The purpose of performing complete, thorough testing of the entire product-user interaction, is to make sure that business processes function correctly across integrated systems.

Formulating a proper end-to-end testing strategy presents many challenges to organizations. Especially if their goal is to automate such complex business processes.

In this article, we would like to share with you the golden rules for managing your testing costs while realizing the benefits of testing. We will also share common pitfalls you need to avoid. Let’s take a look at them in more detail.

Best Practices for E2E Testing

Keep an end-user perspective.

When it comes to test designing and creating test cases, think like an end-user. Focus on the features of the application, rather than its implementation.

It is a good practice to only include high-value paths in your test strategy, focusing on how end-users will go through the system to accomplish a business goal. Examples include users signing up to an e-commerce platform, creating new sales orders, or planning at home or office delivery. Those are high-impact scenarios you have to include in your end-to-end testing. They are relevant to the end-users, and they have a high impact on business objectives (e.g. revenue, quality, speed, etc.).

A good way to capture user perspective is to make use of documents such as user stories, acceptance tests, and BDD scenarios if they are available. Get into the mindset of someone using the application for the first time and answer some questions:

• What is the user trying to achieve?
• Is it easy and straight forward to find what he/she is looking for?
• Can the user accomplish its goal in a few easy steps?

As much as we are big fans of the golden rules above, at Copado we recognize this is not an easy task for a tester alone. That is why we build solutions that also empower business users to create test flows by recording the actions they perform daily. Business users can bring the end-user perspective, and they can save your teams a lot of time in creating meaningful and relevant business scenarios.

Risk-Based Testing.

Risk-Based Testing (RBT) is a critical tool for categorizing your end-to-end business processes based on business criticality. This is useful in several ways. First of all, you can segment functionality into priorities e.g. P1, P2, P3, P4. This allows you to create test plans and schedules focused on priority. You may decide in cycle 1 to test all priorities but during cycle 2 only to focus on P1 and P2 business processes. Also, you can ensure that, when entering a testing cycle that has tight deadlines, you focus the testing from P1 to P2 to P3 to P4. So, if you run out of time, you ensure you test the most critical processes first.

To determine a high-risk feature, consider both how likely failure is to happen, and the potential impact that it would have on end-users. A risk assessment matrix is a useful tool for identifying risk.

There are others but the most common risk profiles rely on these 2 questions:

• What is the impact of failure? E.g. high risk would be impacting revenue generation or halting production.
• What is the likelihood of failure? E.g. high risk would be a large number of interfaces within the process or a high level of customization/complexity.

Think of Amazon.com or Booking.com. What would be the impact of their systems not working, even for a few minutes? And customers from all over the world not being able to generate business on either platform?

Maintain the right order.

The main goal of a proper Quality Assurance process is to identify application errors at the earliest stage in the software delivery life cycle, where the cost of resolving is the lowest. Therefore, investing efforts in the unit and integration testing is paramount to maintain a solid and reliable software delivery process.

E2E testing is no less relevant to such an approach. It helps identify business-process related errors, which are difficult to find in earlier stages of the delivery process. So, execute unit testing and integration testing first. Then, during the end-to-end testing, run your critical smoke tests first to ensure integrated applications are communicating as expected, followed by sanity checks and other high-risk test cases.

When a single unit test fails, it’s quite easy to figure out where the defect occurred. As tests grow in complexity and touch more components of an application, the increase in potential points of failure makes them harder to debug when a failure occurs.

Structure, organization, and a good understanding of business logic are fundamental in end-to-end testing.

Manage your test environment and your test data.

Make your test environment setup process as efficient and consistent as possible. Ensuring the deployment teams have clear requirements for your test environments is critical for success. There is nothing more demotivating for a testing team than getting halfway through a testing cycle only to find out that an integrated component isn’t connected or has different data. Running a smoke test can ensure you check all integrated system components are up and running.

If possible, keep the staging environments as close to production as possible. For this, consider image backup of the production environment. And look to infrastructure as code or automation of your infrastructure to save time and reduce human error.

Once testing is completed, refreshing the test data (and possibly even taking a snapshot) is a good way to ensure the environment can easily be restored to it’s original state. This ensures that running another round of testing is much easier for everyone.

Specifically with SAP, Copado has joined forces with Qlik Gold Client to provide a fully tested, fully documented SAP non-production environment with scrambled, anonymized data in hours instead of months. This is handy for on-demand test data management and production incident fixes. For the use cases, have a look at the solution we built together.

Pitfalls

Let’s also have a look at the common mistakes that you can make while executing your E2E strategy.

Don’t ignore flaky tests.

More often than not, end-to-end tests fail because of the environment and test data not being synchronized across them, rather than an error in the business process itself. Common failures can occur when financial periods have changed, or stock movements have been made, but the test hasn’t taken this into account.

This can lead to ‘flaky tests’ that, if you are not careful, people will start to mistrust. And if they keep failing for what appears to be random reasons, the developers will see such tests as annoying instead of helpful. One flaky test could damage the reputation of your activities. Don’t forget that!

Isolate flaky tests, find out what is causing the issues and, if possible, engineer the solutions into the test case. A good end-to-end test will set up the data it requires, use the data, and, when possible, clean it up at the end. This minimizes the chances of failure being related to the data and environments.

If you are automating your E2E tests, avoid the selector maintenance nightmare.

Even if this is a very specific tip for the companies who choose to automate their end-to-end testing strategy, we decided to include it, as it can save you a lot of time maintaining your test scenarios.

If you are automating a web application, you will need to leverage stable selectors to locate the UI elements on the page. By selecting a stable selector you make sure that your test is not going to fail every release because your script was not able to identify the right object.

There are several ways to locate a UI element but not all of them are robust enough. This is the case of CSS selectors: classes are liable to change, which means more tests failing because of it. You can use ID selectors: they rarely change, but they sometimes do.

The best option is to create a data-test=”unique_value” attribute. This is an attribute added to the target element, used only for test purposes. As long as the value is unique, the test will be able to find it. Also, most developers/UX people know they cannot mess with it.

Tests shouldn't depend on each other.

Each test should run completely independently, and its result should not depend on the outcome of a previous test. Also, if you are testing a web application, make sure that each test starts in a fresh browser window with all cookies cleared.

What you want is to avoid the domino effect where a single failure would cause too much redundant noise in your testing ecosystem. Analyze your tests, and get rid of dependencies.

Here is a very simplistic example:

• Test 1: user logs in
• Test 2: user is logged in and creates a new opportunity
• Test 3: user assigns the opportunity to an owner

You might think that, by running these tests right after each other, you could save some time. However, if test 1 or 2 fails, the following will fail too. You've basically created a waterfall of failing tests.

So, make sure to avoid dependencies and start in a brand new browser window.

Re-run tests before raising an issue.

As we mentioned above, E2E testing is complex and more likely to fail for undetermined reasons. If a test fails unpredictably from time to time and you have run out of ideas for improving reliability, try running it again. And again, and again. We suggest 3 times, just to be safe, before raising an issue.

Don’t expect it to be maintenance-free.

Running end-to-end testing is complex work. Don’t expect it to be maintenance-free. The more complex your application gets with added features, the more maintenance you will need.

Conclusion

Running E2E testing can be challenging but everything starts with a good test design system. Keep an end-user perspective and prioritize the scenarios with higher risk.

Take care of the quality of your test environment and your test data. Run unit and integration testing first. Make sure your tests are stable enough and get rid of the flaky tests.

Simple rules and best practices will make your strategy successful.