Test Automation - How To Automatically Skip Failing Test Cases

Introduction

In this article, I will review an ability to automatically skip failing test cases which is based on annotation reflection. The technological stack used to implement the solution is:

Programing language: Java

Build tool: Maven

Testing Framework: TestNG

Reporting Framework: Allure

The Problem

We have two problems, the first one is how to indicate that a test has issues - automation/product known issues that fail the test each time.

The first problem is solved by allure, which supplies an annotation called Issues for multiple defects and annotation called issue for marking a single defect.

This is the code implementation of a test method that contains issues:

@Issues({
    @Issue("AUTO-50"),
    @Issue("AUTO-60")
})
@Test(description = "Response time assertion")
public void responseTimeAssertion() {
    // some test code
}

The second problem which this article is addressing – how can I skip these failing test cases? In case our automation team wants the report to indicate only the passing test cases.

The Solution – Skipping Test Cases Based On The Issues Annotation

What is reflection

According to Oracle:

Reflection is a feature in the Java programming language. It allows an executing Java program to examine or “introspect” upon itself, and manipulate internal properties of the program. For example, it’s possible for a Java class to obtain the names of all its members and display them.

Our solution uses reflection because it examines the test method annotation presence and values at runtime.

Implementing the Solution

We have implemented a test listener that checks before each test these conditions :

• The method is not null – in case the test method does not contain anything.
• The method contains the “Issues” annotation.
• The “Issues” annotation contains issues - it should not be an empty array.

In case some / all of these conditions are false:

We run the test method normally.

In case all of the conditions are met:

1. We create a new array list that contains the issue values.
2. We throw a TestNG skip exception which causes the test method to be skipped and specifying the issue values from the issue list to the automation developer running these tests.

This is the code implementation of the test listener class is:

public class ListenerClass extends TestListenerAdapter {

    @Override
    public void onTestStart(ITestResult result) {
        Method method = result.getMethod().getConstructorOrMethod().getMethod();
        if (method != null && method.isAnnotationPresent(Issues.class) &&
            !Arrays.asList(method.getAnnotation(Issues.class).value()).isEmpty()) {
            ArrayList<String> issues = new ArrayList<>();
            Arrays.asList(method.getAnnotation(Issues.class).value()).forEach(issue ->
                issues.add(issue.value()));
            throw new SkipException(
                String.format("Open Issues %s Skipping this test", issues));
        }
    }
}

The outcome of the run in the generated report is:

The test case was automatically skipped, and the reason was mentioned with the values of the defects.

In conclusion

In this article, we reviewed the problem we wanted to skip failing test cases to reduce the testing time and improve the percentage of the passing test cases. We implemented a solution to automatically skip failing test cases based on our reporting framework annotation reflection.

Happy testing!