Skip to content

Test Execution

Test Execution

What is Test Execution?

Test execution involves the scheduled execution of manual and automated tests to check the functionality, security and performance of software. Test cases must be fully defined, the test environment must be set up correctly and the test data must be up-to-date. Defects and deviations must be precisely documented and forwarded to the development team in order to identify potential problems at an early stage and rectify them before the software goes live or is delivered.

Challenges in Test Execution

Test execution brings with it various challenges that can occur during both manual and automated test execution. Overcoming these challenges effectively is crucial for the quality and reliability of the test results.

  • Inaccuracies in the specification
    Unclear or vague formulations in the test specification offer too much room for interpretation and lead to varying test results.
  • Management of manual and automated tests
    The parallel management of manual and automated tests requires increased attention to ensure consistency and efficiency.
  • Different specifications
    The differences in the specification of manual and automated test cases make their integration more difficult.
  • Design of automated and manual tests
    It is a challenge to design automated tests in such a way that they can be carried out manually and, conversely, to design manual tests in such a way that they can be automated quickly.
Let’s find a solution together!

Test Execution with TestBench

Assistant for manual testing

TestBench offers the option of specifying tests in various ways: Prose text, textual test steps and formalized test design using Keyword Driven and Data Driven Test. This makes TestBench a powerful tool for test specification and test execution. To ensure optimal manual execution, the testers are supported by an assistant: “imbus Test assistant for Online and Remote eXecution” – iTORX for short – which is adapted to the needs of the testers and offers very good support for all types of specification. iTORX creates an overview, enables tests to be executed repeatedly in the same way and also guides through very precise and detailed test specifications. Test execution can be interrupted and resumed. Intermediate results can be saved in TestBench at any time so that they can be continued by other people.

Screenshot from TestBench showing a test execution report for the scenario 'calculate price – KWDT' with a defect. The selected test case 'special model with additional add-ons' fails at the step 'check price'. The defect ID DC-1234 is recorded with the title 'wrong automatic discount for 3 add-ons'. The execution comment notes that the automatic discount for 3 add-ons was not applied as expected. The required price parameter was 19529.0.
Checking the results of Test Automation
TestBench screenshot displaying a failed test step during the interaction 'Select Special Model' with parameter 'Sondermodellname = Jazz'. The execution log shows a timeout error after 10 seconds. The system attempted to click a UI element labeled 'Edition', but the locator could not find the target. The result is marked as FAIL, and a screenshot of the UI state is shown below the log.

In addition to supporting manual testing, iTORX can also be used to preview the specified tests and to load the test result logs of a test automation run. It is particularly useful that iTORX can also be used offline by the TestBench system, e.g. in special test environments. This offline use is license-free and can therefore be used on any number of systems by any number of users. Long-running, automated regression tests in particular can generate huge test result logs that can contain several hundred thousand calls to test steps. iTORX has special settings for these large logs that enable high-performance work despite the large volumes.

Flexible integration of automation tools

TestBench supports manual and automated testing – if required also in mixed operation and with different test automation tools. The open interface enables the connection of almost all test automation tools, regardless of whether user interfaces such as Web, SAP or Java Swing are being tested or systems with real-time requirements such as HiL systems need to be controlled. The connection of the open source test framework Robot Framework is particularly well integrated.

"Command line execution of automated Robot Framework tests integrated with TestBench. The console shows successful execution of two test cases, both marked as PASS. Output files were generated, converted to a JSON report, and then uploaded to the TestBench server using the TestBench CLI. The upload confirms the start of a job with ID 3218422 and job key 3218321, indicating that all jobs finished successfully.
Fast documentation of observations and error recording
"Screenshot showing the creation of an error reference in TestBench for a failed test case related to price calculation. The defect DC-1234 is titled 'wrong automatic discount for 3 add-ons' with status 'New', priority '3-MustHave', and class '2-Incorrect'. The defect description indicates that the first-level discount was not applied for three add-ons, but only for four. The test case 'check price' failed with a required price of 22157.39.

During the manual test execution or the analysis of the test automation result logs, it is possible to document all observations, whether as comments, screenshots or file attachments. If errors are found, they can be recorded directly on the associated test case or test step.

Management of test protocols and results

The logs and results of manually or automatically executed test cases are transferred to TestBench. Test protocols generated offline can be imported and are then managed in the same way. During automated execution, the execution times are automatically measured and also logged. During manual execution, this time measurement must be activated by the testers and can also be switched off completely.

Screenshot of a successfully executed test case in TestBench. The test case itb-TC-19619 has a status of 'executed' with a verdict of 'PASS'. The responsible tester is Theo Tester, with execution completed on 2024-08-14. All three test case instances (PC-119581, PC-15177, PC-15180) passed within approximately 3 to 5 seconds each. Variants include [all], [new], and [premium] special models. Execution was not blocked and no propagation was triggered.
Integration of CI/CD systems
Diagram of a continuous integration workflow integrating TestBench. The process starts with a developer committing code to a source code control server, which triggers a job on a continuous integration server. The workflow includes building, deploying to a test environment, fetching test instructions from TestBench, executing tests, importing results into TestBench, analyzing results, and providing feedback to the developer.

In addition to the direct execution of automated tests, the tests can also be executed by continuous integration systems such as Jenkins or Atlassian Bamboo. Test instructions can be exported and test results imported via TestBench’s REST API. It is possible to set the associated tests to “running” status so that it is clear which tests are currently being executed. Predefined filters can be used during export so that specific test packages are executed. This allows you to control which tests are to be executed overnight or as regression tests for certain builds or sprints, for example.

Efficient task assignment: Test managers can assign tasks quickly and easily, which avoids long coordination processes during test execution.

User-oriented interface: A convenient test execution view enables testers to familiarize themselves quickly and only see relevant tests and information.

Error recording and precise documentation: Errors can be recorded directly during manual test execution in iTORX. The built-in version management ensures that tests and test results are reliably documented, regardless of the version of a test specification or test run.

Clear instructions: The iTORX test execution assistant displays test instructions step by step, which is particularly helpful for manual testing.

Flexible usage: iTORX is part of TestBench and license-free! It enables both online and offline use, even if there is no connection to the TestBench system.

Seamless integration: Test results are automatically linked to the TestBench test cases and errors are synchronized with external defect management systems.

Flexible automation: Manual tests can be automated at any time, without prior specification. Automation results can be easily imported into the TestBench and reports for manual and automated tests can be created together.

Integration into CI/CD systems: During automatic builds and subsequent test runs, e.g. in Jenkins or Atlassian Bamboo, tests can be automatically retrieved from TestBench, executed and the results imported.

Your questions – our answers

Do you have questions about test execution or are you looking for support in optimizing your test processes? Our experts will be happy to help you. Contact us and we will help you to design your test strategies efficiently and successfully.

How can we help?