NTT DATA Canada Discussion Room

Software Test Automation Framework, What a QA Manager needs to know?

by Anurag Wani, NTT DATA Vertex 

What is a Software Test Automation Framework, standard types of Test Automation Frameworks, characteristics of a Test Automation Framework and how to choose a Test Automation Framework.

Introduction

Recently Test Automation Framework is becoming a buzzword in software testing, and why it should not be?  As it makes it possible to take most of the benefits from test automation on a Cost, Quality and Time basis.

 Here we will discuss what an automation framework is?  Which type of frameworks are available and which do we choose to fulfill our needs in automation.

Constructing the word Test Automation Framework:  the word ‘Framework’ means an abstraction over a generic functionality that can be changed by changing the user code.  A Test Automation framework is a collection of function libraries and associated folder structures to provide an environment to automate the test in an easy and standard way along with a low maintenance cost.  So we can define Test Automation Framework as:

Test Automation Framework: “A set of assumptions, concepts, and practices that constitute a work platform or support for automated testing”

“Software bugs are costing the U.S. economy an estimated $59.5 billion each year.  Improvements in testing could reduce this cost by about a third, or $22.5 billion,” Source: “National Institute of Standards and Technology (NIST)”

TABLE OF CONTENTS

 Why  Do Test Automation

Test Automation Framework

Types of Test Automation Frameworks

Choosing a Test Automation Framework

Characteristics of a Good Test Automation Framework

Summary and Sources

Comments or questions

About NTT DATA Vertex

Why  Do Test Automation:

Automating a test gives us the ability to reuse them.  It’s reliable, eliminating the possibility of human error when executing actions repeatedly.  It’s comprehensive, the ability to cover very complex scenarios repeatedly in every testing cycle.  Automated tests are faster, reducing the overall regressing test cycles and improving accuracy and test coverage.

Test Automation Framework typically contains

• A Driver script: written in some programming language, which Loads and calls the global and test level resources (like settings, data etc.)
• Various function libraries: contains functions/procedures to be performed
• Action inventory: collection of action modules
• Object Repositories: contains AUT object definitions
• Test level scripts: written  in some programming language and specific to a particular test
• Test Data: Data to be used during a test run
• Some Reporting and screenshot mechanism: to report the results, sometime used as a test log

Also a Framework can have various customized mechanisms like Log maintenance, automatic defect reporting mail systems etc.

 

Types of Test Automation Frameworks

Following are standard types of Frameworks according to users automation needs and the type of projects:

1. Linear or simple: Record and Run type frame work
2. Structured: this type of Framework has control structures for execution flow control
3. Data Driven: this type of Framework is used when we need to perform the same activity again and again but with different set of test data every time. Example is a login to a banking application.
4. Keyword Driven: usually key word represents abstraction over a function to be called and script looks like a set of keywords, each key word will be designated to particular set of activities
5. Modular: Contains actions and functions that represent a particular set of functionality. If we want to perform that set of activities, we need to just make a call to that action or function.
6. Hybrid: any mix of the above described Frameworks to use their strengths and to mitigate their weaknesses.

 

Choosing a Test Automation Framework

Every Framework has its limits and assumptions associated with it. So while choosing the Framework, the following points should be kept in mind.

Nature of the project:  if a project contains similar flows to be run with different data, than data driven Frameworks are a best fit. 

Ease of use:  If the chosen Framework must be easy to use. Complexity increases the maintenance time and cost accordingly. A good example is of an easier framework would be a ‘Keyword driven Framework’.

Resource skills:  if development resources are very junior than a keyword driven framework will be proved easy and cost effective. But if the development resources are knowledgeable and experience, we can consider a more complicated Framework like Modular, Flow based etc.

 

Characteristics of a Good Test Automation Framework

Fulfills the user’s automation need: the Framework should satisfy user’s test automation objectives.

Ease of use and maintenance: The chosen Framework must be easy to use. Complexity increases the maintenance time and cost accordingly. A good example of an easier framework would be a ‘Keyword driven Framework’.

Independent: the Framework should be tool and platform agnostic as much as possible, as in long run, it would be prove more cost effective.  If it is largely tool & techno agnostic but the cost to change will greatly reduced and allow for more upgrade freedom.  One example of this is NTT DATA’s  open source framework www.open2test which is also keyword driven, so easy to use.

Flexible and customizable: the Framework should be flexible so that it can be used in other projects or different versions of the same products with minimal changes. In short a Framework must be applicable to a wide range of Applications/projects, because application/projects specific Frameworks are not feasible.

Unattended test run: the Framework should be capable to execute the test cases in an unattended mode i.e. it should support overnight runs without human interaction.

Distributed architecture:  Most of the systems under test (SUT) works on a distributed model. The Framework should be able to work in a distributed test environment as well as with distributed SUT.

Notification: the Framework should have some notification mechanism like email, SMS etc. so one can know about the status of the test execution even when not able to attend the test run.

 

Summary and Sources

In this part of the article we discussed what is a Software Test Automation Framework, standard types of Test Automation Frameworks, characteristics of a Test Automation Framework and how to choose a Test Automation Framework.

In next part we will look into a comparative overview of every type of framework described above.

 

Bibliography: en.wikipedia.org , www.ibm.com, http://www.computerworld.com/s/article/72245/Study_Buggy_software_costs_users_vendors_nearly_60B_annually

 

Comments, questions and copyright

Please forward all comments or questions  to Jorrit.wit@vertexsoft.com.   NTT DATA Vertex reserves the right to choose which to publish, but will only do so with your written permission.  NTT DATA Vertex gives full permission to copy all or part of this article as long as we are referenced.  Copyright NTT DATA 2011.

The opinions expressed on this discussion room are writer's and don't necessarily represent NTT DATA Canada's positions, strategies or opinions.