All Pairs Testing

All-pairs testing or pairwise testing is a combinatorial method of software testing that, for each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of those parameters. Using carefully chosen test vectors, this can be done much faster than an exhaustive search of all combinations of all parameters, by "parallelizing" the tests of parameter pairs.

Mobile Test Automation

Testers are facing more and more challenges with data-centric mobile applications as the growth of features, OS’s, devices is growing with lightning speed. Luckily there is always a nice safe place called automation that comes to assistance covered in shining armor and with a sword-of-making-things-less-hard thingy. What may be automated? Well, pretty much all of functional as well as non-functional features of the app’s behavior. Basically test automation is a blend of Environmental, Performance, Behavioral as well as Complete Ecosystem testing. But let’s Deal with one at a time.

Environmental Testing

• Usage of actual devices. This is the very first thing any tester is to consider with environmental testing. Real deal devices just prove to be more accurate than even the best of emulators. One more thing to remember here is that app’s behavior may be highly affected by target geolocation, latency, carrier infrastructure and, of course, a network type. You are to include various network changes into your testing strategy.
• Automated switching of networks. This may assist with locating changes in the app’s behavior and will assist with identifying performance issues.
• Automate app installation via OTA. A successful app is bond to run on multiple platforms thus you are to be 100% sure it’s installation and update are working as planned. This is also important due many different ways of software distribution.
• Notification services are not to be misses as well. All kinds of notification tests from a low battery charge indicator to an email with updates and error messages are to be automated to make life easier.
• A 3-d party app is also to be installed on the device. An app that will be doing things like log running or multiple server-related operations in order for the original app’s performance is under test with all the data related to things like CPU loads, etc.

Performance Testing

• CPU and memory analysis. A fairly nice practice for testing will be collecting and analyzing various CPU, battery and memory statistics. This is a great starting point for locating possible issues. Thus you will require an automation tool that will do just that very task.
• Responsiveness of the app. A nice thing to do here is calculating the total app’s screen collapse time as well as time required for navigation. All has to be fast and smooth for users to love it.
• UI nav performance. Focus on actual UI-response times in your strategy as it is of the most important things in your app to your end-users.

Behavioral Testing

• Change of Screen Orientation. The tricky part is that those changes may be triggered by different things in different OS’s. If we are talking Android you are to define all of the configuration changes for all the changes like keyboard extension, etc. rememberautomate android testing is different from Windows or iOS automate testing. Plus consider what will happen if the phone is not connected to the network due, let’s say, airplane mode. Also you are to test how the app is handling screen changes with native features of the platform itself.
• System VS User events. Happenings that are called system events are the codetriggering this or that button written by the developer when user events are the actions triggered by the end-user tapping this or clicking that. This means that the final event is to be triggered at the very same location of the user event and your automation tool has to differentiate between those events. Also try to think in advance as users are unpredictable. Test how the app manages inputs of invalid data.

The perfect Automation cycle

• Test-&-result protocol. It has to be defined clear and accurate. Especially at the actual ‘reason of failure’ part.
• Management of the devices. Make sure there is an appropriate system that manages and categorizes your devices.
• Test case management. With the vast amount of test cases that will be emerging a proper system of managing and categorizing those also has to be present. And may also be automated meaning test cases will be assigned to their very own target devices, etc.
• Proper result reporting system. This includes the statistics of crucial defects.

5 Success factors of Multivariate Testing

Every successful online business does it. Whether you want to get higher conversion rates, more newsletter signups, product purchases, news feed subscriptions or anything else - you need to do multivariate testing.

To do this smartly and ensure the best results, you are to remember the following 5 essential things:

1. GOALS

Before diving into the procedure, you have to build a plan outlining your business goals, that is, what you want to achieve by this multivariate testing campaign. The truth no one is going to argue against is you can’t get measurable results without first identifying issues to be resolved and goals to achieve. Thus, your multivariate testing strategy needs careful planning so that you can allocate your efforts and time smartly by focusing on the right design elements and on the brand goal. Once you have an accurate timescale ready to be implemented, you’ll also know if your testing time and data gathering are realistic.

2. METRICS

Another multivariate testing success factor is how well you understand the test outcomes, because this understanding will let you make informed changes and updates on your site. Metrics intend to inform you on the user behavior and its change after the testing. Being a gold mine in identifying your online business problems, metrics can also show you the number of visitors which aren’t converting, bouncing and even what they were attracted by instead. Using this valuable data you’ll be able to know which areas of your site are to be tested first, provided you already know which of them directly affect the plans and goals you’ve set.

3. INTERFERENCE PREVENTION

When you test the design variations on two different pages, chances are test interference will arise. To secure yourself from such issue, make sure the testing solution you are using allows you to keep page designs independent of one another. The testing software should be able to explain the way it handles several tests at work. Otherwise, running multi-page tests containing design variations on both pages, you risk the chance variables having effect on customer performance may change.

4. TESTING ATTRIBUTION

Another problem you’ll want to avoid is getting skewed results as a result of overlooking some testing attributes. For example, inaccurate results are likely to spring when there are multiple paths to one and the same goal. If you forget to change some minor element on the tested pages, this can make the test outcome misleading and you’ll be unable to access with confidence where the users are contacting you from. To make a long story short, whenever there are several ways of completing a task, you should get aware of them to prevent yourself from making wrong conclusions afterwards.

5. STATISTICAL CONFIDENCE

As we all know, the right decisions are made only with some level of confidence achieved from the information everybody has in the form of knowledge and experience. When it comes to multivariate testing, it means that to make the right conclusion from test results, you need to reach a high level of statistical confidence. Statistically significant test results ensure you understand the test outcome correctly and avoid reading too deep into them when you have just a few conversions. Not to be caught on fake statistical significance levels, make sure you know how those percent rates are formed in your reporting data and remember that smaller sample sizes in multivariate testing usually produce more statistically significant results. WRAP-UP Being an essential part of every marketing campaign, multivariate testing reduces the number of decisions taken with the help of your gut feeling and lets you know what your customers really need. To always get positive results of your multivariate testing efforts, avoid the common mistakes by following the advice given in this article.

Differences between QTP (HP UFT) and Selenium

What is QTP?

It provides functional and regression test automation for software applications and environments. HP Unified Functional Testing can be used for enterprise quality assurance.
HP Unified Functional Testing supports keyword and scripting interfaces and features a graphical user interface.[4] It uses the Visual Basic Scripting Edition (VBScript) scripting language to specify a test procedure, and to manipulate the objects and controls of the application under test


What is Selenium?

Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.

Selenium has the support of some of the largest browser vendors who have taken (or are taking) steps to make Selenium a native part of their browser. It is also the core technology in countless other browser automation tools, APIs and frameworks.


Here we have a quick comparison:

HP UFT (QTP)	Selenium
It is commercial tool and hence it requires a license is expensive	Its an open source tool hence it does not require license and is free
It is used for testing client-server applications. It can test web- based as well as desktop applications	Using Selenium only web applications can be tested
QTP  tests can only be developed in QTP IDE	Selenium has the option of using wide range of IDEs like Visual Studio, Eclipse, Netbeans
HP UFT only supports VB script	Selenium supports JAVA, .NET, Ruby, Perl, PHP, and many other programming languages
HP UFT comes with built in object repository.  Object repository development and maintenance is quite easy in HP ALM	Selenium dose not  have such built in object repository, but object can be managed by using UI element user extension
HP UFT integrates with test management tool like HP Quality Center	There is no such tool that integrates with Selenium
All types of dialog box is supported by it	It supports dialog box partially
It supports different environments like SAP, Oracle, .NET  but user need to purchase add-on license for them	It supports addition of plug-ins to include features that are not provided by core Selenium
Automation testing is expensive due to licensing cost	Web automation testing with Selenium incurs less cost
HP QTP offers very good technical support	It has no official support, since it is an open source
HP QTP supports only VBScript . Hence Object Oriented Approach to Test Automation becomes challenging	Selenium supports mature Object Oriented Language like Java
HP QTP is more user friendly and scripts are developed quickly.	It&rsquos a low level tool with less functionality. Script requires more time to develop and maintain
Backward compatibility is powerful. Latest version of HP ALM supports code that was developed 5 years back	With new Selenium release the API changes. So Test Scripts need to be updated

Automating UI With Calabash and Cucumber

VALIDATE SMART!

Simply pressing ‘validate’ is not enough. It never was and it never will be! Testing sessions are held in order to validate whether the app is acting as it is supposed to. Every tester at a certain point of his career realizes he actually sucks at testing. Sure you may take the device with the app installed, swing it around, tap every button and call it a day and hit validate. But will such an approach work for the end-users? Will they get a product of highest quality? Unlikely.

But it’s never late to begin with advancing your skills as a tester. Automating UI is a great way of getting accustomed with both scripting and automation in general. Plus such an approach will bring profit to all the projects you are working in.

LET’S BEGIN WITH GHERKIN

What’s UI automation? The process of making the machine tapping all the buttons you would by personally if speaking in general. You will be using the Gherkin language to do so. The language will convert you language into testing phases you wish to automate.

How’s it working? Well, here is a fine example of Gherkin in use:

Feature: Some words you will use to describe what is the actual value desired by business requirements for This particular feature and, of course, there will be some extra info about the parts that will make the actual feature quite easier to understand.

Scenario: Creating a new document

Given a brand new document

When I touch ‘create new doc’ button

Then I should be asked ‘what will be the name of your new document’

And any other possible action

And one more

Then there has to be a predictable result

That is an example of how Gherkin is working for you. Does not seem too hard, is it? But what was all that blah-blah about and what does it have to do with coding?

• The first few words were starting the feature followed by some descriptive text, supposed to describe the actual business value of the feature
• Then the scenario was started
• The lines afterwards were the steps the scenario should follow

This is an actual real (I am not making things up) scripting language. Some are finding it a bit awkward (the nicest fitting word I’ve came up with) but hey, it works and is doing what it’s supposed to. the steps are centered around one scenario. All of them are either marked with

• Given (is setting up the test’s preconditions)
• When (is specifying an action taken or the any other event)
• Then (is validating the app’s state)

or

• And
• Or
• But

are used to make steps more readable. These labels have no actual meaning and will be ignored during the test process, thus they are simple assistants of readability.

Such scripts may be used in many ways by the way hence they are written in readable English. They may be used as actual documentation hence they are pretty much explaining what a feature has to be capable of and what it is expected to do.

CAN SUCH SCRIPTS GO DEEPER?

Do you know what will happen if these scenarios will be written before the actual app’s code itself? You will be into a process looking like TDD. But regarding on the higher usage level we are actually talking about BDD. How is the BDD process going?

• You are imagining a potential feature
• You are documenting it as a pack of users scenarios (like in the example above)
• Then you code until the code passes steps 1 and 2

That is where you will require assistance of the Cucumber tool for executing your scenarios (as all of testing cant flow in imagination only). Cucumber is designed to handle such exact workflow thus it is by default the perfect thing to use.

CALABASH

If you are into mobile application testing calabash is what you are looking for hence it is cross-platform (meaning it is supporting both Android and iOS apps) and will allow you to write and execute automated acceptance tests for your mobile application. The closest thing is the Selenium WebDriver. But you are to realize that interacting and testing a mobile app from a desktop computer is a totally different experience from what you may experience using a touchscreen.

The Calabash choice is predetermined hence its APIs, specialized on touchscreen device running apps. Calabash will be doing the testing and it’s calls will be done via Gherkin DLS Step Definition files.

SEEING YOUR APP FROM A DIFFERENT PERSPECTIVE

Note: To be more readable and understandable the next part will have examples of an iOS app under test. The process with Android apps is not very different, yet speaking of a particular example has more educational value than just few general words.

Once Calabash is installed you are granted access to a tool named calabash – ios console that will grant you with an accessible Calabash API while an interactive Ruby session is commencing. All that needs to be done is running your app in an iOS simulator. Launch the console and you are ready to go. Go where? Exploring, of course.

You are free of typing in queries as query(“view”) that will allow you to see all the views that are visible. This query will allow you with:

• A list of visible views
• Those view’s classes
• Their screen position
• Accessibility labels

The first step was made and all that is left is an interesting journey around your application in order of discovering what is hidden beneath the fancy looks. Discover which exact queries are identifying the views you were looking for.

Exploring may be quite fascinating, yet it tend to take way too long. You may use a short-cut query named lables. All it does is actually labeling things, thus after typing labels(“button”) you will see a list of UI buttons that are visible on your screen.

WHAT TO ACTUALLY DO?

• First thing’s first. Install calabash as well as Cucumber. You are to use gem package by Ruby to do so.
• Calabash will be using an internal server for interacting with your app thus make sure it’s set up fine.
• Create some cucumber features. Invoking a command $ calabash – ios gen inside your iOS directory will create a sub-directory for your features.
• Write features like the ones in the beginning of this post.
• Cucumber will run your features as well as their scenarios.
• Sett the environment variables if necessary before executing Cucumber with a line like:
  $ SDK_VERSION=7.0 DEVICE=iphone cucumber
• Any specific features may be run by you as well, as arguments to Cucumber itself
  $ cucumber features/rename.feature

Enjoy a well-tested app!

Software Quality Assurance Manager

• Achieves quality assurance operational objectives by contributing information and analysis to strategic plans and reviews; preparing and completing action plans; implementing production, productivity, quality, and customer-service standards; identifying and resolving problems; completing audits; determining system improvements; implementing change.

• Meets quality assurance financial objectives by estimating requirements; preparing an annual budget; scheduling expenditures; analyzing variances; initiating corrective actions.

• Develops quality assurance plans by conducting hazard analyses; identifying critical control points and preventive measures; establishing critical limits, monitoring procedures, corrective actions, and verification procedures; monitoring inventories.

• Validates quality processes by establishing product specifications and quality attributes; measuring production; documenting evidence; determining operational and performance qualification; writing and updating quality assurance procedures.

• Maintains and improves product quality by completing product, company, system, compliance, and surveillance audits; investigating customer complaints; collaborating with other members of management to develop new product and engineering designs, and manufacturing and training methods.

• Prepares quality documentation and reports by collecting, analyzing and summarizing information and trends including failed processes, stability studies, recalls, corrective actions, and re-validations.

• Updates job knowledge by studying trends in and developments in quality management; participating in educational opportunities; reading professional publications; maintaining personal networks; participating in professional organizations.

• Enhances department and organization reputation by accepting ownership for accomplishing new and different requests; exploring opportunities to add value to job accomplishments.

Main Activities of a Test Automation Engineer Leader

• Decide what should be automated, to what degree, and how to select tools to support testing and organize any training in tool use for testers

• Design of Test Scripts (Automated Test Cases)

• Generate reports of scripts execution

• Test automation planning

• Automation Code Review

• Decide about the implementation of the test environment

• Write test summary reports based on the information gathered during testing

Main Activities of a Software Quality Control Leader

• Coordinate the test strategy and plan with the Project managers, team leader and Bussines Analyst.

• Write and review the test strategy for the project

• Plan the tests - considering the context and understanding the test objectives and risks, including selecting test approaches, estimating the time, effort and cost of testing, acquiring resources, defining test levels, cycles, and planning incident management

• Initiate the specification, preparation, implementation and execution of tests, monitor the test results and check the exit criteria

• Adapt planning based on test results and progress (sometimes documented in status reports) and take any action necessary to compensate for problems

• Set up adequate configuration management of testware for traceability

• Introduce suitable metrics for measuring test progress and evaluating the quality of the testing and the product

• Decide what should be automated, to what degree, and how to select tools to support testing and organize any training in tool use for testers

• Write test summary reports based on the information gathered during testing