When using automated tests it is especially important that the results and logs produced can be trusted. In the early stages of implementation we often found ourselves having to manually observe automated tests in order to find out exactly what happened, since our logging was insufficient. After expanding the logging we could save a lot of time since we could actually get sufficient information even though something unexpected happened in the test.
38
When working with large test projects it is important to refactor of reoccurring code into separate functions. These functions should also be as general as possible. We found that lots of the older existing test scripts at the company included multiple versions of the same pieces of code, with slight differences, and thus it was quite difficult to keep track of what was being tested.
It is worth spending time finding out the constraints of the test tool early, and develop your tests accordingly from the start, since it might be time consuming to refactor the code later.
Do not assume functionality of your test tool. Try to find out early if certain operations such as syncing projects and moving stuff around actually work. You might not find out that certain important features are lacking until later on in the project, and had you known that earlier you would’ve developed in a way that would’ve been less dependent on those features.
Do not develop for too long in a "simulated environment". Start running the tests on the intended platform early. This will help you find out if you have designed the tests in an incorrect way
While using a mapping layer between the script and the tested application generally simplifies the process of keeping the test up to date with the software, it is very important to check out the way in which the mapping is handled in the test tool. It is good to find out early if you think you can rely on features such as moving and copying mappings from different projects, and merging mapping with each other. If certain features are cumbersome to use, you might have to design around them.
39
References
[1] “Automatiska GUI-test : En teoretisk och praktisk studie för hållbarare test”, Sara Forslin, 2010
http://www.uppsatser.se/uppsats/1902327764/
[2] “Making the Right Choice: The features you need in a GUI test automation tool”, Elisabeth Hendrickson, 2005
http://www.cnblogs.com/nckiki/articles/250803.html
[3] “Using open source tools for performance testing”, Goranka Bjedov, 2006
http://www.myloadtest.com/open-source-tools-performance-testing/
[4] “The ROI of test automation”, Michael Kelly, 2004
http://www.stickyminds.com/sitewide.asp?ObjectId=8502&Function=DETAILBROWSE &ObjectType=ART
[5] ”Cost benefit analysis of test automation”, Douglas Hoffman, 1999
http://softwarequalitymethods.com/Papers/STAR99%20Model%20Paper.pdf
[6] ”Test Automation Architectures: Planning for Test Automation”, Douglas Hoffman, 1999
http://softwarequalitymethods.com/Papers/autoarch.pdf
[7] Smartbear’s website, 2011
www.smartbear.com
[8] “Seven Steps to Test Automation Success” , Bret Pettichord, 2000
http://www.testpoint.com.au/attachments/093_Seven%20Steps%20to%20Test%20Automatio n%20Success.pdf
[9] “About WMI”, MSDN.com
http://msdn.microsoft.com/en-us/library/windows/desktop/aa384642(v=VS.85).aspx
[10] ”Timer object” Smartbear.com wiki,
http://smartbear.com/support/viewarticle/13306/
[11] “Software Testing”, Ron Patton, 2005, 2nd Edition, ISBN-10: 0-672-32798-8 [12] “Test Automation Snake Oil,” , James Bach, Windows Tech Journal, 1999 [13] “Agile Testing”, Grig Gheorghiu,
http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress- testing.html
[14] “Experiments with High Volume Test Automation”, Pat McGee
http://dl.acm.org/citation.cfm?id=1022494.1022536&coll=DL&dl=ACM&CFID=77 462952&CFTOKEN=60911683
[15] “Success with Test Automation”, Bret Pettichord, 2001
40
Related work
Sara Forslin – “Automatiska GUI-test : En teoretisk och praktisk studie för hållbarare test” adresses some of the issues of creating maintainable auotmated tests. An evaluation of TestComplete is also featured. The report features advice on how to design the scripts to make sure that they will be kept up to date with as little work as possible, for instance discusses the importance of using a mapping layer when GUI-testing. Some recommendations for test designers is to focus on automating parts of the software which will provide the most benefit. Automating Performance Testing of Interactive Java Applications
http://dl.acm.org/citation.cfm?id=1808268&bnc=1
The paper presents an approach on automatic testing of GUI applications’ perceptible performance with different capture & replay tools. It makes it possible to perform comparative studies of the perceptible performance, for example by testing an application’s latency distribution on different operating systems. It also makes it possible to compare results of regression performance testing between different interactive application versions.
Success with test automation Bret Pettichord
http://www.prismnet.com/~wazmo/succpap.htm
This publication talks about how to really succeed with test automation in the real world. Howt to approach the tests in order to discover problems with testability in an early stage is one recommendation. Certain compromises might need to be made, and you might have the best rate of success if you don’t try to do too much, but rather focusing on the areas where the big gains are.
Cost Benefits Analysis of Test Automation Douglas Hoffman
http://softwarequalitymethods.com/Papers/STAR99%20Model%20Paper.pdf
The paper presents research on how test automation is not always necessary, appropriate or cost effective. By analyzing expected return on investment (ROI) it is possible to determine if test automation can be profitable. These results are computed by comparing the costs and gains that are achieved through automated testing compared to manual testing. To compute ROI is not always helpful though, as intangible factor may interfere with the calculations, making them not display the real value of automation. The paper also discusses how to value test automation and shows that management issues can lead to mistakes and misplaced expectations.
The ROI of test automation Michael Kelly
http://www.stickyminds.com/sitewide.asp?ObjectId=8502&Function=DETAILBROWSE&Object Type=ART
The article investigates different test automation offers guideline types and shows how their return on investment (ROI) is calculated. The purpose is to provide better understanding for test automation costs and the ability to what types of automation will yield the greatest ROI. Some of the conclusions that are reached are that the topic of test automation as well as the topic of calculating costs and benefits for test automation is very large and complex; and that it therefore is not possible to provide equations that will work for all kinds of projects. General guidelines, ideas for maximizing ROI and calculation examples with given formulas are provided.
41 Experiments with High Volume Test Automation
http://dl.acm.org/citation.cfm?id=1022494.1022536&coll=DL&dl=ACM&CFID=77462952&CFT OKEN=60911683
The paper describes a broad class of testing techniques that are collectively called High Volume Test Automation (HVTA). The main purpose of the techniques is automated execution and evaluation of large numbers of tests that are run for the purpose of exposing functional error that are otherwise hard to find. The authors propose to find additional existing industry HVTA projects in order to write case studies of them and later on be able to create their own to for the technique’s implementation. HVTA techniques are more effective to use on already stable code, because it is more suitable to test for reliability on such code that passes basic functional tests.
42