The effectiveness of formative evaluation in the development of usable software: a case study

Abstract

The present case study examines the effectiveness of usability testing as a method for designing usable software. An online bookshop was evaluated two times in an iterative development cycle by means of usability testing. Design implications were derived from the usability problems identified in the first test and then implemented. The revised version of the bookshop was evaluated again. The sample included 22 student subjects of whom 11 each were expert and novice users. Ten were male and 12 female. In both usability tests (with 11 subjects each) the same three consecutive tasks were accomplished and used to evaluate usability-problems of the bookshop. In line with the hypotheses the quantity of usability problems decreased from the first to the second usability test. Contrary to expectation, however, no change in the severity of the usability problems was revealed. According to the study's assumptions an effect of the evaluation-design cycle on the subjective assessment of the bookshop by the users was also found. The shop's attractiveness as well as its hedonic and ergonomic quality were rated higher at the end of the cycle. Moreover, the expertise of the users proved to be a relevant factor that influences the number of criticisms made regarding the product. Implications for practice and further research in the field of usability engineering as well as the need for a reasonable definition of the psychological construct of usability problems are discussed.