My Portfolio / Usability Testing and Heuristic Evaluation / DigiTally Mobile Prototype
Usability testing an offline mobile payment system
Aim: Obtain user feedback on a high-fidelity prototype of a mobile payment system, DigiTally, that would allow users in areas with poor connectivity to securely make offline payments.
Problem
DigiTally is an overlay payment scheme for use in mobile phones created by Khaled Baqer as a PhD student, under the supervision of Prof. Ross Anderson at the University of Cambridge. It was created to allow users to make offline payments by copying a short string of digits from one mobile to another in the face of network congestion and outage. Khaled wanted to test how well this would be received by intended end-users and approached me for input.
Process
Product research
I conducted a competitor analysis of MPESA while in Nairobi, Kenya.
2
Cognitive walkthrough
I conducted a cognitive walkthrough of the DigiTally system.
Usability study design
I designed a usability for the developer to implement to implement.
4
Analysis and reporting
I analysed and wrote up the quant. and qual. results for a paper.
1
3
Process
Following product research of the most popular existing payment system in Kenya (MPESA) and a cognitive walkthrough (a type of heuristic evaluation) of the DigiTally system in its early development, I designed a usability study, which was implemented by the developer in Kenya.
20 participants from a university in Nairobi were given dedicated feature phones preloaded with Ksh 2000 (≈ £15.00) to make real transactions using DigiTally with merchants (cafeteria, bookshop, and local coffee shop) who had agreed to accept this payment system during the study.
Following a week of use, participants completed a survey in which I included the System Usability Scale (SUS), alongside open-ended questions. Additionally, the phone SIMs securely collected performance and transaction data: error rates, transaction times, number of transactions, and PIN attempts.
I analysed and reported error rates, transaction times, SUS scores, and answers to open-ended questions. I assessed the qualitative data based on the three sub-constructs of usability outlined in the ISO-9241. The results were discussed and agreed upon with the developer.
20
Participants from a university in Nairobi, Kenya
2
Week usability testing and analysis period
1
Published conference paper (2017)
Outcomes
We demonstrated that DigiTally was easy to use. The average speed for every participant was less than a minute, and the average transaction speed was close to half a minute. Participants also liked key aspects of the system, notably, its security and that it did not require network coverage to process payments.
Some specific technical improvements were still needed, most notably the error recovery process. While errors were not at all common, it was clear that, should errors occur, recovery needs to be more intuitive. The resulting paper was published in the Symposium on Usable Privacy and Security, 2017, titled: "DigiTally: Piloting Offline Payments for Phones".
Challenges and learnings
I served as a voluntary advisor to work with the developer of DigiTally to help him validate his system and to provide the groundwork for improving it. This was at his discretion. As a collaborator on this project, which formed part of a PhD thesis completed by Khaled Baqer in 2018, I could not make final decisions about the direction of research and how the results were presented. I was, however, deeply involved in interesting discussions, free to offer my knowledge and opinions, and happy to contribute to this exciting project.