Validating and Testing an Offline Mobile Payment System in Kenya
Aim. Obtain user feedback on a high-fidelity prototype of a mobile payment system, DigiTally, that would allow users in areas with poor connectivity to securely make offline payments.
My Role
Product research
Heuristic evaluation of prototype
Design study for testing DigiTally
Analyse and write-up results
Methods
Competitor analysis (of MPESA)
Cognitive walkthrough
Usability test planning and analysis
Survey design and analysis
Focus
Scope out usability challenges before further development of the mobile payment system, DigiTally.
Problem Statement
DigiTally is an overlay payment scheme for use in mobile phones created by Khaled Baqer as a PhD student, under the supervision of Prof. Ross Anderson at the University of Cambridge. It was created to allow users to make offline payments by copying a short string of digits from one mobile to another in the face of network congestion and outage. Khaled wanted to test how well this would be received by intended end-users and approached me for input.
Process
Following product research of the most popular existing payment system in Kenya (MPESA) and a cognitive walkthrough of the DigiTally system in its early development, I designed a usability study, which was implemented by the developer in Kenya.
20 participants from a university in Nairobi were given dedicated feature phones preloaded with Ksh 2000 (≈ £15.00) to make real transactions using DigiTally with merchants (cafeteria, bookshop, and local coffee shop) who had agreed to accept this payment system during the study. Following a week of use, participants completed a survey in which I included the System Usability Scale (SUS), alongside open-ended questions. Additionally, the phone SIMs securely collected performance and transaction data: error rates, transaction times, number of transactions, and PIN attempts.
I analysed and reported error rates, transaction times, SUS scores, and answers to open-ended questions. I assessed the qualitative data based on the three sub-constructs of usability outlined in the ISO-9241. The results were discussed and agreed upon with the developer.
Outcomes
We demonstrated that DigiTally was easy to use. The average speed for every participant was less than a minute, and the average transaction speed was close to half a minute. Participants also liked key aspects of the system, notably, its security and that it did not require network coverage to process payments.
Some specific technical improvements were still needed, most notably the error recovery process. While errors were not at all common, it was clear that, should errors occur, recovery needs to be more intuitive. The resulting paper was published in the Symposium on Usable Privacy and Security, 2017, titled: "DigiTally: Piloting Offline Payments for Phones".
Challenges
I served as an advisor to work with the developer of DigiTally to help him validate his system and to provide the groundwork for improving it. This was at his discretion. As a collaborator on this project, which formed part of a PhD thesis completed by Khaled Baqer in 2018, I could not make final decisions about the direction of research and how the results were presented. I was, however, deeply involved in interesting discussions, free to offer my knowledge and opinions, and happy to contribute to this exciting project.