Errors, Irregularities, and Misdirection: Cue Utilisation and Cognitive Reflection in the Diagnosis of Phishing Emails

  • Mitchell Ackerley
  • Ben Morrison Macquarie University
  • Kate Ingrey
  • Mark Wiggins
  • Piers Bayl-Smith
  • Natalie Morrison
Keywords: cyber security, phishing, cue utilisation, cognitive reflection, expertise


The study aimed to examine the role of, and potential interplay between, cue utilisation and cognitive reflection in email users’ ability to accurately (and efficiently) differentiate between phishing and genuine emails. 145 participants completed the Cognitive Reflection Test (CRT), a phishing diagnostic task, and the Expert Intensive Skill Evaluation (EXPERTise 2.0) battery, which provided a gauge of users’ cue utilisation in the domain. The results revealed an interaction between users’ cognitive utilisation and cue reflection, whereby users low in both facets performed significantly worse in diagnosing phishing emails than all other groups. Further, those participants with both higher cue utilisation and cognitive reflection took significantly longer to make their diagnosis. It is concluded that a high level of cognitive reflection was able to compensate for a lower level of cue utilisation, and vice versa. Participants reported using seven types of cue during diagnosis, however, there was no significant relationship between the types of cues used and users’ level of cue utilisation. Taken together, the findings have implications to the design of user-level interventions in relation to the identification of vulnerable users, as well as the need to consider training approaches that extend beyond the use of simple cue inventories.


Anderson, J. R. (1996). ACT: A simple theory of complex cognition. American psychologist, 51(4), 355-365. https://doi:10.1037/0003-066X.51.4.355

Basnet R.B., Sung A.H., & Liu Q. (2012) Feature Selection for Improved Phishing Detection. In: Jiang H., Ding W., Ali M., & Wu X. (eds) Advanced Research in Applied Artificial Intelligence. IEA/AIE 2012. Lecture Notes in Computer Science, vol 7345. Springer, Berlin, Heidelberg.

Bayl-Smith, P., Sturman, D., & Wiggins, M. (2020). Cue utilization, phishing feature and phishing email detection. In M. Bernhard, A. Bracciali, L. J. Camp, S. Matsuo, A. Maurushat, P. B. Rønne, & M. Sala (Eds.), Financial Cryptography and Data Security: FC 2020 International Workshops, AsiaUSEC, CoDeFi, VOTING, and WTSC, Revised Selected Papers (pp. 56-70). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12063). Springer.

Bengtsson, M. (2016). How to plan and perform a qualitative study using content analysis. Nursing Plus Open, 2, 8-14. https/doi:10.1016/j.npls.2016.01.001

Brams, S., Ziv, G., Levin, O., Spitz, J., Wagemans, J., Williams, A. M., & Helsen, W. F. (2019). The relationship between gaze behavior, expertise, and performance: A systematic review. Psychological Bulletin, 145(10), 980–1027.

Brouwers, S., Wiggins, M., & Griffin, B. (2018). Operators who readily acquire patterns and cues, risk being miscued in routinized settings. Journal of Experimental Psychology: Applied, 24(2), 261-274. https://doi:10.1037/xap0000151

Butavicius, M., Parsons, K., Pattison M., & McCormac, A. (2016). Breaching the Human Firewall: Social engineering inPphishing and Spear-Phishing Emails. ArXiv, abs/1606.00887.

Canfield, C. I., Fischhoff, B., & Davis, A. (2016). Quantifying Phishing Susceptibility for Detection and Behavior Decisions. Human Factors: The Journal of the Human Factors and Ergonomics Society, 58(8), 1158–1172.

deGroot, A. D. (1978). Thought and choice in chess. The Hague: Mouton

Downs, J. S., Holbrook, M. B., & Cranor, L. F. (2006). Decision strategies and susceptibility to phishing. In Proceedings of the second symposium on Usable privacy and security (pp. 79-90). ACM. https://doi:10.1145/1143120.1143131

Fette, I., Sadeh, N., & Tomasic, A. (2007). Learning to detect phishing emails. In WWW ‘07 Proceedings of the 16th international conference on World Wide Web (pp. 649-656). New York: ACM. https://doi:10.21236/ada456046

Field, A. (2013). Discovering statistics using IBM SPSS statistics. London, England: Sage.

Fleiss, J. L., Levin, B., & Paik, M. C. (2013). Statistical methods for rates and proportions. Hoboken: John Wiley & Sons.

Frauenstein, E. D., & Flowerday, S. (2020). Susceptibility to phishing on social network sites: A personality information processing model. Computers & Security, 94, 101862.

Frederick, S. (2005). Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 19(4), 25-42.

French, K. E., & Nevett, M. E. (1993). The Development of Expertise in Youth Sport. In J. L. Starkes & F. Allard (Eds.), Advances in Psychology (pp. 255-270). North-Holland.

Gacasan, E. M. P., Wiggins, M. W., & Searle, B. J. (2016). The role of cues in expert project manager sensemaking. Construction Management and Economics, 34(7-8), 492-507. https://doi:10.1080/01446193.2016.1177190

Galanter, C. A., & Patel, V. L. (2005) Medical decision making: A selective review for child psychiatrists and psychologists. Journal of Child Psychology and Psychiatry, 46(7), 675-689. https://doi:10.1111/j.1469-7610.2005.01452.x

Gonzalez, C. (2013). The boundaries of instance-based learning theory for explaining decisions from experience. In V. S. C. Pammi & N. Srinivasan (Eds.), Progress in brain research (Vol. 202, pp. 73-98). Oxford, UK: Elsevier.

Gwet, K. (2001). Handbook of inter-rater reliability. Gaithersburg: STATAXIS Publishing Company.

Harrison, B., Svetieva, E., & Vishwanath, A. (2016). Individual processing of phishing emails. Online Information Review, 40(2), 265–281.

Harré, M., Bossomaier, T., & Snyder, A. (2012). The perceptual cues that reshape expert reasoning. Scientific Reports, 2(1), 502–502.

Hassandoust, F., Singh, H., & Williams, J. (2020). The Role of Contextualization in Individuals’ Vulnerability to Phishing Attempts. Australasian Journal of Information Systems, 24. DOI:

Johnston, D., & Morrison, B. W. (2016). The application of naturalistic decision-making techniques to explore cue use in rugby league playmakers. Journal of Cognitive Engineering and Decision Making, 10(4), 391-410. https://doi:10.1177/1555343416662181

Jones, H.S., Towse, J. N., & Race, N. (2015) Susceptibility to email fraud: A review of psychological perspectives, data-collection methods, and ethical considerations. International Journal of Cyber Behaviour, Psychology and Learning, 5(3). 13-29. https://doi:10.4018/IJCBPL.2015070102

Jones, H.S., Towse, J. N., Race, N., & Harrison, T. (2019). Email fraud: The search for psychological predictors of susceptibility. PloS ONE 14(1), e0209684. https://doi:10.1371/journal.pone.0209684

Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American psychologist, 64(6), 515-526. https://doi:10.1037/a0016755

Klein, G. A. (2008). Naturalistic Decision Making. Human Factors, 50(3), 456–460.

Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid Decision making on the Fire Ground. Proceedings of the Human Factors Society Annual Meeting, 30(6), 576-580.

Kobus, D. A., Proctor, S., & Bank, T. E. (2000). Decision-making in a dynamic environment: the effects of experience and information uncertainty. Technical Report 1832. San Diego, CA: Spawar Systems Center.

Levine, T. R. (2014). Truth-Default Theory (TDT). Journal of Language and Social Psychology, 33(4), 378–392. https://doi:10.1177/0261927x14535916

Loveday, T., Wiggins, M., Festa, M., Schell D., & Twigg, D. (2013). Pattern recognition as an indicator of diagnostic expertise. In C. P. Latorre & F. A. Sanchez (Eds.), Pattern recognition – Applications and methods (pp. 1-11). Berlin: Springer.

Luo, X., Zhang, W., Burd, S., & Seazzu, A. (2013). Investigating phishing victimization with the Heuristic-systematic model: a theoretical framework and an exploration. Computer Security, 38, 28–38.

Moghimi, M., & Varjani, A. Y. (2016). New rule-based phishing detection method. Expert Systems with Applications, 53, 231–242. https://doi:10.1016/j.eswa.2016.01.028

Morrison, B. W., Johnston, D., Naylor, M., Morrison, N. M. V., & Forrest, D. (2020). “You can’t hide your lyin’ eyes”: investigating the relationship between associative learning, cue awareness, and decision performance in detecting lies. Journal of Cognitive Engineering and Decision Making, 14(2), 99-111.

Morrison, B., & Morrison, N. (2015). Diagnostic cues in major crime investigation. In M. W. Wiggins, & T. Loveday (Eds.), Diagnostic expertise in organizational environments (pp. 91-98). Ashgate Publishing

Morrison, B. M., Wiggins, M. W., Bond N. W., & Tyler, M. D. (2013). Measuring relative cue strength as a means of validating an inventory of expert offender profiling cues. Journal of Cognitive Engineering and Decision Making, 7(2), 211-226. https://doi:1177/

Morrison, B. W., Wiggins , M. W., & Morrison, N. V. (2018). Utility of expert cue exposure as a mechanism to improve decision-making performance among novice criminal investigators. Journal of Cognitive Engineering and Decision Making, 12(2), 99-111. https://doi:10.1177/1555343417746570

Nasser, G., Morrison, B. W., Bayl-Smith, P., Taib, R., Gayed, M., & Wiggins, M. W. (2020a). The effects of cue utilization and cognitive load in the detection of phishing emails. In AsiaUSEC’20: proceedings of the Workshop on Usable Security (pp. 1-10). Malaysia: Springer.

Nasser, G., Morrison, B. W., Bayl-Smith, P., Taib, R., Gayed, M., & Wiggins, M. W. (2020b). The Role of Cue Utilization and Cognitive Load in the Recognition of Phishing Emails. Frontiers in big data, 3, 546860.

Parsons, K., Butavicius, M., Pattinson, M., McCormac, A., Calic, D., & Jerram, C. (2016). Do users focus on the correct cues to differentiate between phishing and genuine emails? arXiv preprint arXiv:1605.04717.

Qualtrics (Version 2019). [Web-based software]. Provo, UT: Qualtrics. Available from

Shanteau, J., Weiss, D. J., Thomas, R. P., & Pounds, J. C. (2002). Performance-based assessment of expertise: How to decide if someone is an expert or not. European Journal of Operational Research, 136(2), 253-263. https://doi:10.1016/S0377-2217(01)00113-8

Shekh, S., Auton, J. C., & Wiggins, M. W. (2018). The effects of cue utilization and target-related information on target detection during a simulated drone search and rescue task. Proceedings of the Human Factor and Ergonomics Society Annual Meeting, 62(1), 227-231. https://doi:10.1177/1541931218621053

Shonman, M., Li, X., Zhang, H., & Dahbura, A. (2018). Simulating phishing email processing with instance-based learning and cognitive chunk activation. In S. Wang, V. Yamamoto, J. Su, Y. Yang, E. Jones, L Iasemidis & T. Mitchell (Eds.), Lecture Notes in Computer Science: Vol 11309. Brain Informatics (pp. 468-478). Cham: Springer. https://doi:10.1007/978-3-030-05587-5_44

Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23(5), 645-665. https://doi:10.1017/


Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3(3), 430-454.

Tversky, A., & Kahneman, D. (1974). Heuristics and biases: Judgement under uncertainty. Science, 185(1974), 1124-1130. https://doi:10.1126/science.185.4157.1124

Tversky A., & Kahneman D. (1975) Judgment under Uncertainty: Heuristics and Biases. In: Wendt D., Vlek C. (eds) Utility, Probability, and Human Decision Making. Theory and Decision Library (An International Series in the Philosophy and Methodology of the Social and Behavioral Sciences), vol 11. Springer, Dordrecht.

Vishwanath, A. (2015). Habitual Facebook Use and its Impact on Getting Deceived on Social Media. Journal of Computer-Mediated Communication, 20(1), 83-98

Vishwanath, A., Harrison, B., & Ng, Y.J. (2016). Suspicion, cognition, and automaticity model of phishing susceptibility. Commun. Res. 1–21.

Vishwanath, A., Harrison, B., & Ng, Y. J. (2018). Suspicion, Cognition, and Automaticity Model of Phishing Susceptibility. Communication Research, 45(8), 1146–1166.


Vishwanath, A., Herath, T., Chen, R., Wang, J., & Rao, H.R. (2011), “Why do people get phished? Testing individual differences in phishing vulnerability within an integrated information processing model”. Decision Support Systems, 51(3), 576-586.

Walczyk, J. J., Mahoney, K. T., Doverspike, D., & Griffith-Ross, D. A. (2009). Cognitive lie detection: Response time and consistency of answers as cues to deception. Journal of Business and Psychology, 24, 33-49. https://doi:10.1007/s10869-009-9090-8

Watkinson, J., Bristow, G., Auton, J., McMahon, C. M., & Wiggins, M. W. (2018). Postgraduate training in audiology improves clinicians’ audiology-related cue utilisation. International Journal of Audiology, 57(9), 681-687. https://doi:10.1080/14992027.2018.1476782

Weiss, D. J., & Shanteau, J. (2003). Empirical assessment of expertise. Human Factors: The Journal of the Human Factors and Ergonomics Society, 45(1), 104-116. https://doi:10.1518/hfes.45.1.


Wickens, C. D., Hollands, J. G., Banbury, S., & Parasuraman, R. (2013). Engineering psychology and human performance. New York: Psychology Press. https://doi:10.4324/9781315665177

Wiggins, M. W. (2021). A behaviour-based approach to the assessment of cue utilisation: implications for situation assessment and performance. Theoretical Issues in Ergonomics Science, 22(1), 46-62.

Wiggins, M. W. (2016). Expertise and cognitive skills development for ab-initio pilots. In R. A. Telfer & P. J. Moore (Eds.), Aviation training: Learners, instruction and organization (pp 54-66). Abington, Oxon: Routledge.

Wiggins, M. W., Brouwers, S., Davies, J., & Loveday, T. (2014). Trait-based cue utilization and initial skill acquisition: implications for models of the progression to expertise. Frontiers in Psychology, 5, 541. https://doi:10.3389/fpsyg.2014.00541

Wiggins, M. W., Crane, M., & Loveday, T. (2018). Cue utilization, perceptions, and experience in the interpretation of weather radar returns. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 62(1), 721-725. https://doi:10.1177/1541931218621164

Wiggins, M, W., Griffins, B., & Brouwers, S. (2019). The potential role of context-related exposure in explaining differences in water safety cue utilization. Human Factors: The Journal of the Human Factors and Ergonomics Society, 61(5), 825-838. https://doi:10.1177/

Wiggins, M., Loveday, T., & Lyons, L. (2014). Cues and cue-based processing: Implications for system safety. Procedia Engineering, 84, 55-61. https://doi:10.1016/j.proeng.2014.10.409

Wiggins, M. W., Whincup, E., & Auton, J. C. (2018). Cue utilisation reduces effort but increases arousal during a process control task. Applied Ergonomics, 69, 120-127. https://doi:10.1016/


Williams, E. J., Hinds, J., & Joinson, A. N. (2018). Exploring susceptibility to phishing in the workplace. International Journal of Human-Computer Studies, 120, 1-13. https://doi:10.1016/


Yan, Z., & Gozu, H. Y. (2012). Online decision-making in receiving spam emails among college students. International Journal of Cyber Behavior, Psychology and Learning, 2(1), 1-12. https://doi:10.4018/ijcbpl.2012010101

How to Cite
Ackerley, M., Morrison, B., Ingrey, K., Wiggins, M., Bayl-Smith, P., & Morrison, N. (2022). Errors, Irregularities, and Misdirection: Cue Utilisation and Cognitive Reflection in the Diagnosis of Phishing Emails. Australasian Journal of Information Systems, 26.
Research Articles