Errors, Irregularities, and Misdirection: Cue Utilisation and Cognitive Reflection in the Diagnosis of Phishing Emails

  • Mitchell Ackerley
  • Ben Morrison Macquarie University
  • Kate Ingrey
  • Mark Wiggins
  • Piers Bayl-Smith
  • Natalie Morrison
Keywords: cyber security, phishing, cue utilisation, cognitive reflection, expertise

Abstract

The study aimed to examine the role of, and potential interplay between, cue utilisation and cognitive reflection in email users’ ability to accurately (and efficiently) differentiate between phishing and genuine emails. 145 participants completed the Cognitive Reflection Test (CRT), a phishing diagnostic task, and the Expert Intensive Skill Evaluation (EXPERTise 2.0) battery, which provided a gauge of users’ cue utilisation in the domain. The results revealed an interaction between users’ cognitive utilisation and cue reflection, whereby users low in both facets performed significantly worse in diagnosing phishing emails than all other groups. Further, those participants with both higher cue utilisation and cognitive reflection took significantly longer to make their diagnosis. It is concluded that a high level of cognitive reflection was able to compensate for a lower level of cue utilisation, and vice versa. Participants reported using seven types of cue during diagnosis, however, there was no significant relationship between the types of cues used and users’ level of cue utilisation. Taken together, the findings have implications to the design of user-level interventions in relation to the identification of vulnerable users, as well as the need to consider training approaches that extend beyond the use of simple cue inventories.

References

Anderson, J. R. (1996). ACT: A simple theory of complex cognition. American psychologist, 51(4), 355-365. https://doi:10.1037/0003-066X.51.4.355

Basnet R.B., Sung A.H., & Liu Q. (2012) Feature Selection for Improved Phishing Detection. In: Jiang H., Ding W., Ali M., & Wu X. (eds) Advanced Research in Applied Artificial Intelligence. IEA/AIE 2012. Lecture Notes in Computer Science, vol 7345. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-310

Bayl-Smith, P., Sturman, D., & Wiggins, M. (2020). Cue utilization, phishing feature and phishing email detection. In M. Bernhard, A. Bracciali, L. J. Camp, S. Matsuo, A. Maurushat, P. B. Rønne, & M. Sala (Eds.), Financial Cryptography and Data Security: FC 2020 International Workshops, AsiaUSEC, CoDeFi, VOTING, and WTSC, Revised Selected Papers (pp. 56-70). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12063). Springer. https://doi.org/10.1007/978-3-030-54455-3_5

Bengtsson, M. (2016). How to plan and perform a qualitative study using content analysis. Nursing Plus Open, 2, 8-14. https/doi:10.1016/j.npls.2016.01.001

Brams, S., Ziv, G., Levin, O., Spitz, J., Wagemans, J., Williams, A. M., & Helsen, W. F. (2019). The relationship between gaze behavior, expertise, and performance: A systematic review. Psychological Bulletin, 145(10), 980–1027. https://doi.org/10.1037/bul0000207

Brouwers, S., Wiggins, M., & Griffin, B. (2018). Operators who readily acquire patterns and cues, risk being miscued in routinized settings. Journal of Experimental Psychology: Applied, 24(2), 261-274. https://doi:10.1037/xap0000151

Butavicius, M., Parsons, K., Pattison M., & McCormac, A. (2016). Breaching the Human Firewall: Social engineering inPphishing and Spear-Phishing Emails. ArXiv, abs/1606.00887.

Canfield, C. I., Fischhoff, B., & Davis, A. (2016). Quantifying Phishing Susceptibility for Detection and Behavior Decisions. Human Factors: The Journal of the Human Factors and Ergonomics Society, 58(8), 1158–1172. https://doi.org/10.1177/0018720816665025

deGroot, A. D. (1978). Thought and choice in chess. The Hague: Mouton

Downs, J. S., Holbrook, M. B., & Cranor, L. F. (2006). Decision strategies and susceptibility to phishing. In Proceedings of the second symposium on Usable privacy and security (pp. 79-90). ACM. https://doi:10.1145/1143120.1143131

Fette, I., Sadeh, N., & Tomasic, A. (2007). Learning to detect phishing emails. In WWW ‘07 Proceedings of the 16th international conference on World Wide Web (pp. 649-656). New York: ACM. https://doi:10.21236/ada456046

Field, A. (2013). Discovering statistics using IBM SPSS statistics. London, England: Sage.

Fleiss, J. L., Levin, B., & Paik, M. C. (2013). Statistical methods for rates and proportions. Hoboken: John Wiley & Sons.

Frauenstein, E. D., & Flowerday, S. (2020). Susceptibility to phishing on social network sites: A personality information processing model. Computers & Security, 94, 101862. https://doi.org/10.1016/j.cose.2020.101862

Frederick, S. (2005). Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 19(4), 25-42. https://doi.org/10.1257/089533005775196732

French, K. E., & Nevett, M. E. (1993). The Development of Expertise in Youth Sport. In J. L. Starkes & F. Allard (Eds.), Advances in Psychology (pp. 255-270). North-Holland. https://doi.org/https://doi.org/10.1016/S0166-4115(08)61475-2

Gacasan, E. M. P., Wiggins, M. W., & Searle, B. J. (2016). The role of cues in expert project manager sensemaking. Construction Management and Economics, 34(7-8), 492-507. https://doi:10.1080/01446193.2016.1177190

Galanter, C. A., & Patel, V. L. (2005) Medical decision making: A selective review for child psychiatrists and psychologists. Journal of Child Psychology and Psychiatry, 46(7), 675-689. https://doi:10.1111/j.1469-7610.2005.01452.x

Gonzalez, C. (2013). The boundaries of instance-based learning theory for explaining decisions from experience. In V. S. C. Pammi & N. Srinivasan (Eds.), Progress in brain research (Vol. 202, pp. 73-98). Oxford, UK: Elsevier.

Gwet, K. (2001). Handbook of inter-rater reliability. Gaithersburg: STATAXIS Publishing Company.

Harrison, B., Svetieva, E., & Vishwanath, A. (2016). Individual processing of phishing emails. Online Information Review, 40(2), 265–281. https://doi.org/10.1108/oir-04-2015-0106

Harré, M., Bossomaier, T., & Snyder, A. (2012). The perceptual cues that reshape expert reasoning. Scientific Reports, 2(1), 502–502. https://doi.org/10.1038/srep00502

Hassandoust, F., Singh, H., & Williams, J. (2020). The Role of Contextualization in Individuals’ Vulnerability to Phishing Attempts. Australasian Journal of Information Systems, 24. DOI: https://doi.org/10.3127/ajis.v24i0.2693

Johnston, D., & Morrison, B. W. (2016). The application of naturalistic decision-making techniques to explore cue use in rugby league playmakers. Journal of Cognitive Engineering and Decision Making, 10(4), 391-410. https://doi:10.1177/1555343416662181

Jones, H.S., Towse, J. N., & Race, N. (2015) Susceptibility to email fraud: A review of psychological perspectives, data-collection methods, and ethical considerations. International Journal of Cyber Behaviour, Psychology and Learning, 5(3). 13-29. https://doi:10.4018/IJCBPL.2015070102

Jones, H.S., Towse, J. N., Race, N., & Harrison, T. (2019). Email fraud: The search for psychological predictors of susceptibility. PloS ONE 14(1), e0209684. https://doi:10.1371/journal.pone.0209684

Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American psychologist, 64(6), 515-526. https://doi:10.1037/a0016755

Klein, G. A. (2008). Naturalistic Decision Making. Human Factors, 50(3), 456–460. https://doi.org/10.1518/001872008X288385

Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid Decision making on the Fire Ground. Proceedings of the Human Factors Society Annual Meeting, 30(6), 576-580.

Kobus, D. A., Proctor, S., & Bank, T. E. (2000). Decision-making in a dynamic environment: the effects of experience and information uncertainty. Technical Report 1832. San Diego, CA: Spawar Systems Center.

Levine, T. R. (2014). Truth-Default Theory (TDT). Journal of Language and Social Psychology, 33(4), 378–392. https://doi:10.1177/0261927x14535916

Loveday, T., Wiggins, M., Festa, M., Schell D., & Twigg, D. (2013). Pattern recognition as an indicator of diagnostic expertise. In C. P. Latorre & F. A. Sanchez (Eds.), Pattern recognition – Applications and methods (pp. 1-11). Berlin: Springer.

Luo, X., Zhang, W., Burd, S., & Seazzu, A. (2013). Investigating phishing victimization with the Heuristic-systematic model: a theoretical framework and an exploration. Computer Security, 38, 28–38. https://doi.org/10.1016/j.cose.2012.12.003

Moghimi, M., & Varjani, A. Y. (2016). New rule-based phishing detection method. Expert Systems with Applications, 53, 231–242. https://doi:10.1016/j.eswa.2016.01.028

Morrison, B. W., Johnston, D., Naylor, M., Morrison, N. M. V., & Forrest, D. (2020). “You can’t hide your lyin’ eyes”: investigating the relationship between associative learning, cue awareness, and decision performance in detecting lies. Journal of Cognitive Engineering and Decision Making, 14(2), 99-111. https://doi.org/10.1177/1555343420918084

Morrison, B., & Morrison, N. (2015). Diagnostic cues in major crime investigation. In M. W. Wiggins, & T. Loveday (Eds.), Diagnostic expertise in organizational environments (pp. 91-98). Ashgate Publishing

Morrison, B. M., Wiggins, M. W., Bond N. W., & Tyler, M. D. (2013). Measuring relative cue strength as a means of validating an inventory of expert offender profiling cues. Journal of Cognitive Engineering and Decision Making, 7(2), 211-226. https://doi:1177/

Morrison, B. W., Wiggins , M. W., & Morrison, N. V. (2018). Utility of expert cue exposure as a mechanism to improve decision-making performance among novice criminal investigators. Journal of Cognitive Engineering and Decision Making, 12(2), 99-111. https://doi:10.1177/1555343417746570

Nasser, G., Morrison, B. W., Bayl-Smith, P., Taib, R., Gayed, M., & Wiggins, M. W. (2020a). The effects of cue utilization and cognitive load in the detection of phishing emails. In AsiaUSEC’20: proceedings of the Workshop on Usable Security (pp. 1-10). Malaysia: Springer.

Nasser, G., Morrison, B. W., Bayl-Smith, P., Taib, R., Gayed, M., & Wiggins, M. W. (2020b). The Role of Cue Utilization and Cognitive Load in the Recognition of Phishing Emails. Frontiers in big data, 3, 546860. https://doi.org/10.3389/fdata.2020.546860

Parsons, K., Butavicius, M., Pattinson, M., McCormac, A., Calic, D., & Jerram, C. (2016). Do users focus on the correct cues to differentiate between phishing and genuine emails? arXiv preprint arXiv:1605.04717.

Qualtrics (Version 2019). [Web-based software]. Provo, UT: Qualtrics. Available from http://www.qualtrics.com.

Shanteau, J., Weiss, D. J., Thomas, R. P., & Pounds, J. C. (2002). Performance-based assessment of expertise: How to decide if someone is an expert or not. European Journal of Operational Research, 136(2), 253-263. https://doi:10.1016/S0377-2217(01)00113-8

Shekh, S., Auton, J. C., & Wiggins, M. W. (2018). The effects of cue utilization and target-related information on target detection during a simulated drone search and rescue task. Proceedings of the Human Factor and Ergonomics Society Annual Meeting, 62(1), 227-231. https://doi:10.1177/1541931218621053

Shonman, M., Li, X., Zhang, H., & Dahbura, A. (2018). Simulating phishing email processing with instance-based learning and cognitive chunk activation. In S. Wang, V. Yamamoto, J. Su, Y. Yang, E. Jones, L Iasemidis & T. Mitchell (Eds.), Lecture Notes in Computer Science: Vol 11309. Brain Informatics (pp. 468-478). Cham: Springer. https://doi:10.1007/978-3-030-05587-5_44

Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23(5), 645-665. https://doi:10.1017/

S0140525X00003435

Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3(3), 430-454. https://doi.org/10.1016/0010-0285(72)90016-3.

Tversky, A., & Kahneman, D. (1974). Heuristics and biases: Judgement under uncertainty. Science, 185(1974), 1124-1130. https://doi:10.1126/science.185.4157.1124

Tversky A., & Kahneman D. (1975) Judgment under Uncertainty: Heuristics and Biases. In: Wendt D., Vlek C. (eds) Utility, Probability, and Human Decision Making. Theory and Decision Library (An International Series in the Philosophy and Methodology of the Social and Behavioral Sciences), vol 11. Springer, Dordrecht. https://doi.org/10.1007/978-94-010-1834-0_8

Vishwanath, A. (2015). Habitual Facebook Use and its Impact on Getting Deceived on Social Media. Journal of Computer-Mediated Communication, 20(1), 83-98

Vishwanath, A., Harrison, B., & Ng, Y.J. (2016). Suspicion, cognition, and automaticity model of phishing susceptibility. Commun. Res. 1–21. https://doi.org/10.1177/0093650215627483

Vishwanath, A., Harrison, B., & Ng, Y. J. (2018). Suspicion, Cognition, and Automaticity Model of Phishing Susceptibility. Communication Research, 45(8), 1146–1166. https://doi.org/

1177/0093650215627483

Vishwanath, A., Herath, T., Chen, R., Wang, J., & Rao, H.R. (2011), “Why do people get phished? Testing individual differences in phishing vulnerability within an integrated information processing model”. Decision Support Systems, 51(3), 576-586.

Walczyk, J. J., Mahoney, K. T., Doverspike, D., & Griffith-Ross, D. A. (2009). Cognitive lie detection: Response time and consistency of answers as cues to deception. Journal of Business and Psychology, 24, 33-49. https://doi:10.1007/s10869-009-9090-8

Watkinson, J., Bristow, G., Auton, J., McMahon, C. M., & Wiggins, M. W. (2018). Postgraduate training in audiology improves clinicians’ audiology-related cue utilisation. International Journal of Audiology, 57(9), 681-687. https://doi:10.1080/14992027.2018.1476782

Weiss, D. J., & Shanteau, J. (2003). Empirical assessment of expertise. Human Factors: The Journal of the Human Factors and Ergonomics Society, 45(1), 104-116. https://doi:10.1518/hfes.45.1.

27233

Wickens, C. D., Hollands, J. G., Banbury, S., & Parasuraman, R. (2013). Engineering psychology and human performance. New York: Psychology Press. https://doi:10.4324/9781315665177

Wiggins, M. W. (2021). A behaviour-based approach to the assessment of cue utilisation: implications for situation assessment and performance. Theoretical Issues in Ergonomics Science, 22(1), 46-62. https://doi.org/10.1080/1463922X.2020.1758828

Wiggins, M. W. (2016). Expertise and cognitive skills development for ab-initio pilots. In R. A. Telfer & P. J. Moore (Eds.), Aviation training: Learners, instruction and organization (pp 54-66). Abington, Oxon: Routledge.

Wiggins, M. W., Brouwers, S., Davies, J., & Loveday, T. (2014). Trait-based cue utilization and initial skill acquisition: implications for models of the progression to expertise. Frontiers in Psychology, 5, 541. https://doi:10.3389/fpsyg.2014.00541

Wiggins, M. W., Crane, M., & Loveday, T. (2018). Cue utilization, perceptions, and experience in the interpretation of weather radar returns. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 62(1), 721-725. https://doi:10.1177/1541931218621164

Wiggins, M, W., Griffins, B., & Brouwers, S. (2019). The potential role of context-related exposure in explaining differences in water safety cue utilization. Human Factors: The Journal of the Human Factors and Ergonomics Society, 61(5), 825-838. https://doi:10.1177/

Wiggins, M., Loveday, T., & Lyons, L. (2014). Cues and cue-based processing: Implications for system safety. Procedia Engineering, 84, 55-61. https://doi:10.1016/j.proeng.2014.10.409

Wiggins, M. W., Whincup, E., & Auton, J. C. (2018). Cue utilisation reduces effort but increases arousal during a process control task. Applied Ergonomics, 69, 120-127. https://doi:10.1016/

j.apergo.2018.01.012

Williams, E. J., Hinds, J., & Joinson, A. N. (2018). Exploring susceptibility to phishing in the workplace. International Journal of Human-Computer Studies, 120, 1-13. https://doi:10.1016/

j.ijhcs.2018.06.004

Yan, Z., & Gozu, H. Y. (2012). Online decision-making in receiving spam emails among college students. International Journal of Cyber Behavior, Psychology and Learning, 2(1), 1-12. https://doi:10.4018/ijcbpl.2012010101

Published
2022-05-01
How to Cite
Ackerley, M., Morrison, B., Ingrey, K., Wiggins, M., Bayl-Smith, P., & Morrison, N. (2022). Errors, Irregularities, and Misdirection: Cue Utilisation and Cognitive Reflection in the Diagnosis of Phishing Emails. Australasian Journal of Information Systems, 26. https://doi.org/10.3127/ajis.v26i0.3615
Section
Research Articles