[11] Artificial Intelligence Shows Human-like Lens Capability in Subjectivity: Evaluating Chinese Pragmatic Capability of GPT-4 via Microsoft Copilot with Direct Prompting

Back to Exploring AI in Applied Linguistics

Artificial Intelligence Shows Human-like Lens Capability in Subjectivity: Evaluating Chinese Pragmatic Capability of GPT-4 via Microsoft Copilot with Direct Prompting

This chapter is part of: Chapelle, C. A., Beckett, G. H., & Ranalli, J. (Eds.). (2024). Exploring artificial intelligence in applied linguistics. Iowa State University Digital Press. https://doi.org/10.31274/isudp.2024.154.

Download Chapter

Description
Communication with humans requires the ability to decode and convey “lenses”—subjective evaluations of reality—because humans make sense of the world through lenses and convey them through language. Can Artificial Intelligence decode and convey subjectivity in lenses? Exploring this uncharted area of pragmatic capability sheds light on AI’s linguistic capabilities. This study provides an initial linguistic evaluation of the pragmatic capability of Large Language Models (LLMs) by evaluating GPT-4 through Microsoft Copilot in expressing, comprehending, and explaining subjective lenses in Chinese, using zero-shot, single-turn direct prompting. We qualitatively evaluated 51,168 AI-generated words using the Nursing-Home Test repeated over six months. Findings indicate that Microsoft Copilot exhibits human-like pragmatic capability in lens-related subjectivity, but specific abilities vary and are prompt sensitive. Production matched “Language-Teacher.” Comprehension matched “High-Schooler.” Knowledge matched “Elementary-Schooler” (pragmalinguistic) and “College-Student” (sociopragmatic) with general prompts and “Language-Teacher” with academic prompts. These findings highlight GPT-4’s potential in language education, emphasizing the importance of prompt design and qualitative linguistic evaluation of AI language.

Description

Communication with humans requires the ability to decode and convey “lenses”—subjective evaluations of reality—because humans make sense of the world through lenses and convey them through language. Can Artificial Intelligence decode and convey subjectivity in lenses? Exploring this uncharted area of pragmatic capability sheds light on AI’s linguistic capabilities. This study provides an initial linguistic evaluation of the pragmatic capability of Large Language Models (LLMs) by evaluating GPT-4 through Microsoft Copilot in expressing, comprehending, and explaining subjective lenses in Chinese, using zero-shot, single-turn direct prompting. We qualitatively evaluated 51,168 AI-generated words using the Nursing-Home Test repeated over six months. Findings indicate that Microsoft Copilot exhibits human-like pragmatic capability in lens-related subjectivity, but specific abilities vary and are prompt sensitive. Production matched “Language-Teacher.” Comprehension matched “High-Schooler.” Knowledge matched “Elementary-Schooler” (pragmalinguistic) and “College-Student” (sociopragmatic) with general prompts and “Language-Teacher” with academic prompts. These findings highlight GPT-4’s potential in language education, emphasizing the importance of prompt design and qualitative linguistic evaluation of AI language.

Details

Published	Published By	Pages	DOI
July 31, 2024	Iowa State University Digital Press	27	10.31274/isudp.2024.154.11
License Information
©2024 The authors. Published under a CC BY license.
Citation
Su, D. & Goslar, K. (2024). Artificial intelligence shows human-like lens capability in subjectivity: evaluating Chinese pragmatic capability of GPT-4 via Microsoft Copilot with direct prompting. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 175–201). Iowa State University Digital Press. https://doi.org/10.31274/isudp.2024.154.11.