Loading...
|
Please use this identifier to cite or link to this item:
https://nccur.lib.nccu.edu.tw/handle/140.119/110147
|
| Title: | 聽其所見，觀其所聞：以眼動探討口語與場景互動的跨感官理解歷程
Look while listeng : using eye movements to investigate the interaction between spoken language and visual scene during cross-modal comprehension |
| Authors: | 游婉雲
Yu, Wan-Yun |
| Contributors: | 蔡介立
Tsai, Jie-Li
游婉雲
Yu, Wan-Yun |
| Keywords: | 視聽整合
理解
場景一致性
口語
眼動
Audio-visual integration
Comprehension
Scene consistency
Spoken language
Eye movement |
| Date: | 2017 |
| Issue Date: | 2017-06-01 09:53:01 (UTC+8) |
| Abstract: | 在人類溝通及語言使用行為中，口語和場景是構成人類跨感官理解歷程的兩項重要成分。究竟兩類資訊如何共同改變理解歷程仍待檢驗。本論文旨在探問四項研究問題：一，過去文獻對理解期間的視覺注意力運作提出兩類觀點。階層取徑主張口語優先並決定視覺表徵的處理，互動取徑則認為口語和視覺表徵可獨立影響視覺注意力。二，口語可促進指涉物體的凝視行為，然口語指涉效果是否受作業目標影響的本質仍不清楚。三，以複雜場景作為視覺情境，探討視覺複雜性和語義一致性表徵如何影響理解歷程。四，檢驗視覺刺激的預覽時間如何改變口語和場景表徵因素對理解歷程的影響。
本論文透過一系列視覺情境典範實驗探討以上研究問題。在每ㄧ項嘗試次中，參與者在聆聽中文語句期間同時觀看包含包含兩項物體的圖片：一為鑲嵌在一致（例如：原野）、不一致（例如：天空）和空白背景的口語指涉目標物體（例如：老虎），另一項則為口語未指涉且與背景一致的非目標物體（例如：禿鷹）。其次，四項實驗直交地操弄「作業目標」（「口語理解作業」或「場景理解作業」）和「預覽時間」（「一秒預覽」或「無預覽」）因素。
實驗結果發現：一，無論作業目標為何，所有實驗皆出現穩定的口語指涉效果。二，場景的視覺複雜性和語義一致性表徵不僅可獨立引導物體凝視行為，也可和口語共同決定理解期間的視覺注意力運作。三，作業目標對口語指涉效果及場景一致性效果產生差異化的調節作用。四，預覽時間有效促進口語理解作業的口語指涉效果，場景理解作業則不受影響。
整體而言，本論文的實驗證據支持互動取徑觀點。換言之，在跨感官理解的過程中，人類認知運作可透過協調語言、視覺和記憶等次系統，快速整合口語和場景所提供的物理和語義表徵，並依據當下情境動態地改變人類對外在世界的感官經驗。
In human communication and language use, both speech and scene constitute the cross-modal comprehension process. However, how these two elements combine to affect human comprehension process has not yet been fully resolved. Four research questions will be examined. First, two approaches can account for the comprehension process: the hierarchical approach asserts speech plays the main part whereas the visual feature has only a supporting role, while the interactive approach states that
both speech and visual feature combine to determine the comprehension process. Second, despite the speech can cause the spoken reference effect on having more fixations on its visual referent, the nature of this effect is still unclear. Third, most past studies adopted simple object array as visual context, little is known about the impact of real world scenes on the comprehension process. Fourth, whether the preview time could alter the influence of speech and scene on comprehension will be tested.
A series of visual world paradigm experiments were conducted. Factors of task demand (speech comprehension vs. scene comprehension) and preview time (1-second vs. none) were orthogonally manipulated in four experiments. In each trial, participants listened to a spoken sentence in Chinese while viewing a picture with two critical objects: one is the mentioned target object (e.g., tiger), which was embedded in either a consistent, inconsistent or blank background; the other is an unmentioned non-target object (e.g., eagle) that was always consistent with its background.
Several findings were found. First, the reliable spoken reference effect were shown regardless of the task demand was given. Second, the visual complexity and scene consistency not only can individually guide fixations on objects, but can work together with the speech to determine the visual attention during comprehension. Third, task demand could differently modulate the spoken reference and scene consistency effect, respectively. Fourth, preview time significantly enhances the spoken reference effect in the speech comprehension task, whereas no impact was observed in the scene comprehension task. These evidence supported the view of interactive approach. In conclusion, human’s different cognitive systems, including language, vision and memory, can interact with each other and cause the moment to moment experience of how we understand the complex world around us. |
| Reference: | Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of memory and language, 38(4), 419-439.
Altmann, G. T. (2004). Language-mediated eye movements in the absence of a visual world: The ‘blank screen paradigm’. Cognition, 93(2), B79-B87.
Altmann, G. T. (2011a). Language can mediate eye movement control within 100milliseconds, regardless of whether there is anything to move the eyes to. Acta psychologica, 137(2), 190-200.
Altmann, G. T. (2011b). The mediation of eye movements by spoken language. In S. Liversedge, I. Gilchrist, & S. Everling (Eds.), The Oxford Handbook of Eye Movements (pp. 979-1003). New York: Oxford University Press.
Altmann, G. T., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73(3), 247-264.
Altmann, G. T., & Kamide, Y. (2007). The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of memory and language, 57(4), 502-518.
Altmann, G. T., & Kamide, Y. (2009). Discourse-mediation of the mapping between language and the visual world: Eye movements and mental representation. Cognition, 111(1), 55-71.
Altmann, G. T., & Mirković, J. (2009). Incrementality and prediction in human sentence processing. Cognitive Science, 33(4), 583-609.
Anderson, S. E., Chiu, E., Huette, S., & Spivey, M. J. (2011). On the temporal dynamics of language-mediated vision and vision-mediated language. Acta psychologica, 137(2), 181-189.
Andersson, R., Ferreira, F., & Henderson, J. M. (2011). I see what you`re saying: The integration of complex speech and scenes during language comprehension. Acta psychologica, 137(2), 208-216.
Arai, M., & Keller, F. (2013). The use of verb-specific information for prediction in sentence processing. Language and Cognitive Processes, 28(4), 525-560.
Baddeley, A. (1996). Exploring the central executive. The Quarterly Journal of Experimental Psychology: Section A, 49(1), 5-28.
Baddeley, A. (2000). The episodic buffer: a new component of working memory? Trends in cognitive sciences, 4(11), 417-423.
Bar, M. (2004). Visual objects in context. Nature Reviews Neuroscience, 5(8), 617-629.
Bar, M., Kassam, K. S., Ghuman, A. S., Boshyan, J., Schmid, A. M., Dale, A. M., . . . Rosen, B. R. (2006). Top-down facilitation of visual recognition. Proceedings of the National Academy of Sciences of the United States of America, 103(2), 449-454.
Biederman, I., Mezzanotte, R. J., & Rabinowitz, J. C. (1982). Scene perception: Detecting and judging objects undergoing relational violations. Cognitive Psychology, 14(2), 143-177.
Boersma, P., & Weenink, D. (2002). Praat: doing phonetics by computer (Version Version 4.0.26). Retrieved from http://www.praat.org
Bonitz, V. S., & Gordon, R. D. (2008). Attention to smoking-related and incongruous objects during scene viewing. Acta psychologica, 129(2), 255-263.
Boyce, S. J., Pollatsek, A., & Rayner, K. (1989). Effect of background information on object identification. Journal of Experimental Psychology: Human Perception and Performance, 15(3), 556.
Castelhano, M. S., & Henderson, J. M. (2008). The influence of color on the perception of scene gist. Journal of Experimental Psychology: Human Perception and Performance, 34(3), 660.
Cavicchio, F., Melcher, D., & Poesio, M. (2014). The effect of linguistic and visual salience in visual world studies. Frontiers in Psychology, 5.
Chambers, C. G., Tanenhaus, M. K., & Magnuson, J. S. (2004). Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30(3), 687.
Chen, Q., & Mirman, D. (2015). Interaction Between Phonological and Semantic Representations: Time Matters. Cognitive Science, 39(3), 538-558.
Chun, M. M., & Jiang, Y. (1998). Contextual cueing: Implicit learning and memory of visual context guides spatial attention. Cognitive Psychology, 36(1), 28-71.
Clarke, A. D., Coco, M. I., & Keller, F. (2013). The impact of attentional, linguistic, and visual features during object naming. Frontiers in psychology, 4.
Coco, M. I., & Keller, F. (2015). The interaction of visual and linguistic saliency during syntactic ambiguity resolution. The Quarterly Journal of Experimental Psychology, 68(1), 46-74.
Coco, M. I., Malcolm, G. L., & Keller, F. (2014). The interplay of bottom-up and top-down mechanisms in visual guidance during object naming. The Quarterly Journal of Experimental Psychology, 67(6), 1096-1120.
Cooper, R. M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6(1), 84-107.
Crocker, M. W., Knoeferle, P., & Mayberry, M. R. (2010). Situated sentence processing: The coordinated interplay account and a neurobehavioral model. Brain and language, 112(3), 189-201.
Dahan, D., & Tanenhaus, M. K. (2005). Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychonomic bulletin & review, 12(3), 453-459.
Davenport, J. L., & Potter, M. C. (2004). Scene consistency in object and background perception. Psychological Science, 15(8), 559-564.
De Graef, P., Christiaens, D., & d`Ydewalle, G. (1990). Perceptual effects of scene context on object identification. Psychological research, 52(4), 317-329.
De Groot, F., Huettig, F., & Olivers, C. N. (2015). When Meaning Matters: The Temporal Dynamics of Semantic Influences on Visual Attention.
Elman, J. L., & McClelland, J. L. (1983). Speech Perception as a Cognitive Process: The Interactive Activation Model. Retrieved from
Ferreira, F., Foucart, A., & Engelhardt, P. E. (2013). Language processing in the visual world: Effects of preview, visual complexity, and prediction. Journal of memory and language, 69(3), 165-182.
Friedman, A. (1979). Framing pictures: the role of knowledge in automatized encoding and memory for gist. Journal of experimental psychology: General, 108(3), 316.
Gordon, R. D. (2004). Attentional allocation during the perception of scenes. Journal of Experimental Psychology: Human Perception and Performance, 30(4), 760.
Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological review, 101(3), 371-395.
Henderson, J. M. (2006). Eye movements. In T. R. C. Senior & M. Gazzaniga (Eds.), Methods in Mind (pp. 171-191). Cambridge, MA: MIT Press.
Henderson, J. M. (2011). Eye movements and scene perception. In S. Liversedge, I. D. Gilchrist, & S. Everling (Eds.), Oxford Handbook of Eye Movements (pp. 593-606). Oxford: Oxford University Press.
Henderson, J. M., & Ferreira, F. (2004). Scene Perception for Psycholinguists. In J. M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world (pp. 1-58). New York: Psychology Press.
Henderson, J. M., & Hollingworth, A. (1999a). High-level scene perception. Annual review of psychology, 50(1), 243-271.
Henderson, J. M., & Hollingworth, A. (1999b). The role of fixation position in detecting scene changes across saccades. Psychological Science, 10(5), 438-443.
Henderson, J. M., & Hollingworth, A. (2003). Eye movements and visual memory: Detecting changes to saccade targets in scenes. Perception & Psychophysics, 65(1), 58-71.
Henderson, J. M., Weeks Jr, P. A., & Hollingworth, A. (1999). The effects of semantic consistency on eye movements during complex scene viewing. Journal of Experimental Psychology: Human Perception and Performance, 25(1), 210.
Hintz, F., & Huettig, F. (2015). The complexity of the visual environment modulates language-mediated eye gaze. In R. K. Mishra, N. Srinivasan, & F. Huettig (Eds.), Attention and vision in language processing (pp. 39-55). Berlin: Springer.
Huettig, F., & Altmann, G. T. (2005). Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition, 96(1), B23-B32.
Huettig, F., & Altmann, G. T. (2007). Visual-shape competition during language-mediated attention is based on lexical input and not modulated by contextual appropriateness. Visual Cognition, 15(8), 985-1018.
Huettig, F., & Altmann, G. T. (2011). Looking at anything that is green when hearing “frog”: How object surface colour and stored object colour knowledge influence language-mediated overt attention. The Quarterly Journal of Experimental Psychology, 64(1), 122-145.
Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of memory and language, 57(4), 460-482.
Huettig, F., Mishra, R. K., & Olivers, C. N. (2011). Mechanisms and representations of language-mediated visual attention. Frontiers in psychology, 2.
Huettig, F., Olivers, C. N., & Hartsuiker, R. J. (2011). Looking, language, and memory: Bridging research from the visual world and visual search paradigms. Acta psychologica, 137(2), 138-150.
Huettig, F., Rommers, J., & Meyer, A. S. (2011). Using the visual world paradigm to study language processing: A review and critical evaluation. Acta psychologica, 137(2), 151-171.
Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision research, 40(10), 1489-1506.
Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness: Harvard University Press.
Joubert, O. R., Rousselet, G. A., Fize, D., & Fabre-Thorpe, M. (2007). Processing scene context: Fast categorization and object interference. Vision research, 47(26), 3286-3297.
Kahneman, D., Treisman, A., & Gibbs, B. J. (1992). The reviewing of object files: Object-specific integration of information. Cognitive Psychology, 24(2), 175-219.
Kamide, Y., Altmann, G. T., & Haywood, S. L. (2003). The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of memory and language, 49(1), 133-156.
Knoeferle, P., & Crocker, M. W. (2006). The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye tracking. Cognitive Science, 30(3), 481-529.
Knoeferle, P., & Crocker, M. W. (2007). The influence of recent scene events on spoken comprehension: Evidence from eye movements. Journal of memory and language, 57(4), 519-543.
Kukona, A., Fang, S.-Y., Aicher, K. A., Chen, H., & Magnuson, J. S. (2011). The time course of anticipatory constraint integration. Cognition, 119(1), 23-42.
Kukona, A., & Tabor, W. (2011). Impulse processing: A dynamical systems model of incremental eye movements in the visual world paradigm. Cognitive Science, 35(6), 1009-1051.
Kumar, A., & Krol, G. (1992). Binocular infrared oculography. The Laryngoscope, 102(4), 367-378.
Lavie, N. (1995). Perceptual load as a necessary condition for selective attention. Journal of Experimental Psychology: Human Perception and Performance, 21(3), 451.
Lavie, N., & Cox, S. (1997). On the efficiency of visual selective attention: Efficient visual search leads to inefficient distractor rejection. Psychological Science, 8(5), 395-396.
Lavie, N., Hirst, A., De Fockert, J. W., & Viding, E. (2004). Load theory of selective attention and cognitive control. Journal of experimental psychology: General, 133(3), 339.
Lavie, N., & Tsal, Y. (1994). Perceptual load as a major determinant of the locus of selection in visual attention. Perception & Psychophysics, 56(2), 183-197.
Li, F. F., VanRullen, R., Koch, C., & Perona, P. (2002). Rapid natural scene categorization in the near absence of attention. Proceedings of the National Academy of Sciences, 99(14), 9596-9601.
Loftus, G. R., & Mackworth, N. H. (1978). Cognitive determinants of fixation location during picture viewing. Journal of Experimental Psychology: Human Perception and Performance, 4(4), 565.
Macdonald, J. S., & Lavie, N. (2011). Visual perceptual load induces inattentional deafness. Attention, Perception, & Psychophysics, 73(6), 1780-1789.
MacDonald, M. C., Pearlmutter, N. J., & Seidenberg, M. S. (1994). The lexical nature of syntactic ambiguity resolution. Psychological review, 101(4), 676.
McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception: I. An account of basic findings. Psychological review, 88(5), 375.
McQueen, J. M., & Viebahn, M. C. (2007). Tracking recognition of spoken words by tracking looks to printed words. The Quarterly Journal of Experimental Psychology, 60(5), 661-671.
Mirman, D., Dixon, J. A., & Magnuson, J. S. (2008). Statistical and computational models of the visual world paradigm: Growth curves and individual differences. Journal of memory and language, 59(4), 475-494.
Mishra, R. K., Olivers, C. N., & Huettig, F. (2012). Spoken language and the decision to move the eyes: To what extent are language-mediated eye movements automatic? Progress in Brain Research, Vol. 202, Decision Making: Neural and Behavioural Approaches.
Moores, E., Laiti, L., & Chelazzi, L. (2003). Associative knowledge controls deployment of visual selective attention. Nature neuroscience, 6(2), 182-189.
Neider, M. B., & Zelinsky, G. J. (2006). Scene context guides eye movements during visual search. Vision research, 46(5), 614-621.
Oliva, A. (2005). Gist of the scene. Neurobiology of attention, 696(64), 251-258.
Oliva, A., & Schyns, P. G. (2000). Diagnostic colors mediate scene recognition. Cognitive Psychology, 41(2), 176-210.
Palmer, t. E. (1975). The effects of contextual scenes on the identification of objects. Memory & Cognition, 3, 519-526.
Potter, M. C. (1972). Meaning in visual search. Am. J. Physiol, 222, 503.
Potter, M. C. (1976). Short-term conceptual memory for pictures. Journal of experimental psychology: human learning and memory, 2(5), 509-522.
Potter, M. C. (1993). Very short-term conceptual memory. Memory & Cognition, 21(2), 156-161.
Potter, M. C. (2012). Conceptual short term memory in perception and thought. Frontiers in psychology, 3.
Potter, M. C., & Levy, E. I. (1969). Recognition memory for a rapid sequence of pictures. Journal of experimental psychology, 81(1), 10.
R Development Core Team. (2014). R: A language and environment for statistical computing (Version 3.1.2). Vienna, Austria. Retrieved from http://www.R-project.org/
Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological bulletin, 124(3), 372-422.
Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62(8), 1457-1506.
Rensink, R. A. (2000). The dynamic representation of scenes. Visual Cognition, 7(1-3), 17-42.
Rensink, R. A., O`Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8(5), 368-373.
Salverda, A. P., & Altmann, G. (2011). Attentional capture of objects referred to by spoken language. Journal of Experimental Psychology: Human Perception and Performance, 37(4), 1122.
Salverda, A. P., & Tanenhaus, M. K. (2010). Tracking the time course of orthographic information in spoken-word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(5), 1108.
Simons, D. J., & Levin, D. T. (1997). Change blindness. Trends in cognitive sciences, 1(7), 261-267.
Sorensen, D. W., & Bailey, K. G. D. (2007). The world is too much: Effects of array size on the link between language comprehension and eye movements. Paper presented at the The 14th annual OPAM conference, Houston, Texas.
Spotorno, S., Tatler, B. W., & Faure, S. (2013). Semantic consistency versus perceptual salience in visual scenes: Findings from change detection. Acta psychologica, 142(2), 168-176.
SR Research. (2010). Eyelink 1000 user`s manual. Mississauga, ON: SR Research Ltd.
Subramaniam, S., Biederman, I., & Madigan, S. (2000). Accurate identification but no priming and chance recognition memory for pictures in RSVP sequences. Visual Cognition, 7(4), 511-535.
Tanenhaus, M. K., Magnuson, J. S., Dahan, D., & Chambers, C. (2000). Eye movements and lexical access in spoken-language comprehension: Evaluating a linking hypothesis between fixations and linguistic processing. Journal of Psycholinguistic Research, 29(6), 557-580.
Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217), 1632-1634.
Tanenhaus, M. K., & Trueswell, J. C. (2006). Eye movements and spoken language comprehension. In M. Traxler & M. Gernsbacher (Eds.), Handbook of Psycholinguistics (2nd Edition ed., pp. 863-900). Amsterdam: Elsevier.
Theeuwes, J. (1992). Perceptual selectivity for color and form. Perception & Psychophysics, 51(6), 599-606.
Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of processing in the human visual system. Nature, 381(6582), 520-522.
Torralba, A., Oliva, A., Castelhano, M. S., & Henderson, J. M. (2006). Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological review, 113(4), 766.
Underwood, G., & Foulsham, T. (2006). Visual saliency and semantic incongruency influence eye movements when inspecting pictures. The Quarterly Journal of Experimental Psychology, 59(11), 1931-1949.
Underwood, G., Humphreys, L., & Cross, E. (2007). Congruency, saliency and gist in the inspection of objects in natural scenes. Eye movements: A window on mind and brain, 564-579.
Underwood, G., Templeman, E., Lamming, L., & Foulsham, T. (2008). Is attention necessary for object identification? Evidence from eye movements during the inspection of real-world scenes. Consciousness and cognition, 17(1), 159-170.
Võ, M. L.-H., & Henderson, J. M. (2009). Does gravity matter? Effects of semantic and syntactic inconsistencies on the allocation of attention during scene perception. Journal of Vision, 9(3), 24.
Võ, M. L.-H., & Henderson, J. M. (2010). The time course of initial scene processing for eye movement guidance in natural scene search. Journal of Vision, 10(3), 14.
Võ, M. L.-H., & Henderson, J. M. (2011). Object–scene inconsistencies do not capture gaze: evidence from the flash-preview moving-window paradigm. Attention, Perception, & Psychophysics, 73(6), 1742-1753.
Van Dijk, T. A., Kintsch, W., & Van Dijk, T. A. (1983). Strategies of discourse comprehension. New York: Academic Press.
Vogels, J., Krahmer, E., & Maes, A. (2013). Who is where referred to how, and why? The influence of visual saliency on referent accessibility in spoken language production. Language and Cognitive Processes, 28(9), 1323-1349.
Wolfe, J. M., Võ, M. L.-H., Evans, K. K., & Greene, M. R. (2011). Visual search in scenes involves selective and nonselective pathways. Trends in cognitive sciences, 15(2), 77-84.
Zwaan, R. A., & Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological bulletin, 123(2), 162-185. |
| Description: | 博士
國立政治大學
心理學系
99752501 |
| Source URI: | http://thesis.lib.nccu.edu.tw/record/#G0997525012 |
| Data Type: | thesis |
| Appears in Collections: | [心理學系] 學位論文
|
Files in This Item:
| File |
Size | Format | |
|---|
| 501201.pdf | 3987Kb | Adobe PDF2 | 269 | View/Open |
|
All items in 政大典藏 are protected by copyright, with all rights reserved.
|
