measure self-awareness

put a spot on somebodys face and then put them in front of a mirror.

Do they touch the spot on the mirror or the spot on their head?

HEAD=AWARE

MIRROR=NOT AWARE; SEEING ANOTHER ANIMAL

Children pass mirror test after 18 months

Chimps, orang, dolph, and some birds pass mirror test

DISTINCT MENTAL STATES

(ToM)

We have to infer unobservable mental states of others

this includes beliefs and desires

MENTALIZING

(ToM)

Attribution and representation of mental states

This is the core of the human capacity for forming a ToM of others

it is done EFFORTLESSLY and AUTOMATICALLY

Culturally universal and perhaps unique to humans

Clinical implications for a variety of disorders, most notable Autism Spectrum Disorder

False Belief Task

(ToM)

a test for distinct mental states

can you keep your beliefs separate from others?

if youre given info that changes your belief about a situation, do you assume others will have changed beliefs as well?

to keep mental states separate you have to be able to represent others thoughts as belonging to other people

WHAT'S MINE IS MINE AND MINE ALONE.

The little girl in the youtube video originally thinks that there is Play Doh in the container and that everybody else does... but when they tell her it's coins she says that her and everybody else will say coins are in there... so she doesn't have ToM cause she updated everybody elses beliefs

CHILDREN UNDER 3 FAIL

BY AGE OF 5 THEY PASS

(works in all tested cultures; even hunter-gatherer)

ASD delayed at this

ToM builds on self awareness

They each do not emerge at the same time

First, Self-Aware-- then ToM

Saxe and Kanwisher

ToM

2003

Test adults to see if there is a specific region of the brain associated with reasoning about mental content of other minds

2 Experiments to test for REGIONAL SELECTIVITY to ToM reasoning:

1. Does regional acitvity increase to ToM reasoning compared to non-social reasoning?

2. Does it respond to reasoning about mental states specifically, or to people in general?

4 CATEGORIES OF STORIES

1. Actions caused by FALSE BELIEFS (ToM)

2. Description of human actions (ToM)

3. Stories of MECHANICAL INFERENCE, relying on hidden physical causes (control)

4. Descriptions of nonhuman objects (control)

Saxe and Kanwisher

Experiment 1

2 SAMPLE STORIES

1. ToM- False belief where mom throws out paper mache cause she thinks it's garbage

2. Control- Mechanical Inference- The water disappears because of evaporation, a physical force

RESULTS

A number of regions more activated by ToM than mechanical inference

LEFT AND RIGHT TEMPORAL-PARIETAL JUNCTION (TPJ) were the most specifically and consistently activated (22 or 25 people)

THEN COMPARED ACTIVATION to STILL BODIED PICTURES AND OBJECTS

This isolates the extrastriate body area (EBA), which is near the TPJ

This ensured that areas responding to ToM stimuli were different than the areas responding to human bodies (via imagery) 

Saxe and Kanwisher

Experiment 2

First experiment got bilateral TPJ activity to ToM

but maybe this was because the story having people in them... or was it because of the attribution of mental states?

and the ToM story was the only one to have a false belief... maybe it was activated because of hypothetical reasoning

NEW STORIES!

1. False Belief- false content of mental representaton

2. False photograph- false content of physical representation

3. Desire story- goal or intentions (ToM)

4. Inanimate object description

5. Physical description of people (no mental content)

THE EXPERIMENT

1. False Belief- Emily knows nothing about cars so John says he has a Honda but he has a Camry... Emily sees his car and says its a HONDA

2. False Photograph- Photo taken of an apple on a tree, while it develops the apple falls to the ground... The developed photograph shows an apple ON THE BRANCH

THE RESULT

Bilateral TPJ activated most to FALSE BELIEF story

TPJ was not selective at all for false belief photos... so the area is NOT ACTIVATED BY COUNTERFACTUAL REASONING

Not selective for physical descriptions of people

High degree of ToM activity overlap in subjects that did both experiments

BOTH SIDES RESPOND, RIGHT IS MORE SELECTIVE

Saxe and Kanwisher

CONCLUSION

TPJ has doman specificity for ToM reasoning- especially false beliefs

TPJ does not respond to physical characteristics of people

Desires (hunger, thirst, tiredness) don't activate TPJ

Selectivity for false beliefs may be more lateralized to the RIGHT TPJ

Effects seem consistent (~90% of subjects) and replicable

Right TPJ (most consistent region responsive to ToM stimuli)

Left TPJ

Posterior Cingulate

Medial Prefrontal Cortex

Most responsive to the delivery of mental state information

Experiment gave varied timing of info on mental state after giving background info on a story.

Delay in activation of right TPJ = delay in the story's mental content

1. Not selective to mental states, but external cues for attention

-Activity increases when subjects are forced to interrupt a cognitive task and reorient to another stimulus

-Perhaps 'mental content' is an interruption that triggers an attentional shift

ALTERNATE TPJ

Mitchell

1. Subject respond to target that appears following a spatial cue (r or l or center). Subject attended to cued location, but the target would sometimes go to the wrong location. Such 'invalid' cues require a SHIFT OF ATTENTION

2. The ToM task used the false beliefs and false photographs used in Saxe and Kanwisher

RESULTS

Shifts of attention were found to selectively activate regions near rTPJ

Significant OVERLAP of the areas responding to attention shifts and ToM tasks

perhaps it is a GENERAL ATTENTIONAL FUNCTION, not a domain specific ToM role!

SCHOLZ tested this shit and saw that it was low scanning resolution and partial voluming that fucked shit up-- ultimately the areas are neighboring but spatially distinct... although ToM does activate attention regions

ALTERNATE TPJ

Mirror Neurons

Neurons that respond to the passive viewing of another individual performing an action and when performing the action yourself

The brain regions for the two seem distinct

-Maybe in separate networks that work together-

Mirror more involved in GOAL-ORIENTED ACTIVITY

TPJ more involved in MENTAL CONTENT (more passive in nature)

MAYBE... Mirror neurons provide and entry point for modeling the goal-directed actions of others

ASD more consistent with mirror neuron disfunction than problems with TPJ function

IS THERE A THEORY OF MIND MODULE?

CONCLUSION

1. This is unclear, but the right TPJ is consistently recruited in false belief tasks

2. This is separable from simple attributions like hunger, thirst, tiredness

3. Other regions ARE involved in the processing of ToM information 

RIGHT TPJ

&

Moral Judgment

Amount of moral blame we assign to an individual depends on their BELIEFS WHEN PERFORMING an action AND the OUTCOME of their action.

Young and Saxe

The scope of RPTJ involvement:

-Emotional content (how you feel about something)

-True and false beliefs

-Good and bad reasoning about beliefs

-Auditory and visual representation of belief information

RPTJ

MORAL JUDGMENT

YOUNG & SAXE

EXPERIMENT 1

Two stories about Grace and deadly sugar...

1. Negative belief- negative oucome

2. Neutral belief- negative outcome

MOST BLAME- NEG BEL, NEG OUT

2. NEG BEL, NEUTRAL OUT

BIG GAP

3. NEUTRAL BEL, NEG OUT

LEAST BLAME- NEUTRAL BEL, NEUTRAL OUT

RTPJ active in all cases...

HOWEVER, was more selectively active when the protagonist ATTEMPTED HARM, but had no consequence

...maybe this is because it activates more social reasoning because there was a mismatch between intention and outcome

Does the RTPJ respond to outcomes in moral judgments?

Moral Fact

1. Neutral Outcome- Meat was fresh because of tight package and sealed in the fridge

2. Negative Outcome- The meat has an invisible deadly bacteria because of a small tear in the seal

Nonmoral Fact

1. Meat packaged and purchased at the grocery story next door

Mediates the UPDATE OF BELIEFS in moral judgments

RTPJ was more responsive to the moral fact

..Suggests RTPJ helps the individual make a spontaneous inference about an actors beliefs, attributing beliefs based on moral outcome information

1. RTPJ active during all belief conditions

2. RTPJ selectively EVEN MORE ACTIVE when attempting harm, but no bad outcome.  (Subject may rely more on belief attribution  than outcome when assigning blame)

3.RTPJ active to MORALLY RELEVANT FACTS (spontaneous belief attribution)

1. I know I am separate from others (self-awareness)

2. I know that we all have many different beliefs (ToM)

3. These beliefs and other mental states are socially meaningful (making social moral judgments)

4. HOW CAN I CHANGE YOUR BELIEFS OR MENTAL STATES TO MATCH MINE OF PROVIDE ME WITH SOME BENEFITS?

All animals use COMMUNICATION to alter the behaviors of others.

They do this with SIGNS or displays that are innate (this may not always be intentional)

Human LANGUAGE is unique in its SYMBOLIC form-- we use arbitrary symbols in a complex syntax (grammar) to form potentially infinite meanings

SPEECH SOUNDS AND THEIR PERCEPTION 

Phones (Transmitted)

-Smallest speech sound in human language

-Usually divided into vowel and consonant sounds

-Approximately 200 possible sounds

Phonemes (Perceived)

-The listener's perception of a phone

-A few 10s of these used in any one spoken language

-The basic building blocks of words in a given language

-By 6 months of age, infants begin to prefer the phonemes of their language

Species-Specific Mechanism

Pheromones may signal mating rituals

Scents used to mark territory (dog) or leave trail (ants)

Some calls identify threats, bird songs used to attract mates

Imitation of Behavior in 'higher' mammals- such as primates- may be the foundation for complex social relations

(I smile, you smile)

Onset of a yawn triggered by hearing, seeing, or thinking about another person yawning is the imitation of a particular behavior that is very generalized by humans

Relates to empathy and mental state attribution in humans

Chimps show it, but only for members of their own group

People with ASD show impaired contagious yawning

1. Words are built from phonemes

-Words can also be built from other words (seahorse)

2. Sentences are built from words using language-specific rules called syntax

-These sentences convey meaning in a narrative form know as 'propositional content' (who did what to whom)

3. Discourse provides the larger context of the sentences taken together

-Requires an inference made by the listener, using background knowledge

("Happy holidays everyone!" -- makes you use knowledge not given in the sentence to think about what holiday is coming and you just naturally understand the sentence)

Must use conventional (cultural) symbols and rules that (at least) meet the following criteria:

1. Recursive Grammar: Infinite meanings can be constructed from finite symbols (words) and rules (syntax)

2. Productive: Rapid and (largely) automatic combination of meaningful utterances

3. Displacement: Can refer to things not in the current environment-- whether historical, abstract, or hypothetical

Allows for non-literal or figurative interpretation of linguistic data

"David Ortiz Terrified After Hearing About Red Sox Bats Coming Alive"

1. The Red Sox are doing better

2. OMG THERE'S A LIVE BAT AT THE BASEBALL GAME

CONTEXT-FREE GRAMMAR because the rules allow you to continue building sentences, regardless of the content. The meaning and value can change with each iteration.

VS

FINITE-STATE GRAMMAR which patterns may be biologically fixed (bees dance to signal where food is located- only one meaning)

LANGUAGE

CAUSAL ROLE

ToM

False belief stories seem to implicitly use complex language structures (Grace thinks that [factual argument])

Ages 3 to 5 when children develop theory of mind they also develop large-scale language

META-ANALYSIS by Milligan

studies of language and false-belief in children

Attempted to correlate language ability to ToM

Moderate-to-Large correlation between language performance and false belief task performance (43%)

Correlation was not dependent on ones element of language, but based on general language ability

RELATIONSHIP MAY BE BI-DIRECTIONAL

Language gives symbolic form to the mental content being attributed to others

Grammar of language seems similar to theory of mind constructs

RIGHT FFA = face processing

RIGHT TPJ = ToM

RIGHT EBA = headless bodies

LEFT FUSIFORM GYRUS = Visually presented words (visual word form area or VWFA)

Specialized in rapid reading of words and strings of letters

Specific to visual stimuli

respons to words and pseudowords

response in region are invariant to changes in fonts, CaSe, or other surface features

ERPs show negative electrophysiological responses in this area to words after approx 200ms (N200)

Lesions = PURE ALEXIA - a word form dyslexia in which you can not read words as a whole, but need to spell out the letters

(like prosopagnosia -- in fusiform gyrus)

VWFA (LFG) is also implicated in developmental dyslexia

Adults with a history of impairments in learning to read show less activation in LFG compared to control subjects

Doesn't matter what the language is

1. FFA (right fg) may have developed under the pressure to recognize faces

2. Neural architecture APPROPRIATED for language use and requires long development through childhood (unlike faces)

3. Suggest VWFA develops a VISUAL EXPERTISE for words recognition through progressive specialization

RECENT DATA says lateralization of face and word form processing is due to SPATIAL FREQUENCY OF

Compared faces (low spatial freq) and words (high spatial freq) in fMRI study

NORMAL LATERALIZATION

Compared gratings of low and high spatial frequency

Low spatial frequency overlaps with face area, high spatial frequency overlapped with words

Reversible procedure that allows functional separation of hemispheric operation

-Used to test pre-op patients for language lateralization prior to surgery to treat epilepsy

-Sedative injected into one cerebral hemisphere via a catheter// this temporarily inhibits it

-Patients are then given simple test on language production

-Responses are thought to be processed and controlled by the 'awake' hemisphere

-96% of right handers are left-lateralized for speech

-70% of left handers are left-lateralized for speech

(15% bilateral, 15% right-lateralized)

Corpus Callosum has 200 million axons  that allow transfer of information across the hempispheres

Commissurotomy- For people with severe epilepsy which is a surgical lesion of the nerve fibre tracts connecting the two hemispheres

This separation inhibits the sharing of information, resulting in exposure of some functional specialization

SOOO...

ITEMS PRESENTED TO THE RIGHT VISUAL FIELD GO TO THE LEFT HEMISPHERE, BUT THE INFO DOSN'T TRANSFER OVER

(AND LEFT TO RIGHT, STILL WITH NO TRANSFER)

In the typical patient...

LEFT HEMISPHERE has intact language use (speaking and listening are normal)

RIGHT HEMISPHERE has rudimentary word recognition, but cannot produce speech

(better at pattern matching and holistic processing)

SHOW A SPOON TO RIGHT EYE SAY SPOON! 

SHOW A SPOON THE LEFT EYE -- SEES NOTHING.

Expressive (Broca's) Aphasia- Great difficulty in producing language (both spoke and written)

Associated with damage to the LEFT INFERIOR FRONTAL GYRUS (Broca's Area)

Halting speech, problems with grammar (syntax), and word repetition

Ability to comprehend language generally remains intact (except for complex grammar)

Similar effects seen in people who use ASL

Receptive (Wernicke's) Aphasia- Patients have fluent speech but it is INCOHERENT or contains many PARAPHASIAS (substituting one word for another)

Associated with LEFT POSTERIOR SUPERIOR TEMPORAL GYRUS

Patients have trouble understanding spoken and written language

Grammar remains intact

Patients who recover say they could not understand language when aphasic, but knew it was being used

Can still sing words to songs

Similar with ASL

Primary language areas (production & understanding) are typically on the left hemisphere

-There is a DOUBLE DISSOCIATION between production and understanding-

However... some functions are on the right...

-Prosody- tone and emotional content

-Discourse level information

-Metaphorical meaning

-Appreciation of humor

-Abstract understanding?

LANGUAGE

VENTRAL AND DORSAL

Functionally distinct dorsal and ventral stream...

Dorsal- maps phonological representations to articulatory representation (speech acts)

-Projects from PARIETAL-TEMPORAL BOUNDARY

-Other sensory modalities may project info here

-Ultimately projects to frontal areas involved with speech production

Ventral- maps phonological representation to conceptual representations (word meanings)

-Projects to INFERIOR AND POSTERIOR TEMPORAL CORTEX

-Sound-based representations are mapped onto a widely distributed conceptual network ('what' pathway)

-Has a left hemisphere bias, but it bilateral

At the initial speech processing stage, sound information is processed in the dorsal surface of the SUPERIOR TEMPORAL GYURS

This sound information is mapped onto the phonological representation in the SUPERIOR TEMPORAL SULCUS

From here, the paths diverge into dorsal and ventral streams

Dorsal stream is strongly left dominant and is primarily involved with SPEECH PRODUCTION. Damage here = BROCA'S

Ventral stream is more bilateral (but left-biased) and map sounds to meaning. Damage here = WERNICKE'S

THIS IS HIGHLY DISTRIBUTED

INFO PROCESSED IN BOTH HEMISPHERES

Activity distributed within temporal lobes

No central processing location

Meaning construction and interpretation is a dynamic process and involves rapid changes in content

(You can't miss a bit of something or you lose it all!)

fMRI experiments are useful in isolating regions involved in that process, but don't give sufficient temporal resolution

EEG measures brainwaves at the scalp...

Take the average of all activity associated with a specific event to get the EVENT RELATED POTENTIAL (ERP)

EVENT-RELATED POTENTIAL

(ERP)

WHAT ARE THEY

Formed by averages EEG activity that is time-locked to the onset of a stimulus

Represent post-synaptic neural activity associated with the processing of the stimuli

Background (non-task related) activity is thought to be removed during averaging

The remaining averaged signal reflects a specific process (attention, memory, or language comprehension)

1. Polarity- pos or neg wave

2. Latency- how long after stimulus does it peak

3. Topography- How does the wave look on diff locations of the scale

FUNCTIONAL SIGNIFICANCE

1. what cognitive acitivity is it sensetive to

2. what makes it bigger or smaller

1. Have subject read a sentence presented one word at a time

2. Change one word in the sentence in such a way that it is either CONSISTENT or INCONSISTENT with the meaning of the sentence

3. Compared ERPS to the words and how the brain reacts

"The pizza was too hot to..."

1. eat

2. cry

N400 had largest response to unexpected word

Size of N400 related to how related the word was

(Cry = high, drink = med, eat = low)

Serves as an INDEX for the ease of difficulty of retrieving stored conceptual knowledge

The more inconsistent a word is relative to a sentence, the more work the brain has to do in retrieving additional conceptual knowledge

You get them because...

1. Anomalous meaning in middle or end

2. Smaller when word is repeated at end

3. Smaller for frequent words

4. In sensible sentence-- big for beginning and smaller at end

RESPONDS TO...

Both auditory and written words

Pictures and line drawings

Topography a little different (suggesting different of the areas of the brain process different stimuli in similar ways)

MUSIC

Bad not = P600

Bad lyric = N400

PRODUCED IN TEMPORAL LOBE

GENERATED ON THE LEFT

(N400 SLIGHTLY LARGER ON RIGHT SIDE OF SCALP, DUE TO SHAPE OF LEFT CORTEX THAT PROJECTS SIGNAL TO THE RIGHT)

1. One of two universal behaviors for humans (without non-human analog)

2. Perception and production relies on hearing, motor learning, perceptual experties

3. Rich expertise in production- can use musicians vs non musicians in long-term studies

What is music for?

Perhaps...

music is "auditory cheesecake" that piggybacked on the systems evolved for language perception and production

It's just an auditory fun thing that we do and has very little in terms of brain structure for just music and not language

Amusia

Used as evidence for dissociation between music and language structure

In ability to recognize one tune from another

Hard time telling if two notes played which one is higher and which one is lower

BUT FOR SOME REASON THESE PEOPLE HAVE NO PROBLEM WITH LANGUAGE

DISCRIMINATION TASK

There is a spread for the amusical and control group....

the amusical is just worse

Given pitch change task in language and non-language

We see that amusics are only a little bit worse then the control group when it comes to language, but much worse when it comes to music

Structural context over time (syntax)

-Similar neural correlates of unexpected harmonies and unexpected words

-Behavioral correlate

There is a prediction of word likelihood

Neural evidence includes ERP correlates of expectation violation like N400

WE DO THE SAME THING WITH MUSIC

(melodic direction, likely notes, likely harmonies)

Eran and N5 to irregular cords

MEG localization to BILATERAL IFC (Broca's area)

fMRI localization

Little evidence that music contains semantic meaning in the same way lang does...

but music can clearly have strong associative meaning, and also be a powerful affect regulator independent of association

music is a mood regulator and has the capacity to make you feel a different way

Koelsch

MUSIC FOR SEMANTIC PROCESSING

Sentence to match picture/ Sentence to mismatch picture

This makes an N400 occur because of unexpected words

THEN THEY DID IT WITH MUSIC!

They played sounds and showed words

Got an N400 for the music that didn't match the word

Diverge

Stroke can prevent people from speaking, but not necessarily singing. Not only do they know old songs, but they can learn new ones.

Used to rehabilitate speech because over time you remove melodic contours

DOUBLE DISSOCIATION

Converge

Similar ERP components (N400)

Similar locations in brain at broca's area (LIFG)

Rehab of one can affect the other

Mnemonic powers of music, epic poetry, troubadours

Music can then make language better

Self-paced reading study

Syntactic expectancy and Semantic expectancy

N400 goes up at wrong word

Musical Syntactic violation

(at same time as music)

SHARED SYNTACTIC, BUT NOT SEMANTIC PROCESSING

Syntactic problem in reading and sound = increase reaction time

Syntactic sound but semantic word = no difference in reaction time

Lots of synapses in auditory system before thalamus and A1

Musical experience change how pitch is processed before it gets to auditory cortex

Mesure electrical response generate by nuclei in the brainstem of musicians and non-musicians listening to Mandarin words. None spoke Mandarin.

MUSICIAN

Can easily see how they track difference in pitch

NON-MUSICIAN

Really hard for them to track pitch

1. Beat- Rhythmic things speakers do in high point of speech. Have emphatic value.

2. Emblems- Conventionalized gesture that have meaning that speakers in a language community all understand the meaning. You either know the meaning or you don't.

3. Iconic Gesture- Body movement accompanying speech that convey spatial information

Shape or size of an object

Relative position of two geographic locations

Trajectory of motion

Action- slam dunk!

Deictic Gesture- depends on where you are (humans understand pointing, pigs don't)

(cause u to notice, not represent)

Emblems- gestures meaningful in virtue of the speakers agreeing on it

Sign- Like emblems, but in a fully productive system

Action

DG

IG

Emblems

Signs

No

no good for listener, just speaker

support his info

Yes

arise together

thoughts have propositional part and imagistic part

good for speaker and listener

GOOD FOR SPEAKER ALONE TOO!

Like when talking on the phone, people will do hand shit

• When speech is ambiguous or noisy, listeners use gestural information

– Thompson & Massaro

– Rogers

• Information uptake better in cases of gesture-speech matches than gesture- speech mismatches

– Cassell, McNeill, & McCullough

N400 – Negative-going wave – 200-700 ms post-word – Peak approx. 400 ms

Modulated by – WordClass

• Kutas & Van Petten, 1990 – Contextual Congruity

• Kutas & Hillyard, 1980 – Cloze Probability

• Kutas & Hillyard, 1983

Index of difficulty of lexical integration

Index of processing the meaning of an event

SUGAR - P400

NUTRASWEET- N400

HAPPENS TO PICTURES TOO!

Pipe VS Carrot

RESULT

Words in back of head

(fronto-central)

Pictures in front of head

Picture specific semantic system

FRONT OF HEAD

Unidetify = large response

non match = medium response

match = low response

N300 sensetive to this and just does visual processing by seeing if pictures are related

ALSO WORKS for comic strips with no words

Large N300 to how things will unfold over time

Do contextually incongruous gestures elicit an N300- or N400- like ERP component?

100-200 iconic gestures

– spontaneously produced

– half contextually congruous

– half contextually incongruous

• Ask people to describe cartoons

• Show ERP subjects cartoon clip followed by appropriate or inappropriate video of description

EXP 1

Task:Indicateviabuttonpresswhethersilent video “goes with” the preceding cartoon

EXP 2

Relatedness Decision to Probe Word

See two videos and then a word and say whether it was related to one of the preceding videos

PREDICTION

If comprehending gestures recruits processes similar to those activated by meaningful images, contextually incongruent gestures should elicit enhanced N400 relative to congruent ones.

REULTS

EXP 1

N450

LPC 

EXP 2

N450

NO LPC

Complex visual stimulus over time

Razor or rolling pin?

Static vs. Dynamic Gestures

Static “gestures” may afford better time-locking

Cartoon followed by dynamic or static gesture that was congruent or incongruent with info before

THEN see a probe word asking if it pertains to the picture or video

• Static freeze frames from gesture videos elicited similar ERP congruity effects as did dynamic videos of gesturing

– N400 congruity effects static & dynamic – N300 congruity effect in static only

• Similar brain areas were engaged by both sorts of stimuli

– motivating their usage in comparison with photographs

Gestures vs. Photographs

Toaster --> Toast --> "Kitchen"

YES!

Bigger N300 and N400 to unidentifiable, to unrelated, to related

• What’s the relationship between processing of gestures and other sorts of meaningful visual representations (i.e. photographs)?

• Similar time course, similarly modulated by relationship to context

• But, clearly some differences in underlying neural generators

• Participation of brain areas tuned to the human body

• Relationship between gestures and the things they represent is more abstract than the visual resemblance between photographs and what they represent

CLASSICAL VIEW

VS

SPACE STRUCTURING MODEL

word meaning + syntactic structure --> sentence meaning --> utterance meaning + context

VS

perceptual input + current state + lang input + social context --> Cognitive model

Language prompts the construction of cognitive models (viz. construals) in working memory

MAYBE + GESTURAL INPUT TOO?

Semantic Indeterminacy

• Meaning in natural language is underspecified – even in very straightforward communicative contexts

• Speakers exploit a variety of resources to help evoke construals with the necessary degree of specificity

• Gestures

Gestures and speech jointly activate stored knowledge.

SNAKE

Do gestures prime related words?

Even when task does not highlight probes?

• Participants watch silent videos of spontaneously produced iconic gestures

• Eachvideofollowed by probe word

• Participantsindicate whether probe words were related or unrelated

YES

N400

Do speakers integrate gestural information with linguistic information in the accompanying speech?

“It’s actually a double door.”

Speakers use iconic gestures to enhance their cognitive models

UP DOWN DOOR

or

TWO SIDE BY SIDE DOORS

• Using conceptual integration processes, listeners combine

– propositional information in speech with

– analogue information in iconic gestures

– to form more specific expectations about discourse referents

Empirical Assessment

• Vary Speech-Gesture Congruity

– Test impact on ERPs to picture probes

• Vary Picture Probe Congruity with prior discourse

– Test impact on ERPs to picture probes

CONGRUENT VID PRIME OR INCOGRUENT

ERPs elicited by highly related picture probes

Keying in on if gesture afftects perception of picture

Slightly more natural paradigm

• All speech-gesture pairings congruous

• Vary how well picture probe matches prime

SEE PERSON + HEAR WORDS --> SEE PIC --> PICK WHICH IS MORE RELATED

N300 + N400

HEAR WORDS --> SEE PIC --> PICK WHICH IS MORE RELATED

N400

ALWAYS BIGGER FOR UNREALTED