When Picture and Sound merge
by Sven E Carlsson
Sound is of great importance for how we experience moods and narratives. Especially so in the cinema where the visual and the auditive are linked together in various ways. This audio-visual interity is called synchresis, a central concept in film theory.
The function of sound in a movie is to affirm the moving pictures, thereby animating them. One can speak of the synchronization of the senses, where picture and sound go hand in hand and thereby intensify one another since, a part of our reaction to what we experience is based on our perception of sound and picture as a unity. At the same time we know that the sound is recorded separately in order to make it possible to be treated more freely when editing and mixing.
That this adding of sounds is so extremely essential for the narration
can, among other things, be explained by the fact that film, to a great
extent, is a question of movement, and movements are felt more natural
if combined with sound. It is due to mankind’s hereditary perception
– that sound is created by movements, in other words, every sound
rises from an original source. This conception of the surrounding world
can be connected to a pure instinct for survival, an ancient ability to
learn from sounds about threats, prey etc.
Chion’s basic theory is that the movie and its viewer sign an audiovisual contract implying that the audience “agrees” to ignore perceptual inequalities between film and reality. The moving picture on the screen and the sound from the loudspeakers is perceived as one indivisible unit. This is not a matter of course. In the childhood of the sound-film such a convention-contract was not established. The audience could feel that the people on the screen were moving along with the synchronic sound as in a ballet.
As soon as sound and movement are interpreted as one event a fusion is born in man’s consciousness. The sensory impressions are overlaid so that picture and sound contribute to the nature of each other. If the picture shows a door being closed and the sound from the loudspeakers presents a heavy bang the audience imagines having witnessed how the door is in fact slammed. This kind of sound-work can sometimes be necessary since many side- scenes are rickety enough to start swaying seriously if a door were to be really slammed. Another reason could be that the director during his supplementary work realised that something extra was needed to point out that the door-closing person was angry and asked the sound editor to cut in a heavy door-bang.
Film makers often want to elucidate and strengthen silent movements. The whining of a karate kick is in reality barely perceptible. But in filmic situations it is now an accepted convention to accentuate the movement with volume and force. Another example of accentuating is to make small movements more dramatic by sound-effects, as when Gestur draws his throwing-knife from its sheath in Hrafn Gunnlaugsson’s When the Raven Flies (Hrafninn flygur, 1984). The vigorous swish-sound tells you the knife is huge and thus extra dangerous. Perceptual synchresis also makes it possible to speed up the cutting-rate without making the action inconceivable. This has been carried to an extreme in cartoons. The high speed is made possible by accompanying every movement with a sound-effect or a musical loop. And when sight and hearing are in accord we can process more information.
in genres where a rapid flow of information keeps us constantly busy receiving
the swift torrent of impressions. As a result there is not always time
for us to judge whether something seems correct or not. Completely improbable
sounds can pass unnoticed, as when Linda Hamilton in James Cameron’s
Terminator 2 – Judgement Day, 1991, is running barefoot
in a hallway and we hear the sounds of her clattering shoes! Unrealistic
synchresis is a distinctive mark for the genre of horror movies where
even the camera movements like zooms and tilts can be accompanied by unpleasant
An example of heavy synchresis is found in Hitchcock’s The 39 Steps, 1935. The sound we hear of a terrified woman screaming, at the finding of a corpse, comes from a howling steam-whistle. Since drama and intensity are great, we would not have thought her scream unrealistic had Hitchcock not shown the source of that sound by cutting in a steam-engine afterwards. To enable this form of unrealistic synchresis to work, with strong diverging sounds in a dramatic narration, the spectators must be absorbed in the film. And it is in emotionally strong scenes that odd sounds can be used to express strong feelings, in other words where your critical judgement is disconnected.
Exaggerated sound-effects and music every now and then signals jokes in cartoons. If an animated figure climbs steps, stairs or a ladder the orchestra plays an ascending scale. If he falls to the ground you hear a whistling pipe in a descending glissando and then a cymbal crash. As already mentioned, music accompanying movements is an important ingredient in the cartoon. This is sometimes nicknamed ‘mickeymousing’, which personally I find somewhat derogatory. A corresponding example from the field of the feature movie can be seen in Terence Young’s Dr. No, 1962, when the music reinforces every stamp that James Bond makes when killing a poisonous spider.
There is also a more delicate humour synchresis where you don’t notice the jokes until you have seen the movie a couple of times. How many people have observed in that scene in Mr. Hulot’s Holiday with the silent ping pong match, that at the moment one of the card players takes a swig of beer, you hear a sound reminding you of water streaming in a partially blocked sewage pipe. Anyone who wants to study delicate humour synchresis can benefit a lot from Joel Coen’s Barton Fink, 1991, or by any or all of the films by Jacques Tati.
If a movement starts in synchresis, then the sound has defined the movement in a way that this sound and this movement belong together. When that concordance then disappears and we see foggy pictures difficult to interpret, we keep “seeing by way of our ears”. Referring to the sound, imagination takes over and makes us believe that we have seen what in fact the sound is telling us. In Niels Gaup’s Pathfinder (Ofelas), 1987, a dog is shot with a cross-bow. The audience can hear as well as see the arrow go off. Then they see in blurredness the silhouette of a dog falling backward. What people heard was the hitting sound and a dog whimpering. But inside themselves they saw the arrow hit and how the dog whimpered when falling, dead. >> Listen to the dog
Synchresis is one of the explanations to the flat pictures of the film receiving more depth when the sound is added. Sound is three-dimensional by character and has a spatial extension, while the moving pictures in a film are two-dimensional. By the fact that movement and sound are perceived as an indivisible wholeness, things and people are given a “body” by the sound. This causes us to experience the pictures as even more alive.
Literature: Chion, Michel: Audivision, New York: Columbia University Press, 1990/1994
© Sven E Carlsson
Visit FilmSound.org - Learning Space dedicated to the Art and Analyses of Film Sound Design
|Star Wars Sounds||Film Sound Clichés||Film Sound History||Movie Sound Articles||Bibliography|
|Questions & Answers||Game Audio||Animation Sound||Glossaries||Randy Thom Articles|
|Walter Murch Articles||Foley Artistry||Sci-Fi Film Sound||Film Music||Home Theatre Sound|
|Theoretical Texts||Sound Effects Libraries||Miscellaneous|