Reading Images (including moving ones)

The first important contributions with a systemic functional linguistic twist on the concept of ‘reading images’, a term coined by Kress and van Leeuven (1996), appeared in the 1990s. O’Toole’s 1994 work ‘The Language of Displayed Art’ can be said in many ways to have set the ball rolling in that it was a genuinely multimodal set, consisting of an article and a CD using a range of semiotic modes to provide an SFL perspective on the interpretation of visual works of art. Kress and van Leeuven’s work, both together (1996, 2006) and separately, expanded the field to examine many types of multimodal texts, from school text books to bus tickets. The images that were being read were initially largely static pictures, photos, drawings, diagrams, etc. It was the work of Baldry and Thibault, again working together (2006) and separately (Baldry, 2000, Thibault, 2000), that was most responsible for breaking into moving pictures and providing an SFL interpretation of video text, firstly television advertising but moving into documentary and film, providing Taylor (2004) with the material for an analysis of multimodal texts with a view to their translation (see Kunz and Teich this volume for other perspectives on translation). Other writers have produced important work which hinged on multimodal analysis from an SFL point of view (Ravelli, 2000, O’Halloran, 2000, Martinec, 2000, Iedema, 2001, Unsworth 2001, Lemke, 2002, Martin & Rose, 2008) and continue to do so (Martinec & van Leeuven, 2009, Painter Martin & Unsworth, 2013), but the above-mentioned pathfinders will form the springboard for the approach adopted in this chapter. Dealing with static images Kress and van Leeuven, Martin and others looked at representation in school text books, among other artefacts, as classic examples of the interplay between visual and verbal texts and the complex sets of relations that bind them together in certain environments. They examined the ideational aspects of classifying and composing in the case of explicit or implicit diagrams, labelled pictures, photographs accompanied by explanations, etc. and how the verbal elements (in terms of Halliday’s (1996) logico-semantic relations of enhancing, expanding, projecting) restated, summarized, specified or repeated the information provided in the visual component, or vice-versa. Where activities are concerned, and therefore movement of some kind, the use of vectors was discussed, either through explicit means such as arrows or textual configuration, or by less explicit means such as the interpretation of gaze and the use of insets. In terms of textual organization, Martin and Rose’s network diagram, based on Kress and van Leeuven’s work, illustrates how images are arranged to produce meaning. This diagram itself is of course a multimodal text with a simple connection between the verbal and the visual consisting of words, minimal symbolic representation and vectors. And which can be expressed solely verbally as follows: relevance tends to emanate from the centre of an image towards the margins in the sense that the central part is relevant to the whole text while at the margin the relevance is more specific to certain elements of the text. Given information picks up from preceding images/text or knowledge and tends to appear on the left of an image followed by New information to the right in the theme/rheme pattern. Generalised information, dubbed ideal by Kress and van Leeuven, tends to appear at the top of an image while more specific or practical information is seen below. Verbal text can be separated from the visual, as in bounded captions or embedded in the main discourse, or it can invade the visual space in the case of internal labels or insets. Finally, where multiple images are involved, some will be more salient and attract initial attention. By way of example of the above discussion, Taylor’s 2000 analysis of an advertisement for the Mitsubishi Pajero car, apart from a detailed analysis of the verbal text as an entity in itself, examined the text as a multimodal artefact. A social-semiotic approach to multimodality has interested several writers working with moving images. Boeriis (2008), for example, has attempted to clarify the sender-receiver distinction by separating the on-screen sender/receiver from the screen personality/viewer combination so as to make clear the text/context distinction. In other words the context of situation on screen (be it a film, television advertisement, documentary, whatever) is an artificially produced situation quite separate from the actual context of a viewer interacting with a film, even if the on-screen speaker is in close-up and seemingly communicating directly with the viewer. This will be important in analyzing the ideational and interpersonal components of the film phenomenon. Boeriis also makes a useful contribution to multimodal analysis by emphasizing the multisensory aspects of film interpretation, including the contribution that can be made from senses other than sight and sound, including the effect of comfortable seating and hot popcorn. Again the interpersonal component in multimodality is illustrated through secondary and tertiary inputs in the “multisensory understanding of communication”. In this way “cultural, conventional, habitual and ritualistic elements” are incorporated into any analysis of multimodal text. Among the various contributions applying SFL to the analysis of film text (), a recent European project (ADLAB) has concentrated on a minute examination of the many and varied aspects of Quentin Tarantino’s film ‘Inglourious Basterds’ with a view to producing a set of strategies for the audio description of this film, and by extension of all films, for the benefit principally of the blind and sight-impaired community in Europe. The fall-out from this work has been the identification of a series of ‘crisis points’ that audio describers need to pay attention to. This analysis also contained an SFL imprint in terms of textual organization, cohesion, appraisal and intertextuality. The film will act as a vehicle in the discussion of those factors that can inform SFL theory and of how SFL theory can inform multimodal analysis.