Quantcast
Channel: Refractory: a Journal of Entertainment Media
Viewing all 76 articles
Browse latest View live

Volume 24, 2014

$
0
0

Volume 24, 2014

$
0
0

Volume 25, 2015

$
0
0

Themed Issue: Eye-Tracking the Moving Image

Edited by Sean Redmond & Craig Batty

Contents:

1.  Seeing into Things: Eye Tracking the Moving ImageSean Redmond & Craig Batty

2.  Movement, Attention and Movies: the Possibilities and Limitations of Eye Tracking?Adrian G. Dyer & Sarah Pink

3.  How We Came To Eye Tracking Animation: A Cross-Disciplinary Approach to Researching the Moving Image -  Craig Batty, Claire Perkins & Jodi Sita

4.  Sound and Sight: An Exploratory Look at Saving Private Ryan through the Eye-tracking Lens - Jenny Robinson, Jane Stadler & Andrea Rassell

5.  Subtitles on the Moving Image: An Overview of Eye Tracking Studies - Jan Louis Kruger, Agnieszka Szarkowska & Izabela Krejtz

6.  From Subtitles to SMS: Eye-Tracking, Texting and Sherlock – Tessa Dwyer

7.  Our Sherlockian Eyes: the Surveillance of Vision - Sean Redmond, Jodi Sita, & Kim Vincs

8.  Politicizing Eye-tracking Studies of Film – William Brown

9.  Read, Watch, Listen: A commentary on eye tracking and moving imagesTim J. Smith

Movement, Attention and Movies: the Possibilities and Limitations of Eye Tracking? – Adrian G. Dyer & Sarah Pink

$
0
0

Abstract

Movies often present a rich encapsulation of the diversity of complex visual information and other sensory qualities and affordances that are part of the worlds we inhabit. Yet we still know little about either the physiological or experiential elements of the ways in which people view movies. In this article we bring together two approaches that have not commonly been employed in audience studies, to suggest ways in which to produce novel insights into viewer attention: through the measurement of observer eye movements whilst watching movies; in combination with an anthropological approach to understanding vision as a situated practice. We thus discuss how both eye movement studies that investigate complex media such as movies need to consider some of the important principles that have been developed for sensory ethnography, and in turn how ethnographic and social research can gain important insights into aspects of human engagement from the emergence of new technologies that can better map how an understanding of the world is constructed through sensory perceptual input. We consider recent evidence that top down mediated effects like narrative do promote significant changes in how people attend to different aspects of a film, and thus how film media combined with eye tracking and ethnography may reveal much about how people build understandings of the world.

Introduction

Seeing in complex environments is not a trivial task. Whilst people are often under the impression that you can believe what you see (Levin et al. 2000), physiological and neural constraints on how our visual system operates means that only a very small proportion of an overall visual scene might be reliably perceived at one point in time during the evaluation of a sequence of events. Evidence for the way in which we often only perceive a portion of the vast amount of visual information present in a scene is nicely illustrated in the ‘Gorillas in our Midst’ short (25s) motion sequence where a group of six participants (three dressed respectively in white or black teams) are filmed passing a basketball between team members (Simons and Chabris 1999). Subjects observing the film sequence are required to count the number of passes between the three students dressed in white, and whilst many subjects do correctly count the number of passes, the majority of test subjects fail to observe a large gorilla (an actor dressed as a gorilla) that walks into the middle of the visual field and beats it’s chest, before walking casually out of the scene. People typically don’t see this salient gorilla in the action sequence because their attention has been directed to the basketball catching team in white with the instruction of counting the number of passes. Why do we miss such a salient object as a gorilla, and what does this mean for our understanding of how different subjects might view complex information in real life, or in presentations that encapsulate aspects of real life, such as movies?

In this article we take an interdisciplinary approach to the question of how we might see certain things in complex dynamic environments. We draw together insights from the neurosciences and eye tracking studies, with anthropological understandings of vision and audio-visual media in order to map out an approach to audience research that accounts for the relationship between human perception, vision as a form of practical activity, and the environments through which these are co-constituted. We first build a brief outline of how the eye, visual perception and the subjectivity or selectivity of viewing are currently understood from the perspective of vision sciences. This demonstrates how physiologically there is evidence that the eye sees selectively, yet it does not fully explain why or how perceptual understanding might vary across different persons, or for the same person across different contexts. We then build on this understanding with a discussion of what we may learn from eye tracking studies with moving images. As we will show, eye tracking can offer detailed measurements of how the eye attends to specific instances, movements, and points within sequences of action. This can reveal patterns of attention across a sample of participants, towards specific types of action. Yet eye tracking is limited in that while it can tell us what participants’ eyes are attending to, it cannot easily tell us why, what they are experiencing, what their affective states are, nor how their actions are shaped by the wider social, material, sensory and atmospheric environments of which they are part. Therefore in the subsequent section we turn to phenomenological anthropology, and draw on the possibilities provided by the theoretical-ethnographic dialogue that is at the core of anthropological research, to suggest how the propositions of eye tracking studies might be situated in relation to the ongoingness and movement of complex environments.

We discuss that such an interdisciplinary approach, which brings together monitoring and measurement with qualitative and experiential research, is needed to generate understandings of not only what people view but of how these viewing practices and experiences become relevant as part of the ways in which they both perceive and participate in the making of everyday worlds. However we end the article with a note about the relative complexities of working across disciplines, and in particular between those that measure and those that use empathetic and collaborative modes of understanding and knowing, which can be theorised as part of the ways film is experienced (Bordwell and Thompson 2010; Pink 2013). For a review of how these issues may relate to broader issues about film culture, eye tracking and the moving image readers are also referred to the manuscripts in this special issue by Redmond and Batty (2015), and Smith (2015).

Visual Resolution, Perception and the Human Eye

To enable visual perception the human eye has cone photoreceptors distributed across the retina which enable wide field binocular visual perception of about 180 degrees (Leigh and Zee 2006). In the central fovea region of the eye, cone photoreceptors are much more densely packed, and our resulting high acuity vision is only about 2-3 degrees of visual angle (Leigh and Zee 2006). Visual angle is a convenient way understand the relationship between the actual size of an object and viewing distance, for example, our fovea acuity is approximately equivalent to the width of our thumb held at about 57 cm (at this distance 1cm represents 1 degree of visual angle). This means that to view visual information in detail it is often necessary to direct the gaze of our eyes to different parts of a scene, and this is typically done with either ballistic eye movements termed saccades, or much slower smooth pursuit eye movements like when we follow the movement of a slow object in the distance (Martinez-Conde et al. 2004). Saccades are commonly broken down into two main types that are of high value for interpreting how viewers might perceive their environment, including reflexive saccades mainly thought to be driven by image salience (also termed exogenous control), or volitional saccades (endogenous control) where a viewers’ internal decision making directs attention through top-down mechanisms to where the gaze should be attended within a scene or movie sequence (Martinez-Conde et al. 2004; Parkhurst et al. 2002; Tatler et al. 2014; Pashler 1998; Smith 2013). Thus eye movements can be, in very broad terms, described as ‘bottom up’ processing when the eye makes reflexive saccades to salient stimuli within a scene, or ‘top down’ when a viewer uses their volitional control to direct where the eye should look, and both types of saccade are important for understanding how we interacted with complex scenes in everyday life. For example, on entering a café we might casually gaze at the wonderful variety of cakes with reflective saccades to all the highly colourful icings; but when a friend says to ‘try the chocolate cake’ we direct our eyes only to cakes of chocolate brown colour using volitional saccades. Interestingly, these different types of saccadic eye movements are likely to involve different cortical processing of information (Martinez-Conde et al. 2004), potentially allowing for complex multi modal processing that incorporates the rich and dynamic environment experienced when viewing a movie. It is likely that both these mechanisms operate whilst subjects view a film, and the extent to which mechanism dominates during a particular film sequence may depend upon factors like visual design, narrative, audio input and cinema graphic style, as well as individual experience or demographic profile of observers.

The fact that we typically only perceive the world in low resolution at any one point in time can be easily illustrated with an eye chart in which letters of different parts of our visual field are scaled to make the letters equally legible when a subject fixates their gaze on a central fixation spot, or simulated by selectively Gaussian blurring a photograph such that it matches how we see detail at any one point in time (Figure 1). Human subjects typically shift their gaze about 3 times a second in many real world type scenarios in order to build up a detailed representation of our visual environment (Martinez-Conde et al. 2004; Tatler et al. 2014; Yarbus 1967). To efficiently direct the fovea to different parts of a visual scene, the human eye usually makes saccades, which also require a shift of the observer’s attention (Kustov et al. 1996; Martinez-Conde et al. 2004). One way to record subject gaze is to use a video-based eye tracking system that makes use of the different reflective properties of the eye to infrared radiation (Duchowski 2003), using a wavelength of radiation that is both invisible to the test subject and does not damage the eye. This non invasive technique thus enables very natural behavioural responses to be collected from a wide range of subjects. When the eye is illuminated by infrared light, which is typically provided by the eye tracking equipment, it enters the lens and is strongly reflected back by the retina providing a high contrast signal for an infrared camera to record, whilst some of the carefully placed infrared lights also reflect off the cornea of the eye which provides a constant references signal to enable eye tracker software to disentangle minor head movements from the actual eye movements of a subject. A subject is first calibrated to grid stimulus of known spatial dimensions (Dyer et al. 2006), and then when test images are viewed it is possible to accurately quantify the different regions of a scene to which the subject pays attention, the sequence order off this attention, and thus also what features of a scene may escape the direct visual attention of a viewer (Duchowski 2003). The use of this non invasive technique then directly enables the measurement of subject attention to the different components of a stimulus (Figure 2), and has been extensively employed for static images for many fields including medicine, forensics, face processing, advertising, sport and perceptual learning (Dyer et al. 2006; Horsely 2014; Russo et al. 2003; Tatler 2014; Vassallo et al. 2009; Yarbus 1967).

Figure 1. The way our eye samples the world means that only the central fovea region is viewed in detail. The left had image shows letters scaled to equal legibility when a subject fixates gaze to the central dot, and the right hand image is a photographic reconstruction of how an eye would typically resolve detail of the Sydney Harbour Bridge at one point in time.

Figure 1. The way our eye samples the world means that only the central fovea region is viewed in detail. The left had image shows letters scaled to equal legibility when a subject fixates gaze to the central dot, and the right hand image is a photographic reconstruction of how an eye would typically resolve detail of the Sydney Harbour Bridge at one point in time.

In recent times there has been a growing appreciation that to understand how the human visual system and brain processes complex information, the use of moving images has significant advantages since these stimuli may more accurately represent the very complex and dynamic visual environments in which we typically operate (Tatler et al. 2011). For example, when the eyes of a subject are tracked whilst driving a car, it can be observed that the gaze of subjects tends to be directed ahead of the responding action that a driver will take (Land and Lee 1994), and in other real life activities like making a cup of tea test subjects also tend to fixate on particular objects before an action like picking up an object (Land et al. 1999). This shows visual processing is often dynamic and may be influenced by top down volitional goals of a subject, whilst static images may not always best represent how subjects’ actions are informed by visual input in a dynamic situation (Tatler 2014). Interestingly, the capacity of subjects at visually anticipating tasks may link to performance or experience at a given action, as elite cricket batsmen viewing action can more efficiently predict the location that a ball will bounce in advance of the event, providing significant advantages for facing fast bowling where decisions must be made very quickly and accurately (Land and McLeod 2000). Thus there is evidence that visual perception and eye movements for moving images may be influenced by top down mechanisms and experience, as well as bottom up salience driven mechanisms of visual processing (Tatler 2014).

Subject viewer gaze and attention in dynamic environments can also be significantly influenced by the actions of other people who may be viewed within a scene. For example, when viewing a simple magic trick where an experienced magician in a video waves a hand to make an object disappear, the gaze direction of subjects viewing the video is heavily influenced by the actual gaze direction of the magician in the video clip (Tatler and Kuhn 2007). If the magician appears to pay attention to his waving hand then subjects follow this misdirection of viewer attention and the magi trick, performed with the other hand, cannot be detected and the magic trick is successful. However; this pattern is changed if the magician’s gaze attends the hand performing the apparent magical act, and then trick is readily detected by observes. This simple but highly effective demonstration shows how viewer experience is not only driven by reflexive bottom up salience signals present in complex images, but several top down and/or contextual factors may influence visual behaviour. The effect of dynamic complex environments affecting subject eye movements has also been observed in demonstrations of how people might encounter each other and either divert or attend their gaze depending upon prior experience, the perception of threat and/or chance of a collision (Jovancevic-Misic and Hayhoe 2009). Other evidence of top down type influences on observer gaze behaviour come from our understanding of how instructions or narrative may influence where a subject looks (Land and Tatler 2009; Tatler 2014; Yarbus 1967). For example, in the classic eye movement experiments done by Yarbus (1967), in which he presented to test subjects static images, a variety of different instructions were provided for viewing the painting ‘The Unexpected Visitor’ by Ilya Repin. These instructions included estimating the material circumstances of subjects within the painting, or the age of the subjects, the subject’s clothing; and a very different set of saccades and fixations was observed for different instructions or a free view situation that might be taken as a condition mainly driven by bottom up salience factors on perception; showing that top down view goals strongly influenced the way in which eye gaze is directed (Tatler 2014; Yarbus 1967).

Eye Tracking For Understanding Dynamic and Complex Visual Information

Whilst these clever, and comparatively complex, evaluations of visual perception are currently teaching us a lot about human visual performance and viewer experience, the current rapid advances in computer technology and eye tracking are now starting to enable the testing of how subjects view very complex dynamic environments as encapsulated in movies (Mital et al. 2011; Smith and Henderson 2008; Smith and Mital 2013; Smith et al. 2012; Treuting 2006; Vig et al. 2009). This potentially allows for new insights into increasingly real world type viewer experience, how the visual system potentially processes very complex information, and how viewers from different demographics may interpret information content in films. For example, some recent work has looked at subject viewer attention within movies and observed high levels of attention to faces (Treuting 2006), revealing consistent behaviours to previous work that used static images (Vassallo et al. 2009; Yarbus 1967), but a wealth of opportunities are becoming available for better understanding real world visual processing.

Figure 2. When we view an image our eyes often fixate on key areas on interest for short periods of about a third of a second, and then the eyes may make ballistic shifts (saccades) to other features. When a typical subject viewed sequential images from the film 'UP', fixations (green circles) mainly centred on the respective faces of main characters, whilst lines between fixations show the direction of respective saccades [image from Craig Batty, Adrian Dyer, Claire Perkins and Jodi Sita. 2014 Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative ( Bloomsbury, 2015) with permission].

Figure 2. When we view an image our eyes often fixate on key areas on interest for short periods of about a third of a second, and then the eyes may make ballistic shifts (saccades) to other features. When a typical subject viewed sequential images from the film ‘UP’, fixations (green circles) mainly centred on the respective faces of main characters, whilst lines between fixations show the direction of respective saccades [image from Craig Batty, Adrian Dyer, Claire Perkins and Jodi Sita. 2014 Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative ( Bloomsbury, 2015) with permission].

A current issue of how to interpret eye movement data for subjects viewing a film is how such a large volume of data can be managed and statistically separated to try and interpret viewer experience. One initial solution is gaze plot analyses which shows the average attention of a number of subjects to a particular scene (Fig. 3). Investigations on still images using gaze plot analyses have indicated a strong tendency for central bias to an image that is largely independent of factors like subject matter or composition (Tatler 2007). Studies on moving images appear to confirm a tendency for viewing restricted parts of the overall image in detail (Dorr et al. 2010; Goldstein et al. 2007; Mital et al. 2011; Smith and Henderson 2008; Tosi et al. 1997), which may hold important implications for data compression type algorithms where large amounts of image data may be streamed to a variety of different mobile viewing devices such that certain information does not have to be displayed at high resolution due to the resolution of the human eye (Fig. 1), or even certain parts of the movie may be modified to enhance viewing experience for visually impaired viewers (Goldstein et al. 2007). Despite the qualitative value of gaze plot displays, quantitative analyses can be better facilitated by allocating Areas of Interest (Fig. 4) to certain components of a scene that are hypothesised to be of high value for dissecting different theories about information processing of moving images. For example, one of the current issues in understanding how eye tracking can inform film culture, and how movies can be a useful stimulus for understanding visual behaviour is having a method that can explore the potential effects of narrative which is a hypothesised top down or endogenous control on viewer gaze behaviour when subjects are freely viewing a movie to enable natural behaviour (Smith 2013).

FIGURE 3: Gaze plot shows the mean attention of a number of viewers (n=12) to a particular scene. In this case faces capture most attention consistent with previous reports (Yarbus 1967, Vassallo et al. 2009). [image from Craig Batty, Adrian Dyer, Claire Perkins and Jodi Sita. 2014 Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative ( Bloomsbury, 2015) with permission].

FIGURE 3: Gaze plot shows the mean attention of a number of viewers (n=12) to a particular scene. In this case faces capture most attention consistent with previous reports (Yarbus 1967, Vassallo et al. 2009). [image from Craig Batty, Adrian Dyer, Claire Perkins and Jodi Sita. 2014 Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative ( Bloomsbury, 2015) with permission].

Recently one study has tackled this question by using a montage sequence from the animation movie ‘Up’ (Pete Docter, 2009) to explore if it is possible to collect empirical evidence that supports modulation between bottom up and top down mechanisms. The animation montage is a high value study case as it encapsulates a lifetime of narrative story within a 262s montage film sequence that contains no dialogue (Batty 2011), and the overall salience of the two principle characters ‘Carl’ and ‘Elli’ depicted within the montage is somewhat consistently matched due to the control exhibited by animation production. For example, in an initial opening scene where these two characters are first encountered by subjects viewing the film there was an almost identical percentage of time to Carl and Elli respectively; however, as the montage unfolds with a life story narrative of marriage, dreams of children, miscarriage, dreams of travel, illness and death; there is a significant difference in the amount of attention paid to the respective characters by viewers at different stages in the montage (Batty et al. 2014). This suggests that influences of top down type processing on the overall salience of complex images as have been observed in some studies using short motion displays in laboratory type conditions (Jovancevic-Misic and Hayhoe 2009; Tatler and Kuhn 2007; Tatler 2014), is a promising avenue of investigation for movie studies if it is possible to design protocols for controlling the many factors that can influence image salience (Parkhurst et al. 2002; Martinez-Conde et al. 2004; Tatler et al. 2014).

FIGURE 4. Areas of interest can be programmed to quantify the number and respective duration of fixations to key components within a scene of a movie, which may allow for the dissection of how factors like narrative influence viewer behaviour. saccades [from Craig Batty, Adrian Dyer, Claire Perkins and Jodi Sita. 2014. Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative (Bloomsbury, 2015) with permission].

FIGURE 4. Areas of interest can be programmed to quantify the number and respective duration of fixations to key components within a scene of a movie, which may allow for the dissection of how factors like narrative influence viewer behaviour. saccades [from Craig Batty, Adrian Dyer, Claire Perkins and Jodi Sita. 2014. Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative (Bloomsbury, 2015) with permission].

Yet because eye tracking can only tell us part of the story – that is, what people look at, and not how and why these ways of looking emerge and are enacted – other qualitative research approaches such as those used in visual and sensory ethnography (Pink 2013; Pink 2015) are needed to put eye tracking data into context. This involves approaching viewing and the practices of vision that it entails as situated activities, and as part of a broader experiential repertoire beyond the eye. The subjectivity and selectivity of viewing that the studies outlined above have evidenced, once documented and measured, can only be properly understood as emergent from particular (and always complex) environmental conditions and embodied experiences. In the next section we therefore turn to anthropological approaches to vision and the environment in order to show how this might be achieved. However before proceeding we note that when working across disciplines, there is inevitably a certain amount of conceptual slippage. Here this means that whereas we have ended the previous paragraph by suggesting that eye tracking enables our understanding of how complex environmental information is processed, in the next sections we refigure this way of thinking to consider how human perception and viewer experience are constituted in relation to the affordances of complex environments of which they are also part.

Situating Viewing as Part of Complex Environments

The environment, as a concept, is slippery and is used to different empirical and political ends in different contexts. As the anthropologist Tim Ingold has emphasised, in contemporary discourses ‘the environment’ is often referred to as an entity and as something that we exist as separate from. Indeed, this idea is present in our discussion above whereby we have considered how eye tracking studies might better show us how we process complex environmental information. As Ingold expresses it this means we are ‘inclined to forget that the environment is, in the first place, a world we live in, and not a world we look at’. He argues that ‘We inhabit our environment: we are part of it; and through this practice of habitation it becomes part of us too’ (Ingold T 2011). Following this approach the environment can be understood as an ecology that humans are part of and with which we, and the ways we view, see and experience are mutually constituted. This however does not simply mean that ‘we’ as humans are encompassed by the environment, it means that the environment is co-constituted by us and our relationships with other constituents, which for our purposes in this article we would emphasise includes, film, images, art, technologies, other humans, the weather, the built environment (as well as much more). As advanced by Ingold (Ingold 2000, 2010) and art historian Barbara Stafford (Stafford 2006), approaches that critique linguistic and semiotic studies invite an analysis which acknowledges that – as Stafford puts it – ‘when you open your eyes and actively interrogate the visual scene, what you see is that aspect, or the physical fragments, of the environment that you perform’ (Stafford 2006). This however means also that the experience of film does not simply involve us looking at something that is external to us, but it is through the affordances of film that, in relation to the other constituents of our environments/worlds that viewing becomes meaningful to us. In this interpretation the use of ‘we’ derives from the development of a universal theory of human perception and our relationship to a (complex) environment. Yet, as we explain in the next section this rendering does not dismiss the idea that different people may often perceive the same information differently, and indeed to the contrary invites us to study precisely how and why difference emerges.

If we take Ingold’s approach further, to focus in on how meanings are generated through our engagements with and experiences of visual images, we can gain an appreciation of how the measurement and monitoring patterns that emerge from eye tracking studies are materialisations or representations not just of how the eye (or the mind) responds to the moving image. Rather they can be understood as standing for (but not actually explaining the meaning of) what people do with the moving image. Building on philosophical and other traditions emerging from the work of Merleau-Ponty, Gibson and Jonas, Ingold, has argued that human perception, learning and knowing emerge from movement, specifically as we move through our environments and engaging with the affordances of those other things and processes we encounter (Ingold 2000). With regard to art, he has used this approach to suggest that therefore …

Should the drawing or painting be understood as a final image to be inspected and interpreted, as is conventional in studies of visual culture, or should we rather think of it as a node in a matrix of trails to be followed by observant eyes? Are drawings or paintings of things in the world, or are they like things in the world, in the sense that we have to find our ways through and among them, inhabiting them as we do the world itself? (Ingold 2010; p16)

If we transfer this idea to the question of how we view film we might then ask the question of how we, as viewers, inhabit film? And what then eye tracking studies can tell us about these forms of habitation. If we see the relationship established between the viewer (‘s eyes) and the film by eye tracking visualisations such as those demonstrated in the earlier sections of this article, we can begin to think of how the movement of the eye and the movement of the film become entangled. Indeed while the film and the eye will both inevitably continue to move, the question becomes not simply how the composition and action in the screen influences the movement of the eye, but rather how the eye selects the aspects of the composition and action of the screen with which to move. By taking this perspective, we are able to remove something of the technological determinism that underpins assumptions that eye tracking studies might better enable film, advertising organisations to better influence viewing behaviours. Instead it directs us towards considering what eye tracking studies might tell us about what people do when they view, and how this can inform us about how they inhabit a world of which film and more generally the moving image as an ubiquitous presence.

The work and arguments discussed thus far in this section have focused on interpreting the question of how, at a general level, people see when they are viewing moving images. The theories advanced as yet however neither explain nor discuss the usefulness of attending to the patterning of eye tracking studies. Moreover the examples and visualisations we have shown of eye tracking studies in the earlier sections of this article were undertaken with a sample of people who were likely to have similar viewing perspectives, and as might therefore be expected showed distinct patterns in the ways that people view particular information. Indeed the data that would be needed to tell us to what extent such viewing patterns were universal – that is supported by studies and theories of the ways in which the human brain processes – and to what extent they were situationally and biographically constituted for this particular group of participants still does not exist as far as we know. Such work would be of high value given the increasing globalisation of both entertainment industries and forms of activism that use visual media; where films may be distributed in markets distant to the original context to which audience experience is understood. Indeed, studies of how people learn to look and know, undertaken in culturally specific contexts definitely reveal that where we look and what we see is contingent on processes of learning and apprenticeship, and therefore specific to complex environments.

Vision, Learning and Knowing

Eye tracking studies have shown us that there are sometimes similarities and patterns in the ways people view and remember complex images (Norton and Stark 1971), although if present such patterns are easily changed through instruction (Yarbus 1967; Tatler 2014). We have seen in the earlier sections of this manuscript that participants in studies have consistently fixed their gaze on the faces of film characters (Figs 2, 3), and that visual attention may become focused on a film character whose story line commands (or affords) particularly powerful affective and/or empathetic connections for viewers. Further eye tracking research would be needed to underpin any proposals that such ways of viewing are both gendered and culturally specific, however existing research in visual and media anthropology indicates that this is likely to be the case. Two bodies of literature are relevant here. First the applied visual and media anthropology literature, and second the anthropology of vision.

Applied visual and media anthropology studies (Pink 2007) focus on using anthropological understandings of media, along with audiovisual interventions (often in the form of filmmaking processes and film products) to work towards new forms of social and public awareness, and societal change. This work draws on and advances a strand in film studies developed in the work of Laura Marks, who has advanced the idea of the ‘embodied viewing experience’ (2000: 211). Marks, whose work focuses on intercultural cinema has argued that as ‘a mimetic medium’ cinema is ‘capable of drawing us into sensory participation with its world’ (Marks 2000: 214). The notion of empathy as a route towards creating intercultural understanding through film is also increasingly popular in the visual anthropology literature (discussed in Pink 2015). While on the whole there has been insufficient research into the ways in which people view intervention films of this kind, one example that has been undertaken implies how viewer attention, and importantly viewer’s capacity to engage with and remember film narrative can depend on the ways in which they are able to affectively or empathetically engage with the experiences of film characters. Susan Levine’s media anthropology study of how viewers discussed a film made as part of a South African HIV/AIDs intervention campaign, and which drew on local narratives to communicate the central message, is a good example (Levine 2007). Levine (unsurprisingly) found that participants engaged with the stories of film characters that followed locally relevant narratives, thus generating important lessons for filmmaking campaigns of this kind, where it is often difficult to communicate generic health messages to local audiences. The bridge between this type of anthropological understanding and a capacity to map viewer attention to faces and expressions within visual representations (Vassallo et al. 2009) may allow for more comprehensive understandings of why film is such a powerful medium for communication.

Anthropological studies of vision provide further evidence of the importance of attending to how seeing is situated. Indeed when vision is understood as a practice, rather than as a behavior, it is not just a situated practice, but it is a practice that is learned through participation. The anthropologist Cristina Grasseni has developed a theory of what she calls ‘skilled vision’ though which to explain this (Grasseni 2004, 2007, 2011), as she puts it:

The “skilled visions” approach considers vision as a social activity, a proactive engagement with the world, a realm of expertise that depends heavily on trained perception and on a structured environment (Grasseni 2011).

Emphasizing that skilled visions are ‘positional, political and relational’ as well as sensuous and corporeal, Grasseni points out that ‘Because skilled visions combine aspects of embodiment (as an educated capacity for selective perception) and of apprenticeship, they are both ecological and ideological, in the sense that they inform worldviews and practice’ (Grasseni 2011). As Pink has shown through her work on the Spanish bullfight, what one sees when viewing the performance is highly contingent on how one has learned to view, ones own empathetic embodied ways of sensorially and affectively ‘feeling’ the performance at which a visual representation was created, or how one’s existing ways of knowing and understanding the world can inform perception (Pink 1997; 2011). For example, consider the different ways in which Figure 5, or a film sequence around the same performance, would be interpreted by a bullfighting fan and an animal rights activist. Each will have learned how and what to know about this performance through different trajectories. Whilst an eye tracking investigation of respective subjects might show somewhat similar patterns (especially if bottom up mechanisms dominate), the semantic interpretation of the visual input by respective viewers may be completely different. How such information content might be assessable, or not, through evaluation of bottom up or top down type mechanisms involved with visual processing will be a major challenge for interpretation of information as complex as can typically be perceived in a movie.

Conclusion

Figure 5. How emotive content, as is common in many films, may influence the perception of visual images even if the same information is present to viewers remains a major topic for exploration. For example, we know that the bullfight is interpreted, and affectively experienced, very differently when viewed by bullfight fans and animal rights activists. We also know that learning how to view the bullfight, as a bullfight fan, is a process of cultural apprenticeship (see for example Pink 1997). Consider how for the above image the action of a bull fight could promote very different visual behavior depending upon cultural context, whether a subject was a bullfighting fan and an animal rights activist, or the representation was depicted as animation instead of real life, or motion compared to a still image. Copyright: Sarah Pink.

Figure 5. How emotive content, as is common in many films, may influence the perception of visual images even if the same information is present to viewers remains a major topic for exploration. For example, we know that the bullfight is interpreted, and affectively experienced, very differently when viewed by bullfight fans and animal rights activists. We also know that learning how to view the bullfight, as a bullfight fan, is a process of cultural apprenticeship (see for example Pink 1997). Consider how for the above image the action of a bull fight could promote very different visual behavior depending upon cultural context, whether a subject was a bullfighting fan and an animal rights activist, or the representation was depicted as animation instead of real life, or motion compared to a still image. Copyright: Sarah Pink.

Bringing together measurement and monitoring data with anthropologically informed ethnographic ways of knowing, which are always collaboratively crafted and sensorially and tacitly known is increasingly common. For instance in energy research a number of projects seek to combine ethnographic and energy consumption measurement data (Cosar et al. 2013). Such an approach has not yet been integrated in eye tracking studies of movies, yet this would be the next step if we were to want to understand better the significance and relevance of the types of data and knowledge that eye tracking studies can offer us, for understanding film audiences. This however presents certain challenges, which both impinge on, but are not necessarily unique to, the use of eye tracking data in audience research. The first challenge is to generate sufficient interdisciplinary understanding between the approaches involved. This article has intended to initiate that process. That is it has explained how eye tracking and anthropological-ethnographic (that is at once theoretical and practical) approaches offer different, and differently theorised perspectives on the ways in which people look at and participate in the viewing of film. It has simultaneously however suggested that these different approaches and disciplines offer something to each other that enable new questions to be asked, and therefore is able to develop deeper understandings of how audiences view film.

Future work testing human visual behaviour with complex stimuli as are typically present in movies may help build our understanding of how humans sometimes process very complex information to build an understanding of our surrounding world, but sometimes also miss salient information in complex moving images such as the Gorillas in our midst study. Current theories suggests that perceptual blindness to salient and recognisable stimuli when our attention is captured by other competing stimuli that impose a cognitive load to process (Simons and Chabris 1999; Levin et al. 2000; Memmert 2006), but more fully exploring effects of narrative or instructions, character gaze and other potential top down mechanisms will likely be fruitful contributions to our knowledge on perceptual blindness. Indeed, as discussed above in relation to anthropological and ethnographic factors, the potential role of factors like experience do appear to modulate the ability of subjects to detect a gorilla in a perceptual blindness type test (Memmert 2006), potentially suggesting that future investigations on eye tracking and movies should consider the broad range of human experience that can influence our perception. This type of research is likely to also provide for richer understandings in some ethnographic studies as researchers will have, possibly for the first time, access to precise quantitative data on whether an observer actually failed to even look at certain objects in a scene; or indeed if such information, like an unexpected gorilla in a basketball game, was viewed but not directly perceived (Memmert 2006). Many individual scenes within a film are typically short of about 4s duration and so it is often only possible for viewers to process a small percentage of the entire visual presentation in detail, especially in cases where movies are subscripted (Smith 2013). This means that elements of a film that might be essential to the complete comprehension of narrative story line may be easily missed by a percentage of an audience depending upon their individual knowledgebase, linguistic skills, attention and motivation; and eye tracking potentially offers film makers with a useful vehicle to test different demographic groups to better understand how different components of scenes might be constructed to enhance viewer experience, and also build our understanding of how we process very complex environmental information.

 

 

Acknowledgements. We are very grateful to Dr Craig Batty, Dr Claire Perkins and Dr Jodi Sita for discussions and permission to use images from their collaborative work with one of us (AGD), and for broader discussions with members of the Eye Tracking of the Moving Image research group. AGD acknowledges funding support from the Australian Research Council (LE130100112) for eye tracking equipment. We are grateful to Dr Lalina Muir for her careful proofreading of the manuscript.

 

REFERENCES

Batty, Craig. 2011. Movies That Move Us: Screenwriting and the Power of the Protagonist’s Journey. Basingstoke: Palgrave Macmillan.

Batty, Craig, Dyer, Adrain, G., Perkins, Claire, and Sita, Jodi. 2015. Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative (Palgrave, forthcoming)

Cosar Jorda, P, Buswell, RA, Webb, LH, Leder Mackley, K, Morosanu, R, and Pink, Sarah. 2013. ‘Energy in the home: Everyday life and the effect on time of use.’ In The Proceedings of the 13th International Conference on Building Simulation 2013. Chambery, France. 25-28/8/2013.

Docter, P. 2009. Up. Disney-Pixar Motion Film.

Dorr, M, Martinetz, T, Gegenfurtner, KR, and Barth, E. 2010. ‘Variability of eye movements when viewing dynamic natural scenes.’ Journal of Vision 10 (28): 1-17.

Duchowski, Andrew. 2003. Eye tracking methodology: theory and practice. London: Springer-Verlag.

Dyer, Adrian, G., Found, Brian, and Rogers, Doug. 2006. ‘Visual attention and expertise for forensic signature analysis.’ Journal of Forensic Science 51: 1397–1404.

Goldstein, Robert, B., Woods, Russell,L., and Peli, Eli. 2007. ‘Where people look when watching movies: Do all viewers look at the same place?’ Computers in Biology and Medicine 37 (7): 957-964.

Grasseni, Cristina. 2004. ‘Video and ethnographic knowledge: skilled vision in the practice of breeding.’ In Working Images, edited by S Pink, L Kürti, and AI Afonso, 259-288. London: Routledge.

Grasseni, Cristina. 2007. Skilled Visions. Oxford: Berghahn.

Grasseni, Cristina. 2011. ‘Skilled Visions: Toward an Ecology of Visual Inscriptions.’ In Made to be Seen: Perspectives on the History of Visual Anthropology, edited by M. Banks and J. Ruby. Chicago: University of Chicago Press.

Horsely, Mike. 2014. ‘Eye Tracking as a Research Method in Social and Marketing Applications.’ In Current Trends in Eye Tracking Research, edited by M Horsley et al., 179-182. Springer, London.

Ingold, Tim. 2000. The Perception of the Environment. London: Routledge.

Ingold, Tim. 2010. ‘Ways of mind-walking: reading, writing, painting.’ Visual Studies, 25 (1): 15–23

Ingold, Tim. 2011. Being Alive. Oxford: Routledge. p 95.

Jovancevic-Misic, Jelena, and Hayhoe, Mary. 2009. ‘Adaptive Gaze Control in Natural Environments.’ Journal of Neuroscience 29 (19): 6234–6238. DOI:10.1523/JNEUROSCI.5570-08.2009.

Kustov, Alexander, A., and Robinson, David Lee. 1996. ‘Shared neural control of attentional shifts and eye movements.’ Nature 384: 74–77.

Levine, Susan. 2007. ‘Steps for the Future: HIV/AIDS, Media Activism and Applied Visual Anthropology in Southern Africa.’ In Visual Interventions, edited by S. Pink, 71-89. Oxford: Berghahn.

Marks, Laura. 2000. The Skin of the Film: Intercultural Cinema, Embodiment, and the Senses. Durham and London: Duke University Press

Martinez-Conde, Susana, Macknik, Stephen, L., and Hubel, David, H. 2004. ‘The role of fixational eye movements in visual perception.’ Nature Neuroscience 5: 229–240.

Memmert, Daniel. 2006. ‘The effects of eye movements, age, and expertise on inattentional blindness.’ Consciousness and Cognition 15 (3): 620–627.

Mital, Parag, K., Smith, Tim,J., Hill, Robin, L., and Henderson, John, M. 2011. ‘Clustering of gaze during dynamic scene viewing is predicted by motion.’ Cognitive Computation 3, 5–24.

Nodine. Calvin, F., Mello-Thoms. Claudia, Kundel. Harold, L., and Weinstein, Susan, P. 2002. ‘Time course of perception and decision making during mammographic interpretation.’ American Journal Roentgenol 179: 917–923

Norton, David, and Stark, Lawrence. 1971. ‘Scanpaths in eye movements during pattern perception.’ Science 171: 308–311.

Parkhurst, Derrick, Law, Klinton, and Niebur, Ernst. 2002. ‘Modeling the role of salience in the allocation of overt visual attention.’ Vision Research 42: 107–123.

Pashler, Harold. 1998. Attention. Hove, UK: Psychology Press Ltd.

Russo, Francesco, Pitzalis, Sabrina, and Spinell, Donatella. 2003. ‘Fixation stability and saccadic latency in elite shooters.’ Vision Research 43: 1837–1845.

Pink, Sarah. 1997. Women and Bullfighting. Oxford: Berghahn.

Pink, Sarah. 2007. (ed) Visual Interventions. Oxford: Berghahn.

Pink, Sarah. 2011. ‘From Embodiment to Emplacement: re-thinking bodies, senses and spatialities.’ In Sport, Education and Society (SES), special issue on New Directions, New Questions. Social Theory, Education and Embodiment 16(34): 343-355.

Pink, Sarah. 2013. Doing Visual Ethnography, 3rd edition. London: Sage.

Pink, Sarah. 2015 Doing Sensory Ethnography, 2nd edition London: Sage.

Simons, Daniel, J., and Chabris, Christopher, F. 1999. Gorillas in our midst: sustained inattentional blindness for dynamic events. Perception 28(9): 1059-1074.

Smith, Tim. J., and Henderson, Jordan. 2008. ‘Edit blindness: The relationship between attention and global change blindness in dynamic scenes.’ Journal of Eye Movement Research 2: 1–17.

Smith, Tim, J. 2013. ‘Watching you watch movies: Using eye tracking to inform cognitive film theory.’ In Psychocinematics: Exploring Cognition at the Movies edited by A. P. Shimamura, 165-191. New York: Oxford University Press

Smith, T, Levin, D, and Cutting J. 2012. ‘A Window on Reality: Perceiving Edited Moving Images.’ Current Directions in Psychological Science 21(2): 107-113. doi: 10.1177/0963721412437407

Smith, Tim, j., and Mital, Parag, K. 2013. ‘Attentional synchrony and the influence of viewing task on gaze behaviour in static and dynamic scenes.’ Journal of Vision 13 (8): 16.

Stafford, Barbara Maria. 2006. Echo Objects: the Cognitive Work of Images. Chicago: University of Chicago Press.

Tatler, Ben, W. 2007. ‘The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions.’ Journal of Vision 7(14): 4, 1–17. http:// www.journalofvision.org/content/7/14/4, doi:10.1167/ 7.14.4.

Tatler, Ben, W. 2014. ‘Eye Movements from Laboratory to Life.’ In Current Trends in Eye Tracking Research edited by Horsley et al., p17-35.

Tatler, Ben, W., and Kuhn, Gustav. 2007. ‘Don’t look now: The magic of misdirection.’ In Eye Movements: A window on mind and brain, edited by R van Gopel, M Fischer, W Murray and R Hill, 697–714. Amsterdam: Elsevier.

Tatler, Ben, W., Hayhoe, Mary, M., Land, Michael, F., and Ballard, Dana, H. 2011. ‘Eye guidance in natural vision: Reinterpreting salience.’ Journal of Vision 11 (5): 1–23. http://www.journalofvision.org/content/11/5/5, doi:10.1167/11.5.5.

Tatler, Ben, W., Kirtley, Claire, Macdonald, Ross. G., Mitchell, Katy, MA., and Savage, Steven, W. 2014. ‘The Active Eye: Perspectives on Eye Movement Research.’ In Current Trends in Eye Tracking Research, 3-16. DOI 10.1007/978-3-319-02868-2_16 Print ISBN 978-3-319-02867-5 Online ISBN 978-3-319-02868-2

Treuting, Jennifer. 2006. ‘Eye tracking and cinema: A study of film theory and visual perception.’ Society of Motion Picture and Television Engineers 115 (1): 31-40.

Tosi, Virgilio, Mecacci, Luciano, and Pasquali, Elio. 1997. ‘Scanning eye movements made when viewing film: Preliminary observations.’ International Journal of Neuroscience 92 (1/2): 47-52.

Vassallo, Suzanne, Cooper, Sian, LC., and Douglas, Jacinta, M. 2009. ‘Visual scanning in the recognition of facial affect: Is there an observer sex difference?’ Journal of Vision 9: 1-10.

Vig, Eleonora, Dorr, Michael, and Barth, Erhardt. 2009.’ Efficient visual coding and the predictability of eye movements on natural movies.’ Spatial Vision 22 (2): 397-408.

Yarbus Alfred, L. 1967. Eye Movements and Vision. New York: Plenum.
BIOS:

Adrian Dyer is an Associate Professor in Media and Communication at RMIT University (Australia) investigating vision in complex environments. He is an Alexander von Humboldt Fellow (Germany) and a Queen Elizabeth II Fellow (Australia), and has completed postdoctoral positions at La Trobe University and Monash University (Australia), Cambridge University (UK), and Wuerzburg and Mainz Universities (Germany).

Sarah Pink is Professor of Design and Media Ethnography at RMIT University (Australia). She is visiting/guest Professor at Halmstad University (Sweden), Loughborough University (UK), and Free University Berlin (Germany). Her most recent books include Situating Everyday Life (2012), Doing Visual Ethnography 3rd edition (2013) and Doing Sensory Ethnography 2nd edition (2015).

How We Came To Eye Tracking Animation: A Cross-Disciplinary Approach to Researching the Moving Image – Craig Batty, Claire Perkins, & Jodi Sita

$
0
0

Abstract

In this article, three researchers from a large cross-disciplinary team reflect on their individual experiences of a pilot study in the field of eye tracking and the moving image. The study – now concluded – employed a montage sequence from the Pixar film Up (2009) to determine the impact of narrative cues on gaze behaviour. In the study, the researchers’ interest in narrative was underpinned by a broader concern with the interaction of top-down (cognitive) and bottom-up (salient) factors in directing viewers’ eye movements. This article provides three distinct but interconnected reflections on what the aims, process and results of the pilot study demonstrate about how eye tracking the moving image can expand methods and knowledge across the three disciplines of screenwriting, screen theory and eye tracking. It is in this way both an article about eye tracking, animation and narrative, and also a broader consideration of cross-disciplinary research methodologies.

 

Introduction

Over the past 18 months, a team of cross-disciplinary researchers has undertaken a pilot eye tracking and the moving image study that has sought to understand where spectators look when viewing animation.[i] The original study employed eye tracking methods to record the gaze of 12 subjects. It used a Tobii X120 (Tobii Technology, 2005) remote eye tracking device which allowed viewers to watch the animation sequence on a widescreen PC monitor at 25 frames per second, with sound. The eye tracker pairs the movements of the eye over the screen with the stimuli being viewed by the participant. For each scene viewed, the researchers selected areas of interest; and for these areas, all of the gaze data, including the number and duration of each fixation, was collected and analysed.

Using a well-known montage sequence from the Pixar film Up! (2009), this pilot study focussed on narrative with the aim of discerning whether story cues were instrumental in directing spectator gaze. Focussing on narrative seemed to be useful in that as well as being an original line of enquiry in the eye tracking context, it also offered a natural connection between each of our disciplines and research experiences. The study did not take into account emotional and physiological responses from its participants as a way of discerning their narrative comprehension. Nevertheless, what we found from our data was that characters (especially their faces), key (narrative) objects and visual/scenic repetition seemed to be core factors in determining where they looked.[ii]

In the context of a montage sequence that spans around 60 years of story time, in which the death of the protagonist’s wife sets up the physical and emotional stakes of the rest of the film, it was clear that narrative meaning relating to a character’s journey/arc is important to viewers, more so (in this study) than peripheral action or visual style, for example. With regards to animation specifically, a form ‘particularly equipped to play out narratives that solicit […] emotions because of its capacity to illustrate and enhance interior states, and to express feeling that is beyond the realms of words to properly capture’ (Wells, 2007: 127), the highly controlled nature of the sequence from which the data was drawn seems to suggest that animation embraces narrative techniques fully to control viewer attention.

In this article, three researchers from the team – A, a screenwriter, B, a screen scholar and C, an eye tracking neuroscientist – discuss the approaches they took to conducting this study. Each of us came to the project armed with different expertise, different priorities and a different set of expectations for what we might find out, which we could then take back to our individual disciplines. In this article, then, we purposely use three voices as way of teasing out our understandings before, during and after the study, with the aim of better understanding the potential for cross-disciplinary research in this area. Although other studies in eye tracking and the moving image have been undertaken and reported on, we suggest that using animation with a strongly directed narrative as a test study provides new information. Furthermore, few other studies to date have brought together traditional and creative practice researchers in this way.

What we present, then, is a series of interconnected discussions that draw together ideas from each researcher’s community of thought and practice, guided by the overriding question: how did this study embrace methodological originality and yield innovative findings that might be important to the disciplines of eye tracking and moving image studies? We present these discussions in the format of individual reflections, as a way of highlighting each researcher’s contributions to the study, and in the hope that others will see the potential of disciplinary knowledge in a study such as this one.

How ‘looking’ features in our disciplines, and what we might expect to ‘see’

Researcher A: ‘Looking’ in screenwriting means two things: seeing and reflecting on. By this I mean that a viewer looks at the screen to see what is happening, whilst at the same time reflecting on what they are looking at from on a personal, cultural and/or political level. Some screenwriters focus on theme from the outset: on what they want their work to ‘say’ (see Batty, 2013); some screenwriters focus on plot: on what viewers will see (action) (see Vogler, 2007). What connects these is character. In Aristotelian terms, a character does and therefore is (Aristotle, 1996); for Egri, a character is and therefore does (Egri, 2004). The link here is that what we see on the screen (action) is always performed by a character, meaning that through a process of agency, actions are given meaning, feeding into the controlling theme(s) of the text. In this way, looking at – or seeing – is tied closely to understanding and the feelings that we bring to a text. As Hockley (2007) says, viewers are sutured into the text on an emotional level, connecting them and the text through the psychology of story space.

What we ‘see’, then, is meaning. In other words, we do not just see but we also feel. We look for visual cues that help us to understand the narrative unfolding before our eyes. With sound used to point to particular visual aspects and heighten our emotional states, we bestow energy and emotion in the visuality of the screen, in the hope that we will arrive at an understanding. As this study has revealed, examples include symbolic objects in the frame (the adventure book; the savings jar; the picture of Paradise Falls) that have narrative value in screenwriting because of the meaning they possess (Batty and Waldeback, 2008: 52-3). By seeing these objects repeated throughout the montage, we understand what they mean (to the characters and to the story) and glean a sense of how they will re-appear throughout the rest of the film as a way of representing the emotional space of the story.

Landscape is also something we see, though this is always in the context of the story world (see Harper and Rayner, 2010; Stadler, 2010). In other words, where is this place? What happens here? What cannot happen here? Characters belong to a story world, and therefore landscape also helps us to understand the situations in which we find them. This, again, draws us back to action, agency and theme: when we see landscape, we are in fact understanding why the screenwriter chose put their characters – and us, the audience – there in the first place.

Researcher B: In screen theory, looking is never just looking – never innocent and immediate. The act of looking is the gateway to the experience and knowledge of what is seen on screen, but also of how that encounter reflects the world beyond the screen and our place within it. Looking is over determined as gazing, knowing and being, endlessly charged by the coincidence of eye and I and of real and reel. Psychoanalytic theory imagines the screen as mirror and our identity as a spectatorial effect of recognizing ourselves in the characters and situations that unfold upon it, however refracted. Reception studies seeks out how conversely real individuals encounter content on screen, and how meaning sparks in that meeting—invented anew with every pair of eyes. Television studies emerges from an understanding of a fundamental schism in looking: where the cinematic apparatus enables a gaze, the televisual counterpart can (traditionally) only produce a broken and distracted glance.

All of these theories begin with the act of looking, and are enabled by it in their metaphors, methods and practices. But in no instance is looking attended to as anatomical vision – the process of the “meat and bones” body and brain rather than the metaphysical consciousness. As a scholar of screen theory, my base interest in eye tracking comes down to this “problem”. Is it a problem? Should the biology and theory of looking align? What effects and contradictions arise when they are brought together?

Phenomenological screen theory is a key and complex pathway into this debate, as an approach that values embodied experience, but discredits the ocular—seeking to bring the whole body to spectatorship rather than privilege the centred and distant subject of optical visuality (Marks, 2002: xvi). Vivian Sobchack names film ‘an expression of experience by experience … an act of seeing that makes itself seen, an act of hearing that makes itself heard’ (Sobchack, 1992: 3). Eye tracking shows us the act of seeing – the raw fixations and movements with which screen content is taken in. In the study under discussion here it is this data that is of central interest, with our key questions deriving from what such material can verify about how narrative shapes gaze behaviour. A central question and challenge for me moving forward in this field, though, is to consider this process without ceding to ocularcentrism: that is, without automatically equating seeing to knowing. This ultimately means being cautious about reading gaze behaviour as ‘proof’ of what viewing subjects are thinking, feeling and understanding. This approach will be supported by the inclusion of further physiological measurements.

Researcher C: Interest in vision and how we see the world is an age-old interest, where it has been commonly held that the eyes are the windows to the mind. Where we look is then of great importance, as learning this offers us opportunities to understand more about where the brain wants to spend its time. Human eyes move independently from our heads and so our eyes have developed a specialised operating systems that both allows our eyes to move around our visual environment, and also counteract any movements the head may be making. This has led to a distinct set of eye movements we can study – saccades (the very fast blasts of movement that pivot our eye from focus point to focus point) – and fixations (brief moments of relative stillness where our gaze stops for a moment to allow the receptors in our eye to collect visual information). In addition, only a tiny area of the back of our eyeball, the fovea on the retina, is sensitive enough to gather highly ‘acuitive’ information, thus the brain must drive the eye around precisely in order to get light to fall onto this tiny area of the eye. As such, our eyes movements are an integral and essential part of our vision system.

Eye movement research has seen great advances during the last 50 years, with many early questions examined in the classic work of Buswell (1935) and Yarbus (1967). One question visual scientists and neuroscientists have been, and are still keen to, explore is why we look where we do: what is it about the objects or scene that draws our visual attention? Research over the decades has found that several different aspects are involved, relating to object salience, recognition, movement and contextual value (see Schütz et al., 2011). For animations that are used for learning purposes, Schnotz and Lowe (2008) discussed two major contributing factors that influence the attention-grabbing properties of features that make up this form. One is visuospatial contrast and a second is dynamic contrast; with features that are relatively large, brightly coloured or centrally placed, more likely to be fixated on compared to their less distinctive neighbours; and features that move or change over time drawing more attention.

Eye tracking research, which is now easier than ever to conduct, allows us to delve into examining how these and other features influence us, and is a unique way to gain access to the windows of the mind. Directing this focus to learning more about how we watch films, and in particular to animation, is what drove me to wanting to use eye tracking to better see how people experience these; and to delve into questions such as, what are people drawn to look at, and how might things like the narrative affect the way we direct our gaze?

When looking around a visual world, our view is often full of different objects and we tend to drive our gaze to them so we can recognize, inspect or use them. Not so surprisingly, what we are doing (our task at hand) strongly affects how we direct our gaze; such that as we perform a task, our salience-based mechanisms seem to go offline as people almost exclusively fixate on the task-relevant objects (Hayhoe, 2000; Land et al., 1999). From this, one expectation we have when considering how viewers watch animation is that more than salient features, aspects relating to the narrative components of the viewer’s understanding of the story will be the stronger drive. Another well-known drawcard for visual attention is towards faces, which tend to draw the eye’s attention very strongly (Cerf et al., 2009; Crouzet et al., 2010). For animated films we were interested to see if similar effects would be observed.

Finally, another strong and interesting effect that has been discussed is a tendency for people to have a central viewing bias, in which a large effect on viewing behaviour has been shown to be that people tend to fixate in the centre of a display (Tatler and Vincent, 2009). As this study was moving image screen based, we were keen to compare different scenes and how the narrative affected this tendency.

How we came to the project, and what we thought it might reveal

Researcher A: From a screenwriting perspective, I was excited to think that at last, we might have data that not only privileges the story (i.e., the screenwriter’s input), but that also highlights the minutiae of a scene that the screenwriter is likely to have influenced. This can be different in animation than in live action, whereby a team of story designers and animators actively shape the narrative as the ‘script’ emerges (see Wells, 2010). Nevertheless, if we follow that what we see on screen has been imagined or at least intended by a ‘writer’ of sorts – someone who knows about the composition of screen narratives – then it was rousing to think that this study might provide ‘evidence’ to support long-standing questions (for myself at least) of writing for the screen and authorship. Screenwriters work in layers, building a screenplay from broad aspects such as plot, character and theme, to micro aspects such as scene rhythm, dialogue and visual cues. Being able to ‘prove’ what viewers are looking at, and hoping that this might correlate with a screenwriting perspective of scene composition, was very appealing to me.

I was also interested in what other aspects of the screen viewers might look at, either as glances or as gazes. In some genres of screenwriting, such as comedy, much of the clever work comes around the edges: background characters; ironic landscapes; peripheral visual gags, etc. From a screenwriting perspective, then, it was exciting to think that we might find ways to trace who looks at what, and if indeed the texture of a screenplay is acknowledged by the viewer. The study would be limited and not all aspects could be explored, but as a general method for screen analysis, simply having ideas about what might be revealed led to some very interesting discussions within the team.

Researcher B: All screen theories rest upon a fundamental assumption that different types of content, and different viewing situations, produce different viewing behaviours and effects. Laura Mulvey’s famous theory of the gaze stipulates that classical Hollywood cinema and the traditional exhibition environment (dark cinema, large screen, audience silence) position men as bearers of the look and women as objects of the look, and that avant-garde cinemas avoid this configuration (Mulvey, 1975). New theories of digital cinema speculate upon whether a spectator’s identification with an image is altered when it bears no indexical connection to reality; that is, when the image is a simulated collection of pixels rather than the trace of an event that once took place before a camera (Rodowick, 2007). The phenomenological film theory of Laura Marks suggests that certain kinds of video and multimedia work can engender haptic visuality, where the eyes function like ‘organs of touch’ and the viewer’s body is more obviously involved in the process of seeing that is the case with optical visuality (Marks, 2002: 2-3). It made sense to begin our study into eye tracking by thinking about these different assumptions regarding content and context and formulating methods to analyse them empirically.

For our first project we chose to focus on an assumption regarding spectatorship that is more straightforward and essential than any listed above: namely that viewers can follow a story told only in images. This is an assumption that underpins the ubiquitous presence of the montage sequence in narrative filmmaking, where a large amount of story information is presented in a short, dialogue-free sequence. We hypothesized that by tracking a montage sequence we would be able to ascertain if and how viewers looked at narrative cues, even when these are not the most salient (i.e., large, colourful, moving) features in the scene. The study was in this way designed to start investigating how much film directors and designers can control subjects’ gaze behaviour and top-down (cognitively driven) processes.

The sequence from Up! was chosen in part to act as a ‘control’ against which we could later assess different types of content. The story told in the 4-minute sequence is complex but unambiguous, with its events and emotive power linked by clear relationships of cause and effect. It is in this way a prime example of a classical narrative style of filmmaking, where the emphasis is on communicating story information as transparently as possible (Bordwell, 1985: 160). Our hypothesis was that subjects’ gaze behaviour would be controlled by the tightly directed sequence with its strong narrative cues, and that this study could thereby function as a benchmark against which different types of less story-driven material could be compared later.

Researcher C: A colleague and I set up the Eye Tracking and the Moving Image (ETMI) research group in 2012, following discussions around how evidence was collected to support and investigate current film theory. These conversations grew into a determination to begin a cross-disciplinary research group, initially in Melbourne, to begin working together on these ideas. I had previously been involved in research using eye tracking to study other dynamic stimuli such as decision making processes in sport and the dynamics of signature forgery and detection, and my experience led to a belief that the eye tracker could have enormous potential as a research tool in the analysis and understanding of the moving image. Work on this particular study was inspired by the early aims of a subgroup (of which the other authors are a part), whose members were interested to investigate, in a more objective manner, the effect that narrative cues had on viewer gaze behaviour.

Existing research in our disciplines, and how that influenced our approaches to the study

Researcher A: While there had been research already conducted on eye tracking and the moving image, none of it had focussed on the creational aspects of screen texts: what goes into making a moving image text, before it becomes a finished product to be analysed. Much like screen scholarship that studies in a ‘post event’ way, what was lacking – usefully for us – was input from those who are practitioners themselves. The wider Melbourne-based Eye Tracking and the Moving Image research group within which this study sits has a membership that includes other practitioners, including a sound designer and a filmmaker. Combined, this suggested that our approach might offer something different; that it might ‘do more’ and hopefully speak to the industry as well as other researchers. As a screenwriter, the opportunity to co-research with scholars, scientists and other creative practitioners was therefore not only appealing, but also methodologically important.

As already highlighted, it was both an academic and a practical interest in the intersection of plot, character and theme that underpinned my approach. As Smith has argued, valuing character in screen studies has not always been possible (1995); moving this forward, valuing character, and in particular the character’s journey, has recently become more salient (see Batty, 2011; Marks, 2009), adding weight to a creative practice approach to screen scholarship. In this way, understanding the viewer’s experience of the screen seemed to lend itself well to some of the core concerns of the screenwriter; or to put it another way, had the ability to test what we ‘know’ about creative practice, and the role of the practitioner. Feeding, then, into wider debates about the place of screenwriting in the academy (see Baker, 2013; Price, 2013; 2010), it was important to value the work of the screenwriter, and in a scholarly rigorous – and hopefully innovative – way.

Researcher B: The majority of research on eye tracking and the moving image to date has been designed and undertaken as an extension to cognitive theories of film comprehension. Deriving from the constructivist school of cognitive psychology, and led by film theorist David Bordwell, this approach argues that viewers do not simply absorb but construct the meaning of a film from the data that is presented on screen. This data does not constitute a complete narrative but a series of cues that viewers process by generating inferences and hypotheses (Elsaesser and Buckland, 2002: 170). Bordwell’s approach explicitly opposes psychoanalytic film theory by attending to perceptual and cognitive aspects of film viewing rather than unconscious processes. Psychologist Tim Smith has mobilized eye tracking in connection with Bordwell’s work to demonstrate how this empirical method can “prove” cognitive theories of comprehension—showing that subjects’ eyes do fixate on those cues in a film’s mise-en-scène that the director has controlled through strategies of staging and movement (Smith, 2011; 2013).

The Up study was designed to follow in the wake of Smith’s work, with a particular interest in examining the premise of Bordwell’s theory – which is that narration is the central process that influences the way spectators understand a narrative film (Elsaesser and Buckland, 2002: 170). With this in mind, we deliberately chose a segment from an animated film, where the tightly directed narrative of the montage sequence is competing with a variety of other stimuli that subjects’ eyes could plausibly be attracted to: salient colourful and visibly designed details in the background and landscape of each shot.

We were also interested in this montage sequence for the highly affecting nature of its mini storyline, which establishes the protagonist Carl’s deep love for his wife Ellie as the motivation for his journey in Up! itself. The sequence carries a great deal of emotive power by contrasting the couple’s happiness in their long marriage with Carl’s ultimate sadness and regret at not being able to fulfill their life-long dream of moving to South America before Ellie falls sick and dies. Would it be possible to ‘see’ this emotional impact in viewers’ gaze behaviour?

How we reacted to the initial data, and what it was telling us.

Researcher A: When looking at data for the first time, I certainly saw a correlation between what we know about screenwriting and seeing, and what we could now turn to as evidence. For example, key objects such as the adventure book, the savings jar (see Fig. 1) and the picture of Paradise Falls – all of which recurred throughout the montage sequence – were looked at by viewers intensely, suggesting that narrative meaning was ‘achieved’.

Fig. 1. A heat map showing the collective intensity of viewers’ responses to the savings jar.

Fig. 1. A heat map showing the collective intensity of viewers’ responses to the savings jar.

As another example, when characters were purposely (from a screenwriting perspective) separated within the frame of the action, viewers oscillated between the two, eventually settling on the one they believed to possess the most narrative meaning (see Fig. 2). This further implied the importance of the character journey and its associated sense of theme, which for screenwriting verifies the careful work that has gone into a screenplay to set up narrative expectations.

Fig. 2. A gaze plot showing the fixations and saccades of one viewer in a scene with the prominent faces of Carl and Ellie.

Fig. 2. A gaze plot showing the fixations and saccades of one viewer in a scene with the prominent faces of Carl and Ellie.

Researcher B: We chose to analyse the data on Up! by examining how viewer attention fluctuated in focus between Carl and Ellie across the course of the montage sequence. The two are equal agents in the narrative at the beginning, but the montage’s story unfolds through the action and behaviour of each as it continues – that is, each character carries the story at different points. Overwhelmingly, the data supported this narrative pattern by showing that the majority of viewers fixated on the character who, moment by moment, functions as the agent of the story, even when that figure is not the most salient aspect of the image. Aligning with Bordwell’s cognitive theory of comprehension, this data confirms that viewers do rely principally on narrative cues to understand a film. As a top-down process of cognition, narrative exerts control over viewer attention to keep focus on the story rather than let the gaze wander to other bottom-up (salient) details in the mise-en-scène. It is this process that allowed Smith to show that viewers overwhelmingly will not notice glaring continuity errors on screen (Smith, 2005). As in the famous ‘Gorillas in our Midst’ experiment (Simons and Chabris, 1999), viewer attention is focused so closely on employing narrative schema to spatially, temporally and causally linked events that the salient stimuli on screen appears to be completely missed.

Researcher C: Initially I was quite interested to see the attention paid to faces, and in particular, characters’ eyes and mouths. Being animation, I had been keen to see if similar elements of faces would draw viewers’ eyes in the same ways that we look at human faces, where eyes and mouths are most viewed (Crouzet, et al., 2010). Here, even though the characters were not engaging in dialogue, their mouths as well as their eyes were still searched. Looking at eyes has been linked to looking for contextual emotional information (Guastella et al., 2007), and so with this montage sequence being non-verbal, it was not surprising to see much of the focus on characters’ eyes as viewers attempted to read the emotion though them (see Fig. 3).

Fig, 3. Two viewers’ gaze plots depicting the sequence of fixations made between Carl and Ellie.

Fig, 3. Two viewers’ gaze plots depicting the sequence of fixations made between Carl and Ellie.

Other areas I was interested to observe were instances when other well-known features drew strong viewer attention, such as written text and bright (salient) objects. Two particular scenes we examined contained examples of these. In one scene, in which the savings jar sits at the back of a dark bookshelf, viewers were both drawn to look at the bright candle in the foreground and also to the savings jar. The jar was in the dark, however with narrative cues to draw attention to it as well as the fact that it contained text, viewers were drawn to look at it (see Fig. 1). Surprisingly, in this scene other interesting objects are easily discernible – a wooden colourful bird figure; a guitar; a compass – yet the savings jar as well as the bright candles were viewed. The contextual information, the text and the salience appear to be working here to drive the eye, all within a few seconds of time.

Fig. 4. Gaze plots of fixations made by all viewers over the scene in which Carl purchases airline tickets.

Fig. 4. Gaze plots of fixations made by all viewers over the scene in which Carl purchases airline tickets.

The second scene to see text working as a cue for the eye was in the travel shop scene (Fig. 4). Here, viewers were drawn to look at two text-based posters placed on the back wall of the shop. Again, this scene was only shown momentarily, yet glances towards the text and images, as well as the exchange between the characters, give viewers the elements of the story they need to glean so that they know what is going on, and where the story will go next (Carl’s surprise for Ellie).

How over time we better understood the data, and what we began to know more

Researcher A: I was interested to see that some viewers spent time looking at the periphery. The Up! montage sequence did not necessarily offer ‘alternative’ layers in the margins of the screen, though given its created and controlled animated nature, it perhaps should not be a surprise that away from the centre of the screen there were visual delights, such as the sun setting over the city and a blanket of clouds that changed shape, from clouds to animals to babies. This suggested to me that in animation, because viewers know that images have been created from scratch, there is an expectation that the screen will offer a plethora of experiences, from narrative agency to visual amplification. This, in turn, suggested that in further studies, it might be useful to contrast texts that use the potential of the full screen to engage viewers with those that go in close and privilege the centre. Genre would most likely play a key role in this future endeavour.

Researcher B: As hoped, this pilot study has been instructive as a base from which we can now expand. It has raised many questions. One issue is that this data cannot ‘prove’ subjects were not seeing those elements on-screen that were not fixated upon – were they perhaps seeing them peripherally? This could only be confirmed by conducting interviews after the eye tracking takes place, and could instructively inform an understanding of how story information that is layered in the mise-en-scène (for instance in setting, lighting and costume) contributes to overall narrative comprehension. We are also very interested to determine how the context of viewing affects gaze behaviour. For instance, would subjects still fixate overwhelmingly on narrative cues when watching this sequence in a cinema environment on a large – even an IMAX – screen? In this environment the image on screen is larger and the texture more palpable. Would viewers here perhaps be more focused on these salient pleasures of the image and engage in a different, less cognitive experience of the film; letting their eyes roam across the grain of the shot in its colours, shapes and surfaces? Would results alter between an animated and live action film? Psychoanalytic film theory tells us that the cinematic apparatus promotes identification with characters and, by extension, the ideologies of the social system from which they are produced (Mulvey, 1975). Eye tracking can potentially intervene in this powerful theory of spectatorship by showing if and how viewers do fixate on the cues that give rise to this interpellation.

Researcher C: After looking at some of early scene analyses, I was somewhat surprised by how many eye movements could be made in fleetingly fast scenes, and at how many items in these scenes one could fixate on, if only briefly. I had expected viewers to be taking in some of the surrounding items in a scene using their peripheral vision, and to see more of the centralisation bias (Tatler and Vincent, 2009). Yet for some scenes, in particular for the two scenes in which Carl purchases the surprise airline tickets (see Figs 4 and 5), we see how viewers were drawn to search for narrative clues by looking around the scene.

Fig. 5. Gaze plot showing the fixations made by all viewers as they briefly see the contents of the picnic basket.

Fig. 5. Gaze plot showing the fixations made by all viewers as they briefly see the contents of the picnic basket.

In the first scene (see Fig. 4), Carl in seen in a shop, facing the shop assistant. Viewers had previously seen him in the midst of coming up with a bright idea. This scene thus gives the viewer a chance to work out what his idea was. What can be seen is that most viewers scanned the surrounds for clues. A similar pattern is seen in the next scene, in which we quickly glance at the contents of a picnic basket being carried by Carl (see Fig. 5). In the basket, which is seen close up, viewers scan the basket’s contents. It contains picnic items and the surprise airline ticket, and even though some glances went to other basket items, it was the ticket that captured most of the attention; the item that held the most narrative information. This item was also the most salient, being the clearest and brightest item in the basket, and, importantly, the only item to contain written text. In a very short glimpse of a scene, these features almost ensured that viewers’ eyes were directed to look at and acknowledge the ticket.

What excites us about the future of work in this area, and where we think it might take our own disciplines

Researcher A: If we are to fully embrace the creative practice potential of studies such as this, then we might look to creating new texts that can then be studied. If, in 1971, Norton and Stark created simple drawings to test how their subjects recognised and learned patterns, then over 40 years later, our approach might be to develop a short moving image narrative through which we can test our viewers’ gaze. For example, if we were to develop a short film and play it out of sequence (i.e., narrative meaning altered), might we affect where viewers look? Might they look differently: in different places and for different lengths of time? Similarly, what if we were to musically score a text in different ways, diegetically and non-diegetically? Might we affect the focus of viewer gaze? If so, what might this tell us about narrative attention and filmmaking techniques that sit ‘beyond the screenplay’?

For screenwriting as a discipline, studies such as these would serve two purposes, I feel. Firstly, they would help to strengthen the presence of screenwriting in the academy, especially in regard to innovative research that privileges the role of the practitioner. Accordingly, these studies could provide a variety of methodological approaches that might be of use to other screenwriting scholars; or that might be applied to other creative practice disciplines, in which researchers wish to understand the work that has gone into the creation of a text that might otherwise only be studied once it has been completed. Secondly, and perhaps more importantly, such studies might yield results that benefit, or at least inform, future screenwriting practices. Whether industry-related practices or otherwise, just like all ‘good’ creative practice research, the insights and understandings gained would contribute to the discipline in question in the form of ‘better’ or ‘different’ ways of doing (Harper, 2007). For me, this would reflect both the nature and the value of creative practice research.

Researcher B: All of the potential avenues for future research in this field take an essential interest in how moving images on screen produce a play between top-down and bottom-up cognition. In this, a larger issue for me – going back to the points I raised at the beginning of my section – is how the data can be mobilized beyond a strictly cognitive framework and vocabulary of screen theory. As indicated, the cognitive approach offers a deliberately ‘common sense’ counterpart to a paradigm such as psychoanalysis, with its reliance on myth, desire and fantasy (Elsaesser and Buckland, 2002: 169). Cognitive theory understands a film as a data set that a viewer’s brain processes and completes in an active construction of meaning – an understanding that eye tracking and neurocinematics is very well placed to support and expand. But most screen scholars appreciate and theorize film and television texts as much more than mere sets of data. The moving image is an experience that only ‘works’ by generating emotional affect, by engaging the viewer’s attachments, memories, desires and fears. Film theorist Linda Williams proposes that our investment in following the twists and turns of a narrative is fundamentally reliant upon the emotion of pathos: we continually, pleasurably invest in the expectation that a character will act or be acted upon in such a way that they achieve their goal, and continually, pleasurably have that expectation obscured and dashed by the story (Williams, 1998). So viewer attention is driven not just by a drive to know but also by a desire to feel: to be swept up in waves of hope and disappointment.

The mini storyline of the Up! montage sequence relies entirely on this dialectic of action and pathos. Carl and Ellie’s hopes are repeatedly frustrated, and Carl is finally unable to redeem this pattern before Ellie dies – producing a profound sense of pathos and regret as the defining theme of the sequence. We can see that our subjects’ fixations fell in line with this pattern as the sequence unfolded, consistently focusing on the character who was triggering or carrying the emotional power. But how do we distinguish the ‘felt’ dimension of this gaze out from the viewer’s efforts to simply comprehend what is happening by following characters’ movements, facial expressions or body language? How, that is, can we ‘see’ emotional engagement, and start to appreciate how this crucial dimension of spectatorship – based on feeling not thinking – governs the play between top-down and bottom-up cognition in moving pictures? For me, grappling with this problem – and perhaps experimenting with further measurements of pupil dilation, heart rate and brain activity – offers a fascinating pathway into understanding how eye tracking can move beyond an engagement with cognitive film theory to contribute to phenomenological thinking on genuinely embodied seeing and experience.

Researcher C: There is so much that can be done in this area, and that makes it an exciting pursuit; yet what makes it even more motivating is the way that we hope to go about it: collaboratively. One of the core aspects that members of ETMI are very passionate about is working together, bringing in different fields, different disciplines, different ways of seeing things, and building bridges between them. This work is not only about learning more about how we watch and interact with films, but also about having different perspectives on those insights. Work I would personally like to see undertaken in this way is to explore how black and white viewing compares to colourised viewing, and to explore whether and how 3D viewing affects how we gaze about a scene. To compare the gaze and emotional responses of children and adults to the same visual content, and similarly compare visual and emotional responses to material between males and females, and between genre fans and haters, is also an interesting possibility.

Finally, adding to these, I am excited about the potential collection and analysis of other physiological measures to better gauge emotional engagement. These include blood pressure, pupillometry, skin conduction, breathing rate and volumes, heart rate, sounds made (gasps, holding breath, sighs etc.) and facial expressions made.

Conclusion

By reflecting on each of our research backgrounds, experiences and expectations, what this article has revealed is that while we might have all come to the study with varied approaches and intentions, we have come out of the study with a somewhat surprisingly harmonious set of observations and conclusions. Without knowing it, perhaps, we were all interested in narrative and the role that characters play in the agency of it. We were also similarly interested in landscape and the visual potential of the screen; not in an obvious way, but in relation to subtext, meaning and emotion. The value of a study like this, then, lies not just in its methodological originality, but also in its ability to stir up passions in cross-disciplinary researchers, whereby each can bring to the table their own skills and ways of understanding data to reach mutual and respective conclusions. Although we ‘knew’ this from undertaking the study, the opportunity to reflect fully on the process in the form of an article has given us an even greater understanding of the collaborative potential of cross-disciplinary researchers such as ourselves.

 

References

Aristotle. (1996). Poetics. Trans. Malcolm Heath. London: Penguin.

Baker, Dallas. (2013). Scriptwriting as Creative Writing Research: A Preface. In: Dallas Baker and Debra Beattie (eds.) TEXT: Journal of Writing and Writing Courses, Special Issue 19: Scriptwriting as Creative Writing Research, pp. 1-8.

Batty, Craig, Adrian G. Dyer, Claire Perkins and Jodi Sita. (Forthcoming). Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative. In: CarrieLynn D. Reinhard and Christopher J. Olson (eds.). Making Sense of Cinema: Empirical Studies into Film Spectators and Spectatorship. New York: Bloomsbury.

Batty, Craig. (2013) Creative Interventions in Screenwriting: Embracing Theme to Unify and Improve the Collaborative Development Process. In: Shane Strange and Kay Rozynski. (eds.) The Creative Manoeuvres: Making, Saying, Being Papers – the Refereed Proceedings of the 18th Conference of the Australasian Association of Writing Programs, pp. 1-12.

Batty, Craig. (2011). Movies That Move Us: Screenwriting and the Power of the Protagonist’s Journey. Basingstoke: Palgrave Macmillan.

Batty, Craig and Zara Waldeback. (2008). Writing for the Screen: Creative and Critical Approaches. Basingstoke: Palgrave Macmillan

Bordwell, David. (1985). Narration in the Fiction Film. London: Routledge.

Buswell Guy. T. (1935). How People Look at Pictures. Chicago: Chicago University Press.

Cerf, Moran, E. Paxon Frady and Christof Koch. (2009). Faces and text attract gaze independent of the task: Experimental data and computer model. Journal of Vision, 9(12): 10, pp. 1–15.

Crouzet, Sebastien M., Holle Kirchner and Simon J. Thorpe. (2010). Fast saccades toward faces: Face detection in just 100 ms. Journal of Vision, 10(4): 16, pp. 1–17.

Egri, Lajos. (2004). The Art of Dramatic Writing. New York: Simon & Schuster.

Elsaesser, Thomas and Warren Buckland. (2002). Studying Contemporary American Film: A Guide to Movie Analysis. London: Hodder Headline.

Guastella, Adam J., Philip B. Mitchell and Mark R Dadds. (2008). Oxytocin increases gaze to the eye region of human faces. Biological Psychiatry, 63, pp. 3-5.

Harper, Graeme and Jonathan Rayner. (2010). Cinema and Landscape. Bristol: Intellect.

Harper, Graeme. (2007). Creative Writing Research Today. Writing in Education, 43, p. 64-66.

Hayhoe, Mary. (2000). Vision using routines: A functional account of vision. Visual Cognition, 7, pp. 43–64.

Hockley, Luke. (2007). Frames of Mind: A Post-Jungian Look at Cinema, Television and Technology. Bristol: Intellect.

Land, Michael F., Neil Mennie and Jennifer Rusted. (1999). The roles of vision and eye movements in the control of activities of daily living. Perception, 28, pp. 1311–1328.

Marks, Dara. (2009). Inside Story: The Power of the Transformational Arc. London: A&C Black

Marks, Laura U. (2002). Touch: Sensuous Theory and Multisensory Media.

Minneapolis: University of Minnesota Press.

Mulvey, Laura. (1975). Visual Pleasure and Narrative Cinema. Screen, 16(3), pp. 6-18.

Norton, David, and Lawrence Stark. (1971). Scanpaths in eye movements during pattern perception. Science, 171, pp. 308–311.

Price, Steven. (2013). A History of the Screenplay. Basingstoke: Palgrave Macmillan.

Price, Steven. (2010). The Screenplay: Authorship, Theory and Criticism. Basingstoke: Palgrave Macmillan.

Rodowick, David. (2007). The Virtual Life of Film. Cambridge, MA: Harvard University Press.

Schnotz, Wolfgang and Richard K. Lowe. (2008). A unified view of learning from animated and static graphics. In: Richard K. Lowe and Wolfgang Schnotz (eds.). Learning with animation: Research implications for design. New York: Cambridge University Press, pp. 304-356.

Schütz, Alexander C., Doris I. Braun and Karl R. Gegenfurtner. (2011). Eye movements and perception: A selective review. Journal of Vision, 11(5), pp. 9, 1–30.

Simons, Daniel J. and Christopher F. Chabris. (1999). Gorillas in our Midst: Sustained Inattentional Blindness for Dynamic Events. Perception, 28, pp. 1059-1074.

Smith, Murray (1995). Engaging Characters: Fiction, Emotion, and the Cinema. Oxford: Oxford University Press.

Smith, Tim J. (2005). An Attentional Theory of Continuity Editing. [accessed October 17, 2014].

Smith, Tim J. (2011). Watching You Watch There Will Be Blood. [accessed August 22, 2014].

Smith, Tim J. (2013). Watching you watch movies: Using eye tracking to inform cognitive film theory. In: A. P. Shimamura (ed.). Psychocinematics: Exploring Cognition at the Movies. New York: Oxford University Press, pp. 165-191.

Sobchack, Vivian (1992). The Address of the Eye: A Phenomenology of Film Experience. Princeton, N.J: Princeton University Press.

Stadler, Jane (2010). Landscape and Location in Australian Cinema. Metro, 165.

Tatler, Benjamin W., and Benjamin T. Vincent. (2009). The prominence of behavioural biases in eye guidance. Visual Cognition, 17, pp. 1029–1054.

Tobii Technology (2005). User Manual. Tobii Technology AB. Danderyd, Sweden.

Vogler, Christopher (2007). The Writer’s Journey: Mythic Structure for Writers. Studio City, CA: Michael Wiese Productions.

Wells, Paul (2010). Boards, Beats, Binaries and Bricolage – Approaches to the Animation Script. In: Jill Nelmes (ed.) Analysing the Screenplay, Abingdon: Routledge, pp. 104-120.

Wells, Paul (2007) Basics Animation 01: Scriptwriting. Worthing: AVA Publishing.

Williams, Linda (1998). Melodrama Revised. In: Nick Browne (ed.). Refiguring American Film Genres: History and Theory. Berkeley, CA: University of California Press.

Yarbus, Alfred L. (1967). Eye Movements and Vision. New York: Plenum.

 

List of figures

Fig. 1. A heat map showing the collective intensity of viewers’ responses to the savings jar. Source: author study.

Fig. 2. A gaze plot showing the fixations and saccades of one viewer in a scene with the prominent faces of Carl and Ellie. Source: author study.

Fig, 3. Two viewers’ gaze plots depicting the sequence of fixations made between Carl and Ellie. Source: author study.

Fig. 4. Gaze plots of fixations made by all viewers over the scene in which Carl purchases airline tickets. Source: author study.

Fig. 5. Gaze plot showing the fixations made by all viewers as they briefly see the contents of the picnic basket. Source: author study.

 

Notes

[i] A full analysis of this study, ‘Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative’, will appear in the forthcoming collection Making Sense of Cinema: Empirical Studies into Film Spectators and Spectatorship, edited by CarrieLynn D. Reinhard and Christopher J. Olson.

[ii] See Batty, Craig, Dyer, Adrian G., Perkins, Claire and Sita, Jodi (forthcoming) for full results.

 

Bios

Associate Professor Craig Batty is Creative Practice Research Leader in the School of Media and Communication, RMIT University, where he also teaches screenwriting. He is author, co-author and editor of eight books, including Screenwriters and Screenwriting: Putting Practice into Context (2014), The Creative Screenwriter: Exercises to Expand Your Craft (2012) and Movies That Move Us: Screenwriting and the Power of the Protagonist’s Journey (2011). Craig is also a screenwriter and script editor, with experiences across short film, feature film, television and online drama.

Dr Claire Perkins is Lecturer in Film and Screen Studies in the School of Media, Film and Journalism at Monash University. She is the author of American Smart Cinema (2012) and co-editor of collections including B is for Bad Cinema: Aesthetics, Politics and Cultural Value (2014) and US Independent Film After 1989: Possible Films (forthcoming, 2015). Her writing has also appeared in journals including Camera Obscura, Critical Studies in Television, Celebrity Studies and The Velvet Light Trap.

Dr Jodi Sita is Senior Lecturer in the School of Allied Health at the Australian Catholic University. She works within the areas of neuroscience and anatomy, with expertise in eye tracking research. She has extensive experience with multiple project types using eye tracking technologies and other biophysical data. As well as her current research using into viewer gaze patterns while watching moving images, she is using eye tracking to examine expertise in Australian Rules Football League coaches and players, and to examine the signature forgery process.

Sound and Sight: An Exploratory Look at Saving Private Ryan through the Eye Tracking Lens – Jennifer Robinson, Jane Stadler and Andrea Rassell

$
0
0

Abstract

Using eye tracking as a method to analyse how four subjects respond to the opening Omaha Beach landing scene in Saving Private Ryan (Steven Spielberg, 1998), this article draws on insights from cinema studies about the types of aesthetic techniques that may direct the audience’s attention along with findings about cognitive resource allocation in the field of media psychology to examine how viewers’ eyes track across film footage. In particular, this study examines differences when viewing the same film sequences with and without sound. The authors suggest that eye tracking on its own is a technological tool that can be used to both reveal individual differences in experiencing cinema as well as to find psychophysiologically governed patterns of audience engagement.

Introduction

Steven Spielberg’s Saving Private Ryan (1998) begins at a geriatric pace, ambling alongside an elderly World War II veteran as he visits a military cemetery and begins to reminisce about the men who saved his life during the Battle of Normandy in June, 1944. This is where the story really starts, with a platoon of terrified, seasick servicemen led by Captain John Miller (Tom Hanks) landing on Omaha Beach where they come under heavy fire by German infantry. The Omaha Beach landing scene is gruelling in its experiential intensity as the hand-held camera locates the audience alongside soldiers desperately fighting their way toward the enemy line amidst relentless machine gunfire and bone-shuddering explosions that tear them limb from limb.

An interdisciplinary 2014 study by Vittorio Gallese (one of the scientists credited with the discovery of mirror neurons), fellow neuroscientists Katrin Heimann and Maria Alessandra Umiltà, and film scholar Michele Guerra investigated the effects of camera movement on the audience’s feeling of involvement in film scenes and their ability to place themselves in the position of a screen character. This study was conducted using a high-density electroencephalogram (EEG) to test whether the audience’s experience of what Gallese (2012, 2013) terms “embodied simulation”—that is, neural mirroring responses that are associated with empathy—is affected by camera movement as well as by the action of human figures on screen. The researchers found that the relationship between cognition and action perception is significantly influenced by camera movement and that the use of camera techniques such as steadicam elicit stronger mirroring responses and an augmented sense of involvement in the scene because this type of cinematography more closely resembles human movement than static camera, zooms, or dolly-mounted tracking shots (Heimann et al. 2014, 2098–99).

These findings are consistent with eye tracking studies by Paul Marchant and colleagues who have demonstrated that the audience’s visual attention is captured and guided by mobile framing, focus, the direction of screen characters’ movement and lines of sight, and the colour and motion of other aspects of the mise-en-scene (Marchant et al. 2009, 157–58). This interplay of figure movement and the technical and aesthetic dimensions of cinematography is relevant to Saving Private Ryan in that the arresting beach landing scene at the start of the film is shot almost exclusively using hand-held camera to simulate human movement. The study by Heimann and colleagues suggests that this form of camera movement, teamed with the panicked motion of the figures on screen, functions to elicit a sense of affective identification with Captain Miller and the soldiers he leads by stimulating a shared experience of embodied confusion and sensory overload as the military men shake with fear and scramble to dodge the shrapnel ricocheting across the war-ravaged beach. The unstable gaze of the constantly moving camera makes it as difficult for the audience as it is for the soldiers in the scene to focus attention or see a pathway to safety and this shared perceptual experience may elevate neural mirroring responses or empathic concordance with observed actions.

Venturing into an area that has received less attention from either film scholars, media effects researchers or neuroscientists, we were struck by the acoustic ferocity of the Omaha Beach scene and we sought to understand the ways in which sound functions as a perceptual cue that may affect the cinema audience’s attention and modulate gaze patterns. This interdisciplinary study brings empirical eye tracking research into dialogue with formalist understandings of film style and cognitive engagement with narrative, using the following question to establish a framework for analysis: What audio-visual aesthetic cues guide the audience’s attention and what psychophysiological processes underlie audience responses to the screen? In particular, we draw on existing research on film dialogue by Todd Berliner and others and we supplement eye tracking data by drawing on Lisa Coulthard’s concept of “dirty sound,” Vivian Sobchack’s work on the “sonic imagination” and cognitivist methods of aesthetic film analysis to work through the experiential dimensions of the sonic confusion generated in the scene.

Cognitivist film theory, as advanced by scholars such as David Bordwell and Carl Plantinga, conceptualises film and television spectatorship as the active construction of meaning via the inferential elaboration of perceptual cues and formal screen production conventions. In a quest for greater explanatory power and a more holistic understanding of spectatorship that moves beyond rational thought and conscious inferential processes, film theorists are increasingly drawing upon empirical research in fields such as neuroscience, psychology and media effects to test assumptions about how audiences perceive and respond to screen texts, and to account for the sensory experiences and involuntary physiological reactions of the audience.

Psychophysiological Approaches to Cinema Studies

There are several different empirical approaches to studying audience members’ responses to film, including biometrics, neuroimaging, and psychophysiological techniques. Psychophysics is an area of research that quantifies physiological or bodily responses to psychological states. Neurocinema (Hasson et al. 2008) and Psychocinematics (Shimamura 2013) are emerging fields that connect these psychophysiological methods to cinematic experience. Where the neurocinematic approach involves imaging of the brain while watching cinema, in psychophysiology the subject’s physiological state is understood to be representative of psychological responses (for example, skin conductance and heart rate indicate arousal or an emotional reaction). One such response is an involuntary orienting response that assigns cognitive resources to processing stimuli in screen texts automatically.

Annie Lang (Lang 2000; Lang et al. 2000) proposes a model of responding to dynamic screen media that starts from the position that there are limited cognitive resources that any individual can bring to bear when processing mediated content. Features of the screen content can automatically consume some of those cognitive resources, which leaves less capacity for the intentional interpretation of meaning, formulation of hypotheses or speculation about protagonists’ motives (the very processes that cognitive film theory privileges). While this has been well developed for visual attributes, such as hard edits, movement and new features, Lang and colleagues are developing a similar catalogue of attributes for aural content (sound). Using a physiological indicator of an orienting response (a short, rapid decrease in heart rate just after the feature is introduced), they have identified “voice changes, music onsets, sound effect onsets, production effect onsets, emotional word onsets, silence onsets, and voice onsets” as aural cues that orient attention (Lang et al. 2014, 4).

Embodied responses to film are not necessarily indicative of cognitive processing as some responses occur in the autonomic nervous system (such as the startle response to a loud sound or a sudden movement); other processes involve the conscious allocation of cognitive resources. For example, seeing a poisonous reptile on screen can make the audience form hypotheses about impending danger, which can then prime emotional reactions such as anxiety. Increased heart rate during the shower scene in Psycho (Alfred Hitchcock, 1960), or light perspiration on the palms as viewers watch Grace Kelly fossick through the neighbour’s apartment in Rear Window (Alfred Hitchcock, 1954), are widely understood to be biological evidence of changes in psychological states in response to cinema. In such a state of arousal hormones are released, blood pressure rises, and brain wave patterns shift. These biological changes can be recorded using non-invasive techniques and have proven to be stable markers of psychophysiological changes. Some commonly used psychophysiological measures are eye tracking, Galvanic Skin Response (GSR), and pupillometry; however, in this exploratory study, only eye tracking has been used.

Eye Tracking

Eye tracking is a technique that can measure the movements of the eye by gauging the direction of infrared light bounced off the eye surface. The most common technique utilises the eye’s physiology to create different reflections of the light source from the pupil and cornea that are captured by two cameras and used to used to track the gaze and control for head and eye movements. While there are several types of eye tracking devices, those most pertinent to this study include eye trackers that require the viewer to be in a fixed position such as seated in front of a monitor, and those that can be head-mounted or worn like glasses by a mobile viewer. Eye tracking devices are used in a wide variety of fields including marketing, sports coaching and user experience. The range of Tobii Technology eye trackers are frequently employed as research tools to measure attention, as is the case in this study. Two of the main characteristics of eye movement that can be measured by eye tracking devices are saccades and fixations.

Saccades

In order to collect high-quality visual data about our environment, the eye needs to be constantly redirected. We use movements called saccades in order to do this. Saccades occur at a rate of about 2-3 per second (Tatler 2014) and can be voluntary or reflexive (Duchowski 2007). Their duration ranges from 10-100 ms, rendering the individual effectively blind during this time, but not for long enough to be perceivable: “Visual sensitivity effectively shuts down during a saccade via a process known as saccadic suppression, in order to ensure that the rapid movement of light across the retina is not perceived as motion blur” (Smith 2014, 86).

Fixations

A fixation is a length of time when the eyes stop large movements (saccades) and stay focused on a small visual range (typically about 5 degrees). Fixations should not be thought of as static, as the name implies, but as “miniature eye movements: tremor, drift and microsaccades” (Duchowski 2007, 46). Their duration is usually in the range of 150-600 ms (Duchowski 2007) and most visual information is processed when the eyes stabilize or fixate on a point on the screen (Smith 2014, 86).

Previous findings

A consistent finding from eye tracking research that is relevant to this study of cinema is that when scenes are viewed on a screen or a monitor, the gaze tends to fixate at the centre more than the periphery, even when salient features are not located in the middle of the frame. Because this tendency may be adaptive (for example the centre is a good resting place for fast response to new action that requires attending to), rather than solely visual, Benjamin Tatler (2014) warns against a reductive expectation that these fixations are caused by visual stimuli alone. While this study attends closely to visual stimuli and the aesthetic techniques used by filmmakers to direct attention, we also consider aural stimuli and involuntary biological responses.

Despite the large body of eye tracking research, Antoine Coutrot and colleagues claim that until recently, only two preliminary studies had investigated the influence of sound on eye movements and patterns of attention when watching film or video footage (Coutrot et al. 2012, 2).[i] When studying eye movements in response to the presence and absence of sound in audiovisual stimuli, Coutrot et al. analyse differences in three further eye tracking metrics: dispersion, distance to centre, and Kullback-Lieber Divergence. Dispersion refers to the “variability of eye positions between observers” (2012, 4). Distance to centre is a measurement of “the distance between the barycenter of a set of eye positions and the centre of the screen” (Coutrot et al. 2012, 4). Kullback-Leiber divergence “is used to estimate the difference between two probability distributions. This metric can be compared as a weighted correlation measure between two probability density functions… The lower the KL-divergence is, the closer the two distributions are… If soundtrack impacts on eye position locations, we should find a significant difference between the mean inter and intra KL-divergences” (Coutrot et al. 2012, 4). Dispersion provides information about the variability between eye positions, but does not determine the relative position of the two data sets of the eye positions for the two stimulus conditions (sound on/sound off). For the KL-divergence, it is the opposite.

Coutrot and colleagues (2012) found that eye movements follow a consistent pattern that is involuntary and that is not affected by screen aesthetics, narrative content, genre, sound or other factors in the first second following an edit. After a brief latent phase, the eye automatically refocuses on the centre of the screen after a cut and takes a second to adjust to the new image. Thereafter, they found that sound does influence gaze patterns in the following ways: dispersion is lower in the sound on condition than the sound off condition; fixation locations are different between the two conditions; sound results in larger saccades than the same footage without sound; and sound elicits longer fixations than sound off (Coutrot et al. 2012, 8).

More recently, Coutrot and Guyader found that “removing the original soundtrack from videos featuring various visual content impacts eye positions increasing the dispersion between the eye positions of different observers and shortening saccade amplitudes” (2014, 2). This study also found that in dialogue scenes, the audience’s attention tends to “follow speech turn taking more closely” (Coutrot and Guyader 2014, 1). A 2014 study by Tim Smith also investigated the cross-modal influences of audio on visual attention and found that “When the visual referent is present on the screen, such as the face of a speaker (that is, a diegetic on-screen external sound source), gaze will be biased towards the sound source, and towards the lips if the audio is difficult to interpret” (Smith 2014, 92). This accords with research in film studies into dialogue and conversation in movies. For instance, Berliner notes that movie dialogue is typically scripted to advance the narrative by directing the audience’s attention to key plot points and protagonists;[ii] furthermore, “characters in Hollywood movies communicate effectively and efficiently through dialogue” and “movie characters tend to speak flawlessly” (Berliner 2010, 191). Similarly, Aline Remael identifies the promotion of narrative continuity and textual cohesion as two of the chief functions of film dialogue (2003, 227; 233). Given these findings from two different fields of research, we pay particular attention to gaze patterns during dialogue exchanges in the analysis of Saving Private Ryan that follows.

Method

Building on previous work by Tim Smith, Antoine Coutrot, Nathalie Guyador and other researchers who have used eye tracking to investigate attentional synchrony[iii] (as illustrated in gaze plots and heat maps that represent the concentration of the audience’s gaze), our methodology examines the distribution of fixations across nine smaller central Areas of Interest (AOIs) during film sequences to explore what is occurring for viewers who are not following the predicted pattern and instead are searching for something else. Using two conditions as stimuli (film with sound on, and film with sound off), we conducted a qualitative comparison between and within the viewing patterns of four subjects. Within the limitations of a qualitative and exploratory study with only four subjects, we drilled down to conduct a fine-grained mapping of attention to determine whether it functions in a predictable way in relation to previous findings about dialogue scenes, sonic cues and attention in relation to camera and figure movement.

For the purposes of this study, a Tobii X-120 eye tracker and Tobii Studio 2.3.2 software (Tobii Technology, Stockholm, Sweden) were used to record seven individual subjects (five females, two males) as they watched film footage. As this was an exploratory study, subjects were recruited from the researchers’ networks, with ethics approval. They were seated and positioned 55-65 cm away from the eye tracker for viewing. All subjects were recorded on the same Tobii X-120, in the same room, with the film footage played on the Tobii computer to standardise start times for all subjects to enable comparison in later analysis. Each subject was successfully calibrated by looking at symbols in different areas of the screen, which ensures the eye tracker gets a reliable measure of gaze location across the whole screen. After the viewing session, each subject’s data was analysed for quality, with three subjects excluded because one condition had segments with lower reliability than desired. Thus, the results are reported on the basis of four subjects with high quality and complete data (three females, one male).

We analysed the areas of the screen where subjects looked while watching discrete sequences of the key beach-landing scene at the beginning of Saving Private Ryan. We investigated how different stylistic techniques employed in the following four consecutive sequences of the scene affect the audience’s gaze patterns:

  1. The “Indistinct Dialogue” sequence is an 11-second clip that was chosen with a view to finding out how the audience’s attention is affected when dialogue is overridden by chaotic background noise, forcing the audience to strain to decipher what is being said. This part of the scene occurs immediately after Captain Miller has located the men under his command. The first shot is an unsteady medium close-up of Miller shouting to Horvath (Tom Sizemore) as bullets splash in the surf around him and ping off the metal structure he is crouching behind. Miller yells, “Sergeant Horvath! (Explosion.) Move your men off the beach! (Water splashes up noisily.) Now!” The next shot shows Horvath’s response in a hand-held medium close-up as he points at his men and hollers, “OK you guys, get on my ass! (Directional hand signals as bullet hits metal and drowns out dialogue.) Follow me! (Horvath ducks as a mortar shell explodes, screen right.)
  2. The “Wounded Man” sequence that occurs as the men move up the beach is a 30-second segment that is noteworthy because it includes a subjective sequence that solicits audience engagement with Captain Miller’s experience of temporary hearing loss following the concussive impact of a mortar shell nearby. This clip begins with a hand-held long shot of carnage on the beach as Miller moves to the right, dragging Briggs, a wounded soldier he is trying to help. The audience hears artillery fire, crashing, splashing, and shouting as Miller lugs Briggs into the mid-ground, with explosions and debris visible in the foreground. As mortar shells hit, spraying blood and water upwards, Miller hollers for a medic. Following a massive explosion, the sound of gunfire in the background is muted and is replaced with the subdued drone of a low, echoing, wind-like sound that communicates Miller’s subjective experience of shellshock as sand obscures everything and Miller falls to the ground. In slow motion, we cut to a low level close-up of boots in the sand. Miller scrambles up and the hand held camera follows him. Other soldiers pass in front of the camera, occluding the lens and masking the edit. The sounds of the battlefield are replaced by an echoing, low frequency droning noise and the subdued clink of military gear as Miller is momentarily dazed by shellshock. As he gets up and grabs Briggs’s arm, the sound of artillery returns loudly and we see Miller in long-shot framed with a low level camera. He staggers, looks back, and realises that Briggs is dead: his lower abdomen and legs have been blasted away. This sequence ends with a close-up of Miller’s reaction as he looks at Briggs in shock, abandons him, crawls away from the camera, then stands and runs toward the sand dunes, into enemy fire and gun smoke.
  3. The “Sand Dunes” sequence is one unusually long and complex shot that lasts for a full minute; however, in the interests of generating a more granular analysis we divided the shot in two. The first 25 seconds “Sand Dunes: In Command” begins with a match on action as the body of a soldier that was catapulted into the air by a grenade in the previous shot now hits the ground. Quickly, the hand-held camera tilts down from the long-shot of the falling soldier, pans left, and follows Miller forward as he dives behind a ridge of sand for shelter. The camera pushes in to frame Miller in close up as the dialogue begins:

Miller: (Turns left to address the radio operator.) Shore Party! No armour has made it ashore. We got no DD tanks on the beach. Dog One is not open. (Miller rolls to the right so he is framed in an over-the-shoulder shot as he shouts to other soldiers seen in medium-long shot on the dune.) Who is in command here?

Soldier: You are, Sir.

Miller: Sergeant Horvath!

Horvath: Sir!

Miller: Do you know where we are?

Horvath: Right where we’re supposed to be, but nobody else is …

4. “Sand Dunes: Radio” is a 35-second continuation of the shot detailed in sequence three, beginning when the hand-held camera pans left as Miller rolls back toward the radio operator, facing the camera in close up as he grabs the radio operator’s shoulder and hollers in his ear, straining to be heard over the background gunfire.

Soldier: (Distant, off screen, as Miller rolls to the left.) Nobody’s where they’re supposed to be!

Miller: (To radio operator) Shore Party! First wave ineffective. We do not hold the beach. Say again, we do not hold the beach. (Miller turns and rolls back towards the right, away from the camera. The camera zooms toward Horvath, excluding Miller from the frame as he listens to Horvath.)

Horvath: (Indistinct) We’re all mixed up, sir. We got the leftovers from Fox Company, Able Company and George Company. Plus we got some Navy Demo guys and a Beachmaster.

Miller: (The camera follows Horvath as he rolls to the left, toward Miller; we then see Miller in medium close-up as he turns back to radio operator.) Shore party! Shore party! (Realises radio operator is deceased; grabs hold of the radio himself.) Cat-F, Cat-F, C-… (Miller realises the radio is dead.)

In Overhearing Film Dialogue, Sarah Kozloff states that “although what the characters say, exactly how they say it, and how the dialogue is integrated with the rest of the cinematic techniques are crucial to our experience and understanding of every film since the coming of sound, for the most part analysts incorporate the information given by a film’s dialogue and overlook the dialogue as signifier” (2000, 6). By contrast, our analysis focuses closely on what the characters say, and also on what they hear. It may seem counterintuitive to be investigating the significance of sound in an eye tracking study, because sound is something that the eyes are not normally required to process. However, the Omaha Beach dialogue sequences are unusual because the audience has to rely on their eyes to search for contextual cues in order to fill in gaps in understanding due to indistinct vocals. Such cues include the direction of figure movement and eye lines (when Horvath yells, “Get on my ass and follow me!” but his words are obscured by background sound), or facial expressions and body language when words don’t fully make sense due to the inclusion of military terminology or unfamiliar radio communication codes and incomplete communicative exchanges (when Miller shouts “Dog-One” and “Cat-F” into the radio and receives no response). Coutrot and colleagues identify numerous instances in which aural and visual stimuli interact to affect attention (they offer as one example, “the help given by ‘lip reading’ to understanding speech, even more when speech is produced in poor acoustical conditions,” as is the case in Saving Private Ryan); furthermore, they report that “perceivers gazed more at the mouth as auditory masking noise levels increased” (Coutrot et al. 2012, 2).[iv] Our study builds on this work, as detailed below.

With respect to each of the four sequences outlined above, two different analyses were conducted. Given that most of the viewing was within the central area of the screen, we subdivided that area into a three by three grid providing nine smaller areas of interest, as illustrated in Figure 1.[v] The total time fixated, and mean fixation duration, was calculated for each of the nine Areas of Interest (AOI). For the default method on the Tobii eye tracking system that we used, a fixation is identified when the gaze is steadily focused in the same area of X-Y coordinates on the screen, typically occurring within 35 pixels. This technique of dividing the centre of the screen into nine smaller AOIs allowed greater granularity in determining the primary AOI where most people attended and the instances where individuals diverged and looked at other parts of the screen.

Figure 1: Nine Areas of Interest (AOI)

Figure 1: Nine Areas of Interest (AOI), Saving Private Ryan: Still from Saving Private Ryan (Steven Spielberg, 1998), data illustrated by the authors

The second step was to analyse whether the participants exhibited attentional synchrony and primarily look at the same AOIs as each other or displayed individual variation. For this, we calculated an estimate of attentional distribution. For this exploratory study, a simple ratio of time spent following the guided (dominant) viewing pattern compared to time looking at other parts of the screen was calculated.[vi] The dominant AOIs were determined by which AOIs account for over 50% of fixation time in the sound on condition (control). As some sequences guide the viewer between several of the AOIs, the combination that yielded a majority of viewing time was used rather than simply the AOI with the greatest portion of time. The Distribution ratio was then calculated as follows:

Distribution Ratio =

Number non-dominant AOIs viewed multiplied by amount of time in those AOIs

Number of AOIs in dominant view multiplied by amount of time in the dominant AOIs

Eye Tracking Results and Discussion

The distribution ratio was intended to see whether the findings of Coutrot et al. (2012) applied to what we saw in the sequences from the Omaha Beach landing scene. We hypothesised that a lack of sound would increase divergence away from the dominant AOI, compared to the null condition with sound that should reinforce “attentional synchrony” (Smith 2013) by guiding the viewer to the most important focal area. However, we expected that because some of the sequences we were examining contained many competing audio and visual contextual cues, we might not see such a clear distinction. Furthermore, we anticipated that our findings in relation to dialogue may diverge from Coutrot and Guyader (2014) because the Omaha Beach scene has atypical dialogue sequences that deviate from conventional turn-taking and are overloaded with noise and movement to create a sense of confusion.

We found that averaged across all sequences, three of the viewing subjects followed the expected pattern of greater divergence with sound off, as indicated by a positive difference score in Table 1. The fourth (Subject 3) had slightly higher gaze distribution when there was sound but did have an increase in the mean number of AOIs with fixations in the “sound off” condition. This would suggest that, on the whole, sound does function to focus attention more tightly. Given the small number of participants in this exploratory study, this result is encouraging.

    Table 1: Distribution Ratios Arranged by Subject     *Note: A positive difference indicates that in line with Coutrot et al. (2012), there was greater distribution of attention across AOIs for the no sound condition. The higher distribution ratio when sound was off could be due to either more total time fixated away from the dominant AOI or having fixations in more of the nine AOIs, being more spread out, or a combination of both.

Table 1: Distribution Ratios Arranged by Subject
*Note: A positive difference indicates that in line with Coutrot et al. (2012), there was greater distribution of attention across AOIs for the no sound condition. The higher distribution ratio when sound was off could be due to either more total time fixated away from the dominant AOI or having fixations in more of the nine AOIs, being more spread out, or a combination of both.

However, not all sequences of the beach scene elicited the same results. When the distribution data is averaged by sequence, it turns out that sequences 1 (d = -3.4) and 3 (d = -1.8) were strongly in the predicted direction with less distribution across AOIs when there was sound. For sequence 2 there was little difference (d < 1.0), but it was still in the predicted direction. However, for sequence 4 (d = 1.0), there was greater distribution of the fixations away from the dominant AOI in the sound on condition, indicating greater focus when there was no sound. Sequence 4 (“Sand Dunes: Radio”) does not follow screen conventions for shooting dialogue: it is shot in one long take rather than the customary shot-reverse-shot style and because many of Captain Miller’s lines contain military jargon and receive no response, the audience’s habituated expectations about turn-taking and shifting attention from speaker to speaker are derailed. Breaking with aesthetic and technical conventions may disrupt cognitive process of meaning-making when watching film. Another, more physiologically based reason that the “Sand Dunes: Radio” sequence may not conform to the gaze distribution patterns found in other parts of the scene is that it is the continuation of a very long take (together the sand dune sequences constitute a single, minute-long shot that viewers watched unbroken); consequently, viewers’ eyes are not re-focused on the centre of the screen following a cut and their eyes have more time to rove and explore the visual field for other meaningful cues. Put another way, without any cuts to generate an orienting response during this sequence there is no automatic allocation of cognitive resources to the story or refocusing of attention back onto a particular portion of the screen (Lang 2000, 2014). It is then a very individual response to novel and emotive (signal) cues within the scene that drives where each subject looks over the duration of this sequence. With so many different types of auditory cues that orient the viewer (Lang 2014), it is not surprising that viewers had fixations in the various AOIs we analysed on the screen.

In Saving Private Ryan, as has been found to be the case in other films such as Sergei Eisenstein’s 1938 historical war epic, Alexander Nevsky (see Smith 2014), the overall viewing patterns reflect the intention of the director in that audience members typically look where they are guided to by audio-visual screen conventions. Yet, an important reminder for further investigation is that not all members of an audience respond in the same way to each scene. This leads us to question what cues other than the lack of a sound track might lead to increased gaze distribution.

Even though the size of the nine AOIs and the number of participants was small, paired-sample t-tests were conducted to see whether there were any statistically significant differences at the level of each of the nine AOIs between the sound and no sound viewing experiences of the participants. No significant differences between the sound on and off conditions were found for fixation duration, total time fixated, visit duration or total time visiting any particular AOI for three of the sequences: “Indistinct Dialogue,” “Wounded Man,” or “Sand Dunes: Radio.” However, for “Sand Dunes: In Command,” subjects spent significantly (p < .05) less time looking at areas 5 and 6 when the sound was off (significant greater mean fixation duration, total time fixated and total time visiting AOI 5 when the sound was on; only total time fixated on AOI 6 was greater when sound was present). Interestingly, this did not translate to an increase in any particular AOI so it seems their gazes spread out significantly (dispersed) in the sound off condition.

With a small sample size, it is not surprising that there were few statistical differences, but it was surprising that focusing on the central area as represented by the nine AOIs did not pick up what seemed “obvious” when looking at the aggregated gaze plots. For example, in the “Indistinct Dialogue” sequence (see Figure 2) there is an explosion on the right-hand side of the screen that equally drew the attention of the subjects when sound was off as well as on (the screen characters ducked as the mortar shell whistled in, so aural and visual cues reinforced each other). Although there was one subject who looked down to the lower right part of the screen outside our central area of view when sound was off, the overall pattern was consistent in both conditions.

Figure 2a (above) and Figure 2b (below): Gaze plots of four subjects for “Indistinct Dialogue” (Sequence 1)

Figure 2a (above) and Figure 2b (below): Gaze plots of four subjects for “Indistinct Dialogue” (Sequence 1): Still from Saving Private Ryan (Steven Spielberg, 1998), data illustrated by the authors

We explored whether eye tracking could help reveal differences in viewing experience amongst subjects or even offer insight into what was happening beneath the surface of apparent synchrony. An obvious finding is that each individual subject had a different gaze pattern across the four sequences sampled, as can be seen when examining the pattern as recorded for Subject 2 in Figure 4 and Figure 6. For the longer sequences, their gaze fixated in more than half of the AOIs, while for the shorter sequences they were often more focused on particular AOIs. However, this pattern could change with sound on and off. For example when examining the pattern for Subject 1 for “Indistinct Dialogue”, there was a noticeable difference between sound on and off such that they only viewed three AOIs with sound, but with sound off they spread out to three new AOIs, ranging across six in total. The greatest shift was away from time fixated in the top left corner of our central area (when the footage was played with sound), contracting to the central third of the screen (without sound).

Sean Redmond and colleagues reported that the presence of sound only has an effect on fixation duration (number of fixations) for the “Wounded Man” sequence (forthcoming 2015). In the “Wounded Man” sequence, there was no overall difference in gaze location with sound on or sound off. However, the gaze fixation pattern for Subject 2 showed a large qualitative difference (see Figure 3). With sound off, Subject 2 only looked at AOIs 6, 8 and 9 (bottom right part of the central area, which is consistent with Miller’s screen direction and the action of falling to the ground and dragging the wounded soldier, Briggs, in this sequence). However, with sound on, Subject 2 fixates at least briefly in all nine AOIs, with the most time shifting to the centre of the screen where noisy background action is taking place and where other soldiers rapidly pass in front of the camera.

Figure 3: Areas of Interest for subject 2 in “Wounded Man” (Sequence 2)

Figure 3: Areas of Interest for subject 2 in “Wounded Man” (Sequence 2): Still from Saving Private Ryan (Steven Spielberg, 1998), data illustrated by the authors

However, when comparing all of the subjects and how they responded to the “Wounded Man” sequence (see Figure 4), the other three subjects exhibit similar patterns of scanning across the AOIs when sound is on and off. This pattern is what we expected for this sequence, which incorporates a significant subjective sound component when Miller experiences shellshock and is temporarily stunned and deafened. It is possible that subjective sound may help to anchor the viewer’s attention to the character’s experience.[vii]

Figure 4: Total fixation duration in nine AOIs for “Wounded Man” (all subjects)

Figure 4: Total fixation duration in nine AOIs for “Wounded Man” (all subjects): Graphs produced by the authors

A final illustration is the “Sand Dunes: In Control” sequence. Subject 3 was interesting because their viewing patterns for the “Wounded Man” (sequence 2) and the “Sand Dunes: Radio” (sequence 4) were consistent with the other subjects; however, the focus for “Sand Dunes: In Command” (sequence 3) was inconsistent. There were clear AOIs for the sound off condition, but with sound their eyes wandered over more of the central area of the screen. Attention is focused on the radio operator in the sound off condition (as indicated by the red bar in AOI 4, middle left of Figure 5). However, with sound on (indicated by the blue bars), the subject’s attention extends to new AOIs, including corners (top left, bottom left, and bottom right) that were not fixated on when there were no sound cues.

Figure 5: Areas of Interest for subject 4 in “Sand Dunes: In Command” (Sequence 3)

Figure 5: Areas of Interest for subject 4 in “Sand Dunes: In Command” (Sequence 3): Still from Saving Private Ryan (Steven Spielberg, 1998), data illustrated by the authors

A comparison between how the subjects fixated during this segment when the sound was on and off (Figure 6), illustrates a similar pattern of having eyes fixate in different AOIs when sound was on and off, except for Subject 1. Given the length of this sequence, the focus on three main AOIs for all subjects in the sound off condition is interesting in its consistency. However, even though the fixation data averages out to no difference between sound on and off, each of our subjects had a different response when sound was on—from fixating in all of the AOIs by Subject 4 to just staying longer on the same AOIs for Subject 1. The variation between sound on and off for this sequence may simply be an artefact of camera and figure motion, where shifts in the location of the protagonists’ faces on the screen can result in fixations in non-dominant AOIs (Mital et al. 2011, 19). However, the fact that these shifts did not occur in both conditions indicates that there is something different about those shifts when sound is on and the viewer can hear the dialogue. This is the only sequence where there was a significant difference in total time spent fixated in AOIs 5 and 6. The much lower time spent on key AOIs when sound was off suggests the subjects were looking to the periphery of the screen and did not look as much to the nine central AOIs.

Figure 6: Total fixation duration for nine AOIs in “Sand Dunes: In Command” (all subjects)

Figure 6: Total fixation duration for nine AOIs in “Sand Dunes: In Command” (all subjects): Graphs produced by the authors

Relating Eye Tracking Findings to Film Aesthetics

Our qualitative, exploratory analysis of gaze patterns in Saving Private Ryan has used eye tracking to offer an empirical account of cognitive-perceptual processing that includes sound and attends closely to audio-visual cues in the film’s stylistic system. In this way we have sought both to redress the limitations of theoretical approaches to film analysis that privilege inferential cognitive processes and to counterbalance the tendency of empirical studies to neglect the role of screen aesthetics in informing audience responses. In particular, we have built on existing work on eye tracking by taking account of how the aesthetic and experiential deployment of sound might affect perceptual processing, given that sound waves have “palpable force,” which means that sound seems “more materialized, more concrete, and more present to our experience than what we can see” (Sobchack 2005, 7). We have worked from the premise that expectations regarding character and narrative are chief among the ways that screen texts engage audiences in the construction of meaning, yet we have also acknowledged that the process of meaning-making is informed by aesthetic cues and by physiological sense-making, which is in part involuntary. The dangerous and chaotic Omaha Beach scene is what Man-Fung Yip refers to as an “intensified sensory environment” in which, “as a concrete visual and visceral force rather than a mere vehicle for semiotic signification, film violence offers an intensity of raw, immediate sensation that powerfully engages the eye and body of the spectator” (2014, 78). Like Yip, our interest has been sustained by “the complex interplay between the capacity of the human body and the resources of the cinematic medium” (2014, 89).

In her acoustic study of extreme cinema, Lisa Coulthard refers to the use of “deliberate sonic imperfections” (2013, 115) in films in which “visual assaultiveness” is paired with a disturbing soundscape: “Capable of impacting the body in palpable ways, sound is mined in many of these films for its viscerality: as one listens to extremities of acoustic proximity, frequency and volume, one’s own body responds in subconscious ways to those depicted and heard on screen” (2013, 117). These insights about sound are pertinent to Saving Private Ryan in that the Omaha Beach scene is designed to bombard the audience with the relentless onslaught of noise and action that the characters themselves face. In analysing this scene, we began with the hypothesis that “when the intensity of a background sound exceeds a certain threshold, mental activity can be paralyzed” (Augoyard and Torgue 2005, 41 qtd in Coulthard 2013, 118). In other words, we questioned whether the frenzied barrage of sound might cause a form of sensory-cognitive overload that could affect typical patterns of perceptual processing.

The eye tracking results did not reveal a significant pattern for scenes where we predicted this would occur. It would be helpful to explore this in the future with other physiological or neuro-measures that are better at identifying moments of cognitive overload or resource allocation. What does seem to emerge from our exploratory study is that even in films that firmly direct attention, as is characteristic of Spielberg’s directorial style, individual audience members bring their own complexity and experience to the viewing.[viii] Lang and colleagues point out that “complexity should be indexed not by how much of something is in the message but rather by how many human processing resources will be consumed when the message is processed” (Lang et al. 2014, 2). With respect to understanding the specific effects of what individual viewers bring with them to the screen, or teasing out how the audience is affected when watching footage that uses hand-held camera, induces cognitive overload, invokes the acoustic imagination, or uses indistinct dialogue, we conclude that this eye tracking study has raised fruitful questions that may best be answered by an approach that includes multiple measures provided by electroencephalograms, pupillometry or galvanic skin response techniques, as well as eye tracking technologies.

The “Indistinct Dialogue” and “Sand Dunes” sequences have what Sobchack terms a larger number of “‘synch points’ (‘salient’ and ‘prominent’ moments of audio-visual synchrony),” such as lines of dialogue, bullets pinging off metal and mortar shells landing, and these sonic cues “are firmly attached in a logically causal—and conventionally ‘realistic’—relation to the image’s specificity” (2005, 6–7). These synchronised sounds are “not as acousmatically imaginative and generative” as we contend that the subjective sound in the “Wounded Man” sequence is because the sounds appear to be “generated from the physical action” seen on the screen (Sobchack 2005, 7). In the “Indistinct Dialogue” and “Sand Dunes” conversation sequences, our eye tracking experiment did not necessarily reveal greater attentional focus with dialogue. While this counters what Coutrot and colleagues found in their 2012 study and Smith’s finding that sound reinforces visual synchrony (Smith 2013), it is in line with our expectation that the unconventional use of indistinct dialogue and chaotic background sound and imagery would disperse attention. Perhaps dialogue is not something that focuses visual attention, but rather something that focuses engagement. When the dialogue is clear, the viewer is able to look around the screen and absorb other cues about context. Precisely because the linguistic meaning is clear, such expository dialogue does not require as many cognitive resources to process and leaves some free for assessing other audio-visual cues. However, when the dialogue is indistinct, the viewer must then use other cues to work out the importance of the speech; in such cases the audience is essentially in the same position as watching without sound—although they may even be worse off in terms of cognitive resource allocation because there is also a barrage of other sound being processed in concert with the visual stimuli.

Overall, our use of eye tracking in conjunction with aesthetic analysis in our investigation of Saving Private Ryan has supported Coutrot and colleagues’ 2012 findings that dispersion (the degree of variability between observers’ eye positions) was lower with sound than without, so sound generally acted to concentrate perceptual attention. However, unlike Coutrot et al., we teamed eye tracking with qualitative film analysis to explore the effect of aesthetic variation and individual differences on gaze patterns as well as to identify common psychophysiologically governed patterns of attention. In this exploratory study, we found that differences in aesthetic techniques within segments of footage in the same film scene do make a difference to the audience’s gaze patterns and attentional fixation, and we found that within these patterns individual subjects exhibited divergent perceptual processes as well. Although our study is more restricted than comparable work undertaken by Coutrot and others, our attention to screen aesthetics and to variations in subjects’ responses within a single scene affords our method broader explanatory power than a study that excludes outliers and looks for commonalities across a wide range of video styles and genres.

 

References

Alexander Nevsky. Directed by Sergei Eisenstein, 1938. Mosfilm, DVD.

Augoyard, Jean-Francois, and Henry Torgue. 2005. Sonic Experience: A Guide to Everyday Sounds. Translated by Andra McCartney and David Paquette. Montreal: McGill Queen’s University Press.

Berliner, Todd. 2010. Hollywood Incoherent: Narration in Seventies Cinema. Austin: University of Texas Press.

Bordwell, David. 2009. “Cognitive Theory.” In Routledge Companion to Philosophy and Film, edited by Paisley Livingston and Carl Plantinga. 356–367. London: Routledge.

Coulthard, Lisa. 2013. “Dirty Sound: Haptic Noise in New Extremism.” In The Oxford Handbook of Sound and Image in Digital Media, edited by Carol Vernallis, Amy Herzog and John Richardson. 115–126. New York: Oxford University Press.

Coutrot, Antoine, Gelu Ionescu, Nathalie Guyader and Bertrand Rivet. “Audio Tracks do not Influence Eye Movements when Watching Videos.” Paper presented to the 34th European Conference on Visual Perception, Toulouse, France August 30, 2011.

Coutrot, Antoine, Nathalie Guyader, Gelu Ionescu and Alice Caplier. 2012. “Influence of Soundtrack on Eye Movements During Video Exploration.” Journal of Eye Movement Research 5.5: 1–10.

Coutrot, Antoine and Nathalie Guyader. 2014. “How Saliency, Faces, and Sound Influence Gaze in Dynamic Social Scenes.” Journal of Vision 14.8: 5.

Duchowski, Andrew T. 2007. Eye Tracking Methodology Theory and Practice. Dordrecht, Springer.

Gallese, Vittorio. 2013. “Mirror Neurons, Embodied Simulation and a Second-person Approach to Mind-reading.” Cortex in press: 1–3. Accessed August 28, 2014, http://dx.doi.org/10.1016/j.cortex.2013.09.008

Gallese, Vittorio and Michel Guerra. 2012. “Embodying Movies: Embodied Simulation and Film Studies.” Cinema: Journal of Philosophy and the Moving Image 3: 183–210.

Hasson, Uri, Ohad Landesman, Barbara Knappmeyer, Ignacio Vallines, Nava Rubin and David J. Heeger. 2008. “Neurocinematics: The Neuroscience of Film” Projections 2.1: 1-26.

Heimann, Katrin, Maria Alessandra Umiltà, Michele Guerra and Vittorio Gallese. 2014. “Moving Mirrors: A High-density EEG Study Investigating the Effect of Camera Movements on Motor Cortex Activation during Action Observation.” Journal of Cognitive Neuroscience 26.9: 2087–2101.

Kozloff, Sarah. 2000. Overhearing Film Dialogue. Berkeley: University of California Press.

Land, Michael, Neil Mennie and J. Rusted. 1999. “The Roles of Vision and Eye Movements in the Control of Activities of Daily Living.” Perception 28.11: 1311–1328.

Lang, Annie. 2000. “The Limited Capacity Model of Mediated Message Processing.” Journal of Communication 50.1: 46–70.

Lang, Annie, Shuhua Zhou, Nancy Schwartz, Paul D. Bolls and Robert F. Potter. 2000. “The Effects of Edits on Arousal, Attention, and Memory for Television Messages: When an Edit is an Edit Can an Edit be too Much?” Journal of Broadcasting & Electronic Media 44.1: 94–109.

Lang, Annie, Ya Gao, Robert F. Potter, Seungjo Lee, Byungho Park and Rachel L. Bailey 2014. “Conceptualizing Audio Message Complexity as Available Processing Resources.” Communication Research, published online before print. Accessed September 28, 2014, doi: 10.1177/0093650213490722

Marchant, Paul, David Raybould, Tony Renshaw and Richard Stevens. 2009. “Are you seeing what I’m seeing? An Eye-tracking Evaluation of Dynamic Scenes.” Digital Creativity 20.3: 153–163.

McGurk, Harry and John MacDonald. 1976. “Hearing Lips and Seeing Voices.” Nature 264.5588: 746–8. doi:10.1038/264746a0.

Mital, Parag, Tim J. Smith, Robin Hill and Jim Henderson. 2011. “Clustering of Gaze During Dynamic Scene Viewing is Predicted by Motion.” Cognitive Computing 3: 5–24

Plantinga, Carl. 2009. Moving Viewers: American Film and the Spectator’s Experience. Berkeley: University of California Press.

Psycho. Directed by Alfred Hitchcock, 1960. Shamley Productions, DVD.

Rear Window. Directed by Alfred Hitchcock, 1954. Paramount, DVD.

Redmond, Sean, Sarah Pink, Jane Stadler, Jenny Robinson, Andrea Rassell and Darrin Verhagen. 2015 (forthcoming). “Seeing, Sensing Sound: Eye Tracking Soundscapes in Saving Private Ryan and Monsters Inc.” In Making Sense of Cinema: Empirical Studies into Film Spectators and Spectatorship, edited by CarrieLynn D. Reinhard and Christopher J. Olson. New York: Bloomsbury.

Remael, Aline. 2003. “Mainstream Narrative Film Dialogue and Subtitling.” The Translator 9.2: 225–247.

Saving Private Ryan. Directed by Steven Spielberg. 1998. Dreamworks/Paramount. DVD.

Shimamura, Arthur, ed. 2013. Psychocinematics: Exploring Cognition at the Movies. New York: Oxford University Press.

Sita, Jodi. 2014. Personal Communication. 19 June 2014. Australian Catholic University: Victoria, Australia.

Smith, Tim J. 2014. “Audiovisual Correspondences in Sergei Eisenstein’s Alexander Nevsky: A Case Study in Viewer Attention.” In Cognitive Media Theory (AFI Film Reader), edited by Paul Taberham and Ted Nannicelli. 85–105. New York: Routledge.

Smith, Tim J. 2013. “Watching You Watch Movies: Using Eye Tracking to Inform Cognitive Film Theory.” In Psychocinematics: Exploring Cognition at the Movies, edited by Arthur P. Shimamura. 165–191. New York: Oxford University Press.

Sobchack, Vivian. 2005. “When the Ear Dreams: Dolby Digital and the Imagination of Sound.” Film Quarterly 58.4: 2–15.

Song, Guanghan, Denis Pellerin and Lionel Granjon. 2011. “Sound Effect on Visual Gaze When Looking at Videos.” In 19th European Signal Processing Conference. 2034–2038. Barcelona: EUSIPCO 2011.

Tatler, Benjamin. 2014. “Eye Movements from Laboratory to Life.” Current Trends in Eye Tracking Research, edited by Mike Horsley, Matt Eliot, Bruce Allen Knight and Ronan Reily. 17–35. London: Springer.

Võ, Melissa, Tim J. Smith, Parag Mital and John Henderson. 2012. “Do the Eyes Really Have it? Dynamic Allocation of Attention when Viewing Moving Faces.” Journal of Vision. 12.13(3): 1–14 http://www.journalofvision.org/content/12/13/3.full

Yip, Man-Fung. 2014. “In the Realm of the Senses: Sensory Realism, Speed, and Hong Kong Martial Arts Cinema.” Cinema Journal 53.4: 76–97.

 

List of figures

Figure 1: Nine Areas of Interest (AOI), Saving Private Ryan: Still from Saving Private Ryan (Steven Spielberg, 1998), data illustrated by the authors

Figure 2a: Gaze plots of four subjects for “Indistinct Dialogue” (Sequence 1): Still from Saving Private Ryan (Steven Spielberg, 1998), data illustrated by the authors

Figure 2b: Gaze plots of four subjects for “Indistinct Dialogue” (Sequence 1): Still from Saving Private Ryan (Steven Spielberg, 1998), data illustrated by the authors

Figure 3: Areas of Interest for subject 2 in “Wounded Man” (Sequence 2): Still from Saving Private Ryan (Steven Spielberg, 1998), data illustrated by the authors

Figure 4: Total fixation duration in nine AOIs for “Wounded Man” (all subjects): Graphs produced by the authors

Figure 5: Areas of Interest for subject 4 in “Sand Dunes: In Command” (Sequence 3): Still from Saving Private Ryan (Steven Spielberg, 1998), data illustrated by the authors

Figure 6: Total fixation duration for nine AOIs in “Sand Dunes: In Command” (all subjects): Graphs produced by the authors

 

Notes

[i] The two preliminary sound-based eye tracking studies preceding Coutrot et al’s 2012 publication are a conference presentation by Coutrot et al. (2011), and a conference paper by Song, Pellerin, and Granjon (2011). However, in 2012 Melissa Võ and colleagues also published a study that investigated the effects on attention to faces in videos when the auditory speech track was removed. This study found that when speech was not present, observers’ gaze allocation changed: they looked more at the scene background and decreased fixations to faces generally and especially decreased concentration on the mouth region (Võ et al. 2012, 12).

[ii] A study of everyday attention indicates that people exhibit visual search behaviours that anticipate, locate, and monitor action, which is evidence of top down influences on visual perception (see Land et al. 1999).

[iii] Tim Smith states that “The degree of attentional synchrony observed for a particular movie frame will vary depending on whether it is from a Hollywood feature film or from unedited real-world footage, the time since a cut and compositional details such as focus or lighting but attentional synchrony will always be greater in moving images than static images” (2014, 90).

[iv] The lip-reading phenomenon is called the “McGurk effect” (see McGurk 1976).

[v] For further discussion of central areas of interest in Saving Private Ryan, see Redmond et al. (2015).

[vi] Established formulae for dispersion and other measures of individual variation in gaze pattern exist (e.g., Coutrot 2012). As an exploratory study, we were limited by both number of subjects and post hoc data analysis. This distribution estimate was a sufficient way to capture dominant and non-dominant viewing. However, we would recommend future research develop a better variance measure of asynchronous viewing, such as the Kullback-Lieber Divergence formula referred to above.

[vii] Note that similar results were obtained in a related study of a sequence earlier in the beach-landing scene that depicts Captain Miller’s experience of shellshock (Redmond et al. forthcoming 2015).

[viii] A neuroimaging study comparing responses to film clips ranging from a sequence directed by Alfred Hitchcock to a segment of actuality footage shot in Washington Square Park found that higher levels of aesthetic control generate greater viewer synchrony or inter-subject correlation in the audience’s viewing patterns and brain activity (Hasson et al. 2008, 15).

 

Bios

Dr Jennifer Robinson is Lecturer in Public Relations, School of Media and Communication at RMIT University. She authors industry reports and has published in J Advertising, BMC Public Health, J Interactive Marketing and the J Public Relations Research. Her media effects research investigates new media and media audiences using neuro-measures.

Jane Stadler is Associate Professor of Film and Media Studies, School of Communication and Arts at the University of Queensland. She is author of Pulling Focus: Intersubjective Experience, Narrative Film and Ethics, and co-author of Screen Media and Media and Society.

Andrea Rassell is a PhD student and Research Assistant in the School of Media and Communication at RMIT University. She has a professional background in both science and film and researches at the nexus of the two disciplines.

Archived volumes

Subtitles on the Moving Image: an Overview of Eye Tracking Studies – Jan Louis Kruger, Agnieszka Szarkowska and Izabela Krejtz

$
0
0

Abstract

This article provides an overview of eye tracking studies on subtitling (also known as captioning), and makes recommendations for future cognitive research in the field of audiovisual translation (AVT). We find that most studies in the field that have been conducted to date fail to address the actual processing of verbal information contained in subtitles, and rather focus on the impact of subtitles on viewing behaviour. We also show how eye tracking can be utilised to measure not only the reading of subtitles, but also the impact of stylistic elements such as language usage and technical issues such as the presence of subtitles during shot changes on the cognitive processing of the audiovisual text as a whole. We support our overview with empirical evidence from various eye tracking studies conducted on a number of languages, language combinations, viewing contexts as well as different types of viewers/readers, such as hearing, hard of hearing and Deaf people.

Introduction

The reading of printed text has received substantial attention from scholars since the 1970s (for an overview of the first two decades see Rayner et al. 1998). Many of these studies, conducted from a psycholinguistic angle, made use of eye tracking. As a result, a large body of knowledge exists on the eye movements during reading of people with varying levels of reading skills and language proficiency, with a range of ages, different first languages and cultural backgrounds, and in different contexts. Studies on subtitle reading, however, have not achieved the same level of scientific rigour largely for practical reasons: subtitles are not static for more than a few seconds at a time; they compete for visual attention with a moving image; and they compete for overall cognitive resources with verbal and non-verbal sounds. This article will identify some of the gaps in current research in the field, and also illustrate how some of these gaps can be bridged.

Studying the reading of subtitles is significantly different from studying the reading of static text. In the first place, as far as eye tracking software is concerned, the subtitles appear on a moving image as image rather than text, which renders traditional text-based reading statistics and software all but useless. This also makes the collection of data for reading research on subtitles a painstakingly slow process involving substantial manual inspections and coding. Secondly, the fact that subtitles appear against the background of the moving image means that they are always in competition with this image, which renders the reading process fundamentally different from the reading process of static texts: on the one hand because the reading of subtitles compete with the processing of the image, sometimes resulting in interrupted reading, but on the other hand the limited time the subtitles are on screen means that readers have less time to reread or regress to study difficult words or to check information. Either way, studying this reading process, and the cognitive processing that takes place during the reading, is much more complicated than in the case of static texts where we know that the reader is mainly focussing on the words before her/him without additional auditory and visual information to process.

While the viewing of subtitles has been the object of many eye tracking studies in recent years, with increasing frequency (see, for example Bisson et al. 2012; d’Ydewalle and Gielen 1992; d’Ydewalle and De Bruycker 2007; Ghia 2012; Krejtz et al. 2013; Kruger 2013; Kruger et al. 2013; Kruger and Steyn 2014; Perego et al. 2010; Rajendran et al. 2013; Specker 2008; Szarkowska et al. 2011; Winke et al. 2013), the study of the reading of subtitles remains a largely uncharted territory with many research avenues still to be explored. Those studies that do venture to measure more than just attention to the subtitle area, seldom do this for extended texts.

In this article we provide an overview of studies on how subtitles change the way viewers process audiovisual material, and also of studies on the unique characteristics of the subtitle reading process. Taking an analysis of the differences between reading printed (static) text and subtitles as point of departure, we examine a number of aspects typical of the way subtitle text is processed in reading. We also look at the impact of the dynamic nature of the text and the competition with other sources of information on the reading process (including scene perception, changes in the viewing process, shifts between subtitles and image, visual saliency of text, faces, and movement, and cognitive load), as well as discussing studies on the impact of graphic elements on subtitle reading (e.g. number of lines, and text chunking), and studies that attempt to measure the subtitle reading process in more detail.

We start off with a discussion of the way in which watching an audiovisual text with subtitles alters viewing behaviour as well as of the complexities of studying subtitles due to the dynamic nature of the image it has as a backdrop. Here we focus on the fleeting nature of the subtitle text, the competition between reading the subtitles and scanning the image, and the interaction between different sources of information. We further discuss internal factors that impact on subtitle processing, like the language and culture of the audience, the language of the subtitles, the degree of access the audience has to sound, and other internal factors, before turning to external factors related to the nature of the audiovisual text and the presentation of the subtitles. Finally, we provide an overview of studies attempting to measure the processing of subtitles as well as findings from two studies that approach the processing of subtitles

The dynamic nature of the subtitle reading process

Reading subtitles differs substantially from reading printed text in a number of respects. As opposed to “static text on a stable background”, the viewer of subtitled audiovisual material is confronted with “fleeting text on a dynamic background” (Kruger and Steyn 2014, 105). In consequence, viewers not only need to process and integrate information from different communication channels (verbal visual, non-verbal visual, verbal auditory, non-verbal auditory, see Gottlieb 1998), but they also have no control over the presentation speed (see Kruger and Steyn 2014; Szarkowska et al. forthcoming). As a consequence, unlike in the reading of static texts, the pace of reading is in part dictated by the text rather than the reader – by the time the text is available to be read – and there is much less time for the reader to regress to an earlier part of a sentence or phrase, and no opportunity to return to previous sentences. Reading traditionally takes place in a limited window which the reader is acutely aware will disappear in a few seconds. Even though there are exceptions to the level of control a viewer has, for example in the case of DVD and PVR as well as other electronic media where the viewer can rewind and forward at will, the typical viewing of subtitles for most audiovisual products happens continuously and without pauses just as when watching live television.

Regressions, which form an important consideration in the reading of static text, take on a different aspect in the context of the knowledge (the viewer has) that dwelling too much on any part of a subtitle may make it difficult to finish reading the subtitle before it disappears. Any subtitle is on screen for between one and six seconds, and the viewer also has to simultaneously process all the other auditory (in the case of hearing audiences) and visual cues. In other words, unlike when reading printed text, reading becomes only one of the cognitive processes the viewer has to juggle in order to understand the audiovisual text as a whole. Some regressions are in fact triggered by the change of the image in shot changes (and to a much lesser extent scene changes) when the text stays on across these boundaries, which means that the viewer sometimes returns to the beginning of the subtitle to check whether it is a new subtitle, and sometimes even re-reads the subtitle. For example, in a recent study, Krejtz et al. (2013) established that participants tend not to re-read subtitles after a shot change or cut. But their data also revealed that a proportion of the participants did return their gaze to the beginning of the subtitle after such a change (see also De Linde and Kay, 1999). What this means for the study of subtitle reading is that these momentary returns (even if only for checking) result in a class of regressions that is not in fact a regression to re-read a word or section, but rather a false initiation of reading for what some viewers initially perceive to be a new sentence.

On the positive side, the fact that subtitles are embedded on a moving image and are accompanied by a soundtrack (in the case of hearing audiences) facilitates the processing of language in context. Unfortunately, this context also introduces competition for attention and cognitive resources. For the Deaf and hard of hearing audience, attention has to be divided between reading the subtitles and processing the scene, extracting information from facial expressions, lip movements and gestures, and matching or checking this against the information obtained in the subtitles. For the hearing audience who makes use of subtitles for support or to provide access to foreign language dialogue, attention is likewise divided between subtitles and the visual scene, and just as the Deaf and hard of hearing audiences have the added demand on their cognitive resources of having to match what they read with what they get from non-verbal signs and lip movements, the hearing audience matches what they read with what they hear, checking for correspondence of information and interpreting intonation, tenor and other non-verbal elements of speech.

What stands beyond doubt is that the appearance of subtitles changes the viewing process. In 2000, Jensema et al. famously stated that “the addition of captions to a video resulted in major changes in eye movement patterns, with the viewing process becoming primarily a reading process” (2000a, 275). Having examined the eye movements of six subjects watching video clips with and without subtitles, they found that the onset of a subtitle triggers a change in the eye movement pattern: when a subtitle appears, viewers move their gaze from whatever they were watching in order to follow the subtitle. In a more wide-scale study it was concluded by d’Ydewalle and de Bruycker (2007,196) that “paying attention to the subtitle at its presentation onset is more or less obligatory and is unaffected by major contextual factors such as the availability of the soundtrack, knowledge of the foreign language in the soundtrack, and important episodic characteristics of actions in the movie: Switching attention from the visual image to “reading” the subtitles happens effortlessly and almost automatically”.

Subtitles therefore appear to be the cause of eye movement bias similar to faces (see Hershler & Hochstein, 2005; Langton, Law, Burton, & Schweinberger, 2008; Yarbus, 1967), the centre of the screen, contrast and movement. In other words, subtitles attract the gaze at least in part because of the fact that the eye is drawn to the words on screen just as the eye is drawn to movement and other elements. Eyes are drawn to subtitles not only because the text is identified as a source of meaningful information (in other words a top-down impulse as the viewer consciously consults the subtitles to obtain relevant information), but also because of the change to the scene that the appearance of a subtitle causes (in other words a bottom-up impulse, automatically drawing the eyes to what has changed on the screen).

As in most other contexts, the degree to which viewers will process the subtitles (i.e. read them rather than merely look at them when they appear and then look away) will be determined by the extent to which they need the subtitles to follow the dialogue or to obtain information on relevant sounds. In studying visual attention to subtitles it therefore remains a priority to measure the degree of processing, something that has not been done in more than a handful of studies, and something to which we will return later in the article.

Viewers usually attend to the image on the screen, but when subtitles appear, it only takes a few frames for most viewers to move their gaze to read the subtitles. The fact that people tend to move their gaze to subtitles the moment they appear on the screen is illustrated in Figures 1 and 2.

Figure. 1 Heat maps of three consecutive film stills – Polish news programme Fakty (TVN) with intralingual subtitles.

Figure. 1 Heat maps of three consecutive film stills – Polish news programme Fakty (TVN) with intralingual subtitles.

Figure 2. Heat maps of two consecutive film stills – Polish news programme Wiadomości (TVP1) with intralingual subtitles

Figure 2. Heat maps of two consecutive film stills – Polish news programme Wiadomości (TVP1) with intralingual subtitles

Likewise, when the gaze of a group of viewers watching an audiovisual text without subtitles is compared to that of a similar group watching the same text with subtitles, the split in attention is immediately visible as the second group reads the subtitles and attends less to the image, as can be seen in Figure 3.

Figure 3. Heat maps of the same scene seen without subtitles and with subtitles – recording of an academic lecture.

Figure 3. Heat maps of the same scene seen without subtitles and with subtitles – recording of an academic lecture.

Viewer-internal factors that impact on subtitle processing

The degree to which the subtitles are processed is far from straightforward. In a study performed at a South African university in the context of Sesotho students looking at a recorded lecture with subtitles in their first language and audio in English (their language of instruction), students were found to avoid looking at the subtitles (see Kruger, Hefer and Matthew, 2013b). Sesotho students in a different group who saw the same lecture with English subtitles processed the subtitles to a much larger extent. This contrast is illustrated in the focus maps in Figures 4.

4-Kruger

Figure 4. Focus maps of Sesotho students looking at a lecture in intralingual English subtitles (left) and another group looking at the same lecture with interlingual Sesotho subtitles (right) – recording of an academic lecture.

The difference in eye movement behaviour between the conditions is also evident when considering the number of subtitles skipped. Participants in the above study who saw the video with Sesotho subtitles skipped an average of around 50% of the Sesotho subtitles (median at around 58%), whereas participants who saw the video with English subtitles only skipped an average of around 20% of the English subtitles (with a median of around 8%) (see Kruger, Hefer & Matthew, 2014).

This example does not, however, represent the conventional use of subtitles where viewers would rely on the subtitles to gain access to a text from which they would have been excluded without the subtitles. It does serve to illustrate that subtitle reading is not unproblematic and that more research is needed on the nature of processing in different contexts by different audiences. For example, in a study in Poland, interlingual subtitles (English to Polish) were skipped slightly less often by hearing viewers compared to intralingual subtitles (Polish to Polish), possibly because hearing viewers didn’t need them to follow the plot (see Szarkowska et al., forthcoming).

Another important finding from eye tracking studies on the subtitle process relates to how viewers typically go about reading a subtitle. Jensema et al. (2000) found that in subtitled videos, “there appears to be a general tendency to start by looking at the middle of the screen and then moving the gaze to the beginning of a caption within a fraction of a second. Viewers read the caption and then glance at the video action after they finish reading” (2000, 284). This pattern is indeed often found, as illustrated in the sequence of frames from a short video from our study in Figure 5.

Figure 5. Sequence of typical subtitle reading – a recording of Polish news programme Fakty (TVN) with intralingual subtitles.

Figure 5. Sequence of typical subtitle reading – a recording of Polish news programme Fakty (TVN) with intralingual subtitles.

Some viewers, however, do not read so smoothly and tend to shift their gaze between the image and the subtitles, as demonstrated in Figure 6. The gaze shifts between the image and the subtitle, also referred to in literature as ‘deflections’ (de Linde and Kay 1999) or ‘back-and-forth shifts’ (d’Ydewalle and De Bruycker (2007), can be regarded as an indication of the smoothness of the subtitle reading process: the fewer the gaze shifts, the more fluent the reading and vice versa.

Figure 6. Scanpath of frequent gaze shifting between text and image – a recording of Polish news programme Fakty (TVN) with intralingual subtitles.

Figure 6. Scanpath of frequent gaze shifting between text and image – a recording of Polish news programme Fakty (TVN) with intralingual subtitles.

An important factor that influences subtitle reading patterns is the nature of the audience. In Figure 7 an interesting difference is shown between the way a Deaf and a hard of hearing viewer watched a subtitled video. The Deaf viewer moved her gaze from the centre of the screen to read the subtitle and then, after having read the subtitle, returned the gaze to the centre of the screen. In contrast, the hard of hearing viewer made constant comparisons between the subtitles and the image, possibly relying on residual hearing and trying to support the subtitle reading process with lip-reading. Such a result was reported by Szarkowska et al. (2011), who found differences in the number of gaze shifts between the subtitles and the image in the verbatim subtitles condition, particularly discernible (and statistically significant) in the hard of hearing group (when compared to the hearing and Deaf groups).

Figure 7. Scanpaths of Deaf and hard of hearing viewers. Left: Gaze plot illustrating the viewing pattern of a Deaf participant watching a clip with verbatim subtitles.  Right: Gaze plot illustrating the viewing pattern of a hard of hearing participant watching a clip with verbatim subtitles.

Figure 7. Scanpaths of Deaf and hard of hearing viewers. Left: Gaze plot illustrating the viewing pattern of a Deaf participant watching a clip with verbatim subtitles. Right: Gaze plot illustrating the viewing pattern of a hard of hearing participant watching a clip with verbatim subtitles.

These provisional qualitative indications of differences between eye movements of users with different profiles require more in-depth quantitative investigation and the subsequent section will provide a few steps in this direction.

As mentioned above, subtitle reading patterns largely depend on the type of viewers. Fluent readers have been found to have no difficulty following subtitles. Diao et al. (2007), for example, found a direct correlation between the impact of subtitles on learning and the academic and literacy levels of participants. Similarly, given that “hearing status and literacy tend to covary” (Burnham et al. 2008, 392), some previous studies found important differences in the way hearing and hearing-impaired people watch subtitled programmes. Robson (2004, 21) notes that “regardless of their intelligence, if English is their second language (after sign language), they [i.e. Deaf people] cannot be expected to have the same comprehension levels as hearing people who grew up exposed to English”. This is indeed confirmed by Szarkowska et al. (forthcoming) who report that Deaf and hard of hearing viewers in their study made more fixations on subtitles and that their dwell time on the subtitles was longer compared to hearing viewers. This result may indicate a larger effort needed to process subtitled content and more difficulty in extracting information (see Holmqvist et al. 2011, 387-388). This, in turn, may stem from the fact that for some Deaf people the language in the subtitles is not their mother tongue (their L1 being sign language). At the same time, for hearing-impaired viewers, subtitles provide an important source of information on the words spoken in the audiovisual text as well as other information contained in the audio track, which in itself explains the fact that they would spend more time looking at the subtitles.

Viewer-external factors that impact on subtitle processing

The ‘smoothness’ of the subtitle reading process depends on a number of factors, including the nature of the audiovisual material as well as technical and graphical aspects of subtitles themselves. At a general level, genre has an impact on both the role of subtitles in the total viewing experience, and on the way viewers process the subtitles. For example, d’Ydewalle and Van Rensbergen (1989) found that children in Grade 2 paid less attention to subtitles if a film involved a lot of action (see d’Ydewalle & Bruycker 2007 for a discussion). The reasons for this could simply be that action film tends to have less dialogue in the first place, but secondly and more significantly, the pace of the visual editing and the use of special effects creates a stronger visual element which shifts the balance of content towards the action (visual content) and away from dialogue (soundtrack and therefore subtitles). This, however, is an area that has to be investigated empirically. At a more specific level, technical characteristics of an audiovisual text such as film editing have an impact on the processing of subtitles.

1 Film editing

Film editing has a strong influence on the way people read subtitles, even beyond the difference in editing pace as a result of genre (for example, action and experimental films could typically be said to have a higher editing pace than dramas and documentaries). In terms of audience perception, viewers have been found to be unaware of standard film editing techniques (such as continuity editing) and are thus able to perceive film as a continuous whole in spite of numerous cuts – the phenomenon termed “edit blindness” (Smith & Henderson, 2008, 2). With more erratic and fast-paced editing, it stands to reason that the cognitive demands will increase as viewers have to work harder to sustain the illusion of a continuous whole.

When subtitles clash with editing such as cuts (i.e. if subtitles stay on screen over a shot or scene change), conventional wisdom as passed on by generations of subtitling guides (see Díaz Cintas & Remael 2007, ITC Guidance on Standards for Subtitling 1999) suggests that the viewer will assume that the subtitle has changed with the image and as a consequence they will re-read it (see above). However, Krejtz et al. (2013) reported that subtitles displayed over shot changes are more likely to cause perceptual confusion by making viewers shift their gaze between the subtitle and the rest of the image more frequently than subtitles which do not cross film cuts (cf. de Linde and Kay 1999). As such, the cognitive load is bound to increase.

2 Text chunking and line segmentation

Another conventional wisdom, perpetuated in subtitling guidelines and standards, is that poor line segmentation will result in less efficient processing (see Díaz Cintas & Remael 2007, Karamitroglou 1998). In other words, subtitles should be chunked per line and between subtitles in terms of self-contained semantic units. The line of dialogue: “He told me that he would meet me at the red mailbox” should therefore be segmented in something like the following ways:

He told me he would meet me
at the red mailbox.

Or

He told me
he would meet me at the red mailbox.

Neither of the following segmentations would be optimal because the prepositional phrase ‘at the red mailbox’ and the verb phrase ‘he would meet me’, respectively, are split, which is considered an error:

He told me he would meet me at the
red mailbox

He told me he
would meet me at the red mailbox.

However, Perego et al. (2010) found that poor line segmentation in two-line subtitles did not affect subtitle comprehension negatively. They also investigated 28 subtitles viewed by 16 participants using a threshold line between the subtitle region and the upper part of the screen, or main film zone, but did not find a statistically significant difference between the well-segmented and ill-segmented subtitles in terms of fixation counts, total fixation time, or number of shifts between subtitle region and upper area. The only statistically significant difference they found was between the mean fixation duration within the subtitle area between the two conditions, with the mean fixation duration in the ill-segmented subtitles being on average 12ms longer than in the well-segmented subtitles. Although the authors downplay the importance of this difference on the grounds that the difference is so small, it does seem to indicate at least a slightly higher cognitive load when the subtitles are ill-segmented. The small number of subtitles and participants, however, make it difficult to generalize from their results, again a result of the fact that it is difficult to extract reading statistics for subtitles unless the reading behaviour can be quantified over longer audiovisual texts.

In a study conducted a few years later, Rajendran et al. (2013) found that “chunking improves the viewing experience by reducing the amount of time spent on reading subtitles” (2013, 5). This study compared conditions different from those investigated in the previous study, excluding the ill-segmented condition of Perego et al. (2010), and focused mostly on live subtitling with respeaking. In the earlier study, which focused on pre-recorded subtitling, the subtitles in the two conditions were essentially still part of one sense unit that appeared as one two-line subtitle. In the later study, the conditions were chunked by phrase (similar to the well-segmented condition of the earlier study but with phrases appearing one by one on one line), no segmentation (where the subtitle area was filled with as much text as possible with no attempt at segmentation), word by word (where words appeared one by one) and chunked by sentence (where the sentences showed up one by one). Regardless of the fact that this later study therefore essentially investigated different conditions, they did find that the most disruptive condition was where the subtitle appeared word by word – eliciting more gaze points (defined less strictly than in fixation algorithms used by commercial eye trackers) and more “saccadic crossovers” or switches between image and subtitle area. However, in this study by Rajendran et al. (2013), the videos were extremely short (under a minute), and the sound was muted, hampering the ecological validity of the material, and once again making the findings less suitable to generalization.

Although both these studies have limitations in terms of generalizability, they both provide some indication that segmentation has an impact on subtitle processing. Future studies will nonetheless have to investigate this aspect over longer videos to determine whether the graphical appearance, and particularly the segmentation of subtitles, has a detrimental effect on subtitle processing in terms of cognitive load and effectiveness.

3 Language

The language of subtitles has received considerable attention from psycholinguists in the context of subtitle reading. D’Ydewalle and de Bruycker (2007) examined eye movement behaviour of people reading standard interlingual subtitles (with the audio track in a foreign language and subtitles in their native language) and reversed subtitles (with the audio in their mother tongue and subtitles in a foreign language). They found more regular reading patterns in the standard interlingual subtitling condition, with the reversed subtitling condition having more subtitles skipped, fewer fixations per subtitle, etc. (see also d’Ydewalle and de Bruycker 2003 and Pavakanun 1993). This is an interesting finding in itself, as it is the reversed subtitling that has been found to be particularly conducive to foreign language learning (see Díaz Cintas and Fernández Cruz 2008, and Vanderplank 1988).

Szarkowska et al. (forthcoming) examined differences in reading patterns of intralingual (Polish to Polish) and interlingual (English to Polish) subtitles among a group of Deaf, hard of hearing and hearing viewers. They found no differences in reading for the Deaf and hard of hearing audiences, but hearing people made significantly more fixations to subtitles when watching English clips with interlingual Polish subtitles than Polish clips with intralingual Polish subtitles. This confirms that the hearing viewers processed the subtitles to a significantly lower degree when they were redundant, as in the case of intralingual transcriptions of the soundtrack. What would be interesting to investigate in this context is those instances when the hearing audience did in fact read the subtitles, to determine to what extent and under what circumstances the redundant written information is used by viewers to support their auditory intake of information.

In a study on the influence of translation strategies on subtitle reading, Ghia (2012) investigated the differences in the processing of literal vs. non-literal translations into Italian of an English film clip (6 minutes) when watched by Italian EFL learners. According to Ghia, just as subtitle format, layout, and segmentation have the potential to affect visual and perceptual dynamics, the relationship translation establishes with the original text means that “subtitle translation is also likely to influence the perception of the audiovisual product and viewers’ general reading patterns” (2012,175). Ghia particularly wanted to investigate the processing of different translation strategies in the presence of sound and image with the subtitles. In her study she found that the non-literal translations (where the target text diverged from the source text) resulted in more deflections between text and image. This is similar to the findings of Rajendran et al. (2013) in terms of less fluent graphics in word-by-word subtitles.

As can be seen from the above, the aspect of language processing in the context of subtitled audiovisual texts has received some attention, but has not to date been approached in any comprehensive manner. In particular, there is a need for more psycholinguistic studies to determine how subtitle reading differs from the reading of static text, and how this knowledge can be applied to the practice of subtitling.

Measuring subtitle processing

1 Attention distribution and presentation speed

In the study by Jensema et al. (2000), subjects spent on average 84% of the time looking at subtitles, 14% at the video picture and 2% outside of the frame. The study represents an important early attempt to identify reading patterns in subtitle reading, but it has considerable limitations. The study had only six participants, three deaf and three hearing, and the video clips were extremely short (around 11 seconds each), presented with English subtitles (in upper case) without sound. The fact that there was no soundtrack therefore impacted on the time spent on the subtitles. In Perego et al’s study (2010), the ratio is reported as 67% on the subtitle area and 33% on the image. In this study there were 41 Italian participants who watched a 15-minute clip with Hungarian soundtrack and subtitles in Italian. As in the previous study, the audience therefore had to rely heavily on the subtitles in order to follow the dialogue. Kruger et al. (2014), in the context of intralingual subtitles in a Psychology lecture in English, found a ratio of 43% on subtitles, 43% on the speaker and slides and 14% on the rest of the screen. When the same lecture was subtitled into Sotho, the ratio changed to 20% on the subtitles, 66% on the speaker and slides, and 14% on the rest of the screen. This wide range is an indication of the difference in the distribution of visual attention in different contexts with different language combinations, different levels of redundancy of information, and differences in audiences.

In order to account for “the audiovisual nature of subtitled programmes”, Romero-Fresco (in press) puts forward the notion of ‘viewing speed’ – as opposed to reading speed and subtitling speed – which he defines as “the speed at which a given viewer watches a piece of audiovisual material, which in the case of subtitling includes accessing the subtitle, the accompanying images and the sound, if available”. The perception of subtitled programmes is therefore a result of not only the subtitle reading patterns, but also the visual elements of the film. Based on the analysis of over seventy-one thousand subtitles created in the course of the Digital Television for All project, Romero Fresco provides the following data on the viewing speed, reflecting the proportion of time spent by viewers looking at subtitles and at the images, proportional to the subtitle presentation rates (see Table 1).

Viewing speed Time on subtitles Time on images
120wpm ±40% ±60%
150wpm ±50% ±50%
180wpm ±60%-70% ±40%-30%
200wpm ±80% ±20%

Table 1. Viewing speed and distribution of gaze between subtitles and images (Romero-Fresco) 

Jensema et al. also suggested that the subtitle presentation rate may have an influence on the time spent reading subtitles vs. watching the rest of the image: “higher captioning speed results in more time spent reading captions on a video segment” (2000, 275). This was later confirmed by Szarkowska et al. (2011), who found that viewers spent more time on verbatim subtitles displayed at higher presentation rates compared to edited subtitles displayed with low reading speed, as illustrated by Figure 8.

Figure 8. Fixation-count based heatmaps illustrating changes in attention allocation of hearing and Deaf viewers watching videos subtitled at different rates.

Figure 8. Fixation-count based heatmaps illustrating changes in attention allocation of hearing and Deaf viewers watching videos subtitled at different rates.

2 Mean fixation duration

Irwin (2004, 94) states that “fixation location corresponds to the spatial locus of cognitive processing and that fixation or gaze duration corresponds to the duration of cognitive processing of the material located at fixation”. Within the same activity (e.g. reading), longer mean fixation durations could therefore be said to reflect more cognitive processing and higher cognitive load. One would therefore expect viewers to have longer fixations when the subject matter is more difficult, or when the language is more specialized. Across activities, however, comparisons of fixation duration is less meaningful as reading elicits more shorter fixations than scene perception or visual scanning simply because of the nature of the activities. It is therefore essential in eye tracking studies of subtitle reading to distinguish between the actual subtitles when they are on screen, the rest of the screen, and the subtitle area when there is no text (between successive subtitles).

The difference between reading and scene perception is illustrated in Figure 9, demonstrating that fixations on the image tend to be longer (indicated here by a bigger circle) than those on subtitles (which indicates more focused viewing), and more exploratory in nature (see the distinction between focal and ambient fixations in Velichkovsky et al. 2005).

Figure 9. Differences in fixation durations between the image and subtitle text – from Polish TV series Londyńczycy.

Figure 9. Differences in fixation durations between the image and subtitle text – from Polish TV series Londyńczycy.

Rayner (1984) indicated the impact of different tasks on mean fixation durations, as reflected in Table 2 below:

Task Mean fixation duration (ms) Mean saccade size (degrees)
Silent reading 225 2 (about 8 letters)
Oral reading 275 1.5 (about 6 letters)
Visual search 275 3
Scene perception 330 4
Music reading 375 1
Typing 400 1 (about 4 letters)

 Table 2. Approximate Mean Fixation Duration and Saccade Length in Reading, Visual Search, Scene Perception, Music Reading, and Typing[1]

In subtitling, silent reading is accompanied by simultaneous processing of the same information in the soundtrack (in the same or another language) as well as of other sounds and visual signs (for a hearing audience, that is – for a Deaf audience, it would be text and visual signs). The difference in mean fixation duration in these different tasks therefore reflects the difference in cognitive load. In silent reading of static text, there is no external competition for cognitive resources. When reading out loud, the speaker/reader inevitably monitor his/her own reading, introducing additional cognitive load. As the nature of the sign becomes more abstract, the load, and the fixation duration increases, and in the case of typing, different processing, production and checking activities are performed simultaneously, resulting in even higher cognitive load. This is inevitably an oversimplification of cognitive load, and indeed the nature of information acquisition between reading successive groups of letters (words) in a linear fashion is significantly different from that of scanning a visual scene for cues.

Undoubtedly, subtitle reading imposes different cognitive demands, and these demands are also very much dependent on the audience. In an extensive study on the differences in subtitle reading between Deaf, hard of hearing and hearing participants, we found a high degree of variation in mean fixation duration between the groups, and also a difference between the mean fixation duration in the Deaf and the hard of hearing groups between subtitles presented at 12 characters per second and 15 characters per second (see Szarkowska et al. forthcoming).

  12 characters per second 15 characters per second
Deaf 241.93 ms 232.82 ms
Hard of hearing 218.51 ms 214.78 ms
Hearing 186.66 ms 186.58 ms

Table 3. Differences in reading subtitles presented at different rates

Statistical analyses performed on the three groups with mean fixation duration as a dependent variable and groups and speed as categorical factors produced a statistically significant main effect, further confirmed by subsequent t-tests that yielded statistically significant differences in mean fixation duration for both subtitling speeds between all three groups. The difference within the Deaf and hard of hearing groups was also significant between 12cps and 15cps. What this suggests is that reading speed has a more pronounced effect on Deaf and hard of hearing viewers than on hearing ones.

3 Subtitle reading

As indicated at the outset, one of the biggest hurdles in studying the processing of subtitles is the fact that the subtitles appear as image on image rather than text on image as far as eye tracking analysis software is concerned. Whereas reading statistics software can therefore automatically mark words as areas of interest in static texts, and then calculate number of regressions, refixations, saccade length, fixation duration and count as related to the specific words, this process has to be done manually for subtitles. The fact that it is virtually impossible to create similar areas of interest on the subtitle words that are embedded in the image over large numbers of subtitles makes it very difficult to obtain reliable eye tracking results on subtitles as text. This explains the predominance of measures such as fixation count and fixation duration as well as shifts between subtitle area and image in eye tracking studies on subtitle processing. As a result, many of these studies do not distinguish directly between looking at the subtitle area and reading the subtitles, and, “they tend to define crude areas of interest (AOIs), such as the entire subtitle area, which means that eye movement data are also collected for the subtitle area when there are no subtitles on screen, which further skews the data” (Kruger and Steyn, 2014, 109).

Although a handful of studies come closer to studying subtitle reading by going beyond the study of fixation counts, mean fixation duration, and shifts between subtitle area and image area, most studies tend to focus on amount of attention rather than nature of attention. Briefly, the exceptions can be identified in the following studies: Specker (2008) looks at consecutive fixations; Perego et al. (2010) add the path length (sum of saccade lengths in pixels) to the more conventional measures; Rajendran et al. (2013) add the proportion of gaze points; Ghia (2012) looks at fixations on specific words as well as regressions; Bisson et al. (2012) look at the number of subtitles skipped, and proportion of successive fixations (number of successive fixations divided by total number of fixations); and in one of the most comprehensive studies on the subject of subtitle processing, d’Ydewalle and De Bruycker (2007) look at attention allocation (percentage of skipped subtitles, latency time, and percentage of time spent in the subtitle area), fixations (number, duration, and word-fixation probability), and saccades (saccade amplitude, percentage of regressive eye movements, and number of back-and-forth shifts between visual image and subtitle).

In a recent study, Kruger and Steyn (2014) provide a reading index for dynamic texts (RIDT) designed specifically to measure the degree of reading that takes place when subtitled material is viewed. This index is explained as “a product of the number of unique fixations per standard word in any given subtitle by each individual viewer and the average forward saccade length of the viewer on this subtitle per length of the standard word in the text as a whole” (2014, 110). Taking the location and start time of successive fixations within the subtitle area when a subtitle is present as the point of departure, the number of unique fixations (i.e. excluding refixations, and fixations following a regression) is determined, as well as the average length of forward saccades in the subtitle. This information gives an indication of the meaningful processing of the words in the subtitle when the number of fixations per word, as well as the length of saccades as ratio of the length of the average word in the audiovisual text are calculated. Essentially, the formula quantifies the reading of a particular subtitle by a particular participant by measuring the eye movement during subtitle reading against what is known about eye movements during reading and perceptual span.

In a little more detail, the formula can be written as follows for video v, with participant p viewing subtitle s”:

10

(Kruger and Steyn, 2014, 110).

This index was validated by performing a comparison of the manual inspection of the reading of 145 subtitles by 17 participants, and makes it possible to study the reading of subtitles over extended texts. In their study, Kruger and Steyn (2014) use the index to determine the relationship between subtitle reading and performance in an academic context, finding a significant positive correlation between the degree to which participants read the subtitles and their performance in a test written after watching subtitled lectures. The RIDT therefore presents a robust index of the degree to which subtitles are processed over extended texts, and could add significant value to psycholinguistic studies on subtitles. Using the index, previous claims that subtitles have a positive or negative impact on comprehension, vocabulary acquisition, language learning or other dependent variables, can be correlated with whether or not viewers actually read the subtitles, and to what extent the subtitles were read.

Conclusion

From this overview of studies investigating the processing of subtitles on the moving image it should be clear that much still needs to be done to gain a better understanding of the impact of various independent variables on subtitle processing. The complexity of the multimodal text, and in particular the competition between different sources of information, means that a subtitled audiovisual text is a substantially altered product from a cognitive perspective. Much progress has been made in coming to grips with the way different viewers behave when looking at subtitled audiovisual texts, but there are still more questions than answers – relating, for instance, to differences in how people process subtitled content on various devices (cf. the HBBTV4ALL project). The use of physiological measures like eye tracking and EEG (see Kruger et al. 2014) in combination with subjective measures like post-report questionnaires is, however, continually bringing us closer to understanding the impact of audiovisual translation like subtitling on the experience and processing of audiovisual texts.

 

Acknowledgements

This study was partially supported by research grant No. IP2011 053471 “Subtitling for the deaf and hard of hearing on digital television” from the Polish Ministry of Science and Higher Education for the years 2011–2014.

 

References

Bisson, Marie-Josée, Walter Van Heuven, Kathy Conklin, and Richard Tunney. 2014. “Processing of Native and Foreign Language Subtitles in Films: An Eye Tracking Study.” Applied Psycholinguistics 35(2):399-418.

Burnham, Denis, Leigh Greg, Noble William, Jones Caroline, Tyler Michael, Grebennikov Leonid and Alex Varley. 2008. Parameters in television captioning for deaf and hard-of-hearing adults: effects of caption rate versus text reduction on comprehension. Journal of Deaf Studies and Deaf Education 13 (3):391-404.

de Linde, Zoé and Neil Kay. 1999. The Semiotics of Subtitling. Manchester: St. Jerome.

Diao, Y., Chandler, P., Sweller, J. 2007. The effect of written text on comprehension of spoken English as a foreign language. The American Journal of Psychology 120(2): 237-261.

Díaz Cintas, Jorge and Marco Fernandez Cruz. (2008) “Using subtitled video materials for foreign language instruction”. In The Didactics of Audiovisual Translation edited by Jorge Díaz Cintas, 201-214. Amsterdam/Philadelphia: John Benjamins.

Díaz Cintas, Jorge and Aline Remael. 2007. Audiovisual Translation: Subtitling. Manchester: St. Jerome.

d’Ydewalle, Géry and Wim De Bruycker. 2003. Reading native and foreign language television subtitles in children and adults. In The mind’s eyes: Cognitive and applied aspects of eye movement research, edited by J. Hyönä, R. Radach and H. Deubel, 444-461. New York: Springer-Verlag.

d’Ydewalle, Géry and Wim De Bruycker. 2007. “Eye Movements of Children and Adults while Reading Television Subtitles.” European Psychologist 12:196–205.

d’Ydewalle, Géry and Ingrid Gielen. 1992. “Attention Allocation with Overlapping Sound, Image, and Text.” In Eye Movements and Visual Cognition: Scene Perception and Reading, edited by Keith Rayner, 415–427. New York: Springer-Verlag.

d’Ydewalle, Géry, Johan Van Rensbergen, and Joris Pollet. 1987. Reading a message when the same message is available auditorily in another language: The case of subtitling. In Eye Movements: From Physiology to Cognition edited by J.K O’Reagan and A. Lévy Schoen, 313-321. Amsterdam: Elsevier Science Publishers B.V. (North-Holland).

Ghia, Elisa. 2012. “The Impact of Translation Strategies on Subtitle Reading.” In Eye Tracking in Audiovisual Translation, edited by Elisa Perego, 155–182. Roma: Aracne Editrice.

Gottlieb, Henrik. 1998. Subtitling. In Routledge Encyclopaedia of Translation Studies, edited by Mona Baker, 244-248. London & New York: Routledge.

Hershler, Orit and Shaul Hochstein. 2005. At first sight: a high-level pop out effect for faces. Vision Research, 45, 1707–1724.

Holmqvist, Kenneth et al. 2011. Eyetracking. A Comprehensive Guide to Methods and Measures. Oxford: Oxford University Press.

Irwin, David E. 2004. Fixation location and fixation duration as indices of cognitive processing. In J.M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world, 105-133. New York, NY: Psychology Press.

ITC Guidance on Standards for Subtitling. Online at: http://www.ofcom.org.uk/static/archive/itc/itc_publications/codes_guidance/standards_for_subtitling/subtitling_1.asp.html

Jensema, Carl. 2000. Eye movement patterns of captioned TV viewers. American Annals of the Deaf vo. 145, no. 3, 275-285.

Karamitroglou, Fotios. 1998. A Proposed Set of Subtitling Standards in Europe. Translation Journal 2(2). http://translationjournal.net/journal/04stndrd.htm

Krejtz, Izabela, Agnieszka Szarkowska, and Krzysztof Krejtz. 2013. “The Effects of Shot Changes on Eye Movements in Subtitling.” Journal of Eye Movement Research 6 (5): 1–12.

Kruger, Jan-Louis and Faans Steyn. 2014. “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–120.

Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew. 2013a. “Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts.” Proceedings of Eye Tracking South Africa 29-31 August 2013, Cape Town.

Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew. 2013b. The impact of subtitles on academic performance at tertiary level. Paper presented at the Linguistics Society of Southern Africa annual conference in Stellenbosch, June, 2013.

Kruger, Jan-Louis. 2013. “Subtitles in the Classroom: Balancing the Benefits of Dual Coding with the Cost of Increased Cognitive Load.” Journal for Language Teaching 47(1):29–53.

Kruger, Jan-Louis, Hefer, Esté, and Gordon Matthew. 2014. Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2. Journal of Eye Movement Research 7(5):4, 1–15.

Langton, Stephen R.H., Anna S. Law, Burton, A. Mike and Stefan R. Schweinberger. 2008. Attention capture by faces. Cognition, 107:330-342.

Pavakanun, Ubowanna. 1992. Incidental acquisition of foreign language through subtitled television programs as a function of similarity with native language and as a function of presentation mode. Unpublished doctoral thesis, Leuven, Belgium, University of Leuven.

Perego, Elisa, Fabio Del Missier, Marco Porta and Mauro Mosconi. 2010. “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13(3):243–272.

Rajendran, Dhevi, Andrew Duchowski, Pilar Orero, Juan Martínez, and Pablo Romero-Fresco. 2013. “Effects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination.” Perspectives: Studies in Translatology 21(1):5–31.

Rayner, Keith. 1984. Visual selection in reading, picture perception, and visual search: A tutorial review. In Attention and performance edited by H. Bouma and D. Bouhwhuis, vol. 10. Hillsdale, NJ: Erlbaum.

Rayner, Keith 1998. “Eye movements in reading and information processing: Twenty years of research.” Psychological Bulletin, 124:372–422.

Robson, Gary D. 2004. The closed captioning handbook. Amsterdam: Elsevier.

Romero Fresco, Pablo (in press) The Reception of Subtitles for the Deaf and Hard of Hearing in Europe. Peter Lang.

Smith, Tim, and John M. Henderson. 2008. Edit Blindness: The relationship between attention and global change blindness in dynamic scenes. Journal of Eye Movement Research 2(2), 6:1-17.

Specker, Elizabeth, A. 2008. L1/L2 Eye Movement Reading of Closed Captioning: A Multimodal Analysis of Multimodal Use. Unpublished PhD thesis. University of Arizona.

Szarkowska, Agnieszka, Krejtz, Izabela, and Łukasz Dutka. (forthcoming) The effects of subtitle presentation rate, text editing and type of subtitling on the comprehension and reading patterns of subtitles among deaf, hard of hearing and hearing viewers. To appear in: Across Languages and Cultures 2016, vol. 2.

Szarkowska, Agnieszka, Krejtz, Izabela, Kłyszejko, Zuzanna and Anna Wieczorek. 2011. “Verbatim, standard, or edited? Reading patterns of different captioning styles among deaf, hard of hearing, and hearing viewers”. American Annals of the Deaf 156 (4):363-378.

Vanderplank, Robert. 1988 “The value of teletext sub-titles in language learning”. ELT Journal 42(4):272-81.

Velichkovsky, Boris M., Joos, Markus, Helmert, Jens R., and Sebastian Pannasch. 2005. Two Visual Systems and Their Eye Movements: Evidence from Static and Dynamic Scene Perception. InCogSci 2005: Proceedings of the XXVII Conference of the Cognitive Science Society, 2283–2288.

Winke, Paula, Susan Gass, and Tetyana Syderenko. 2013. “Factors Influencing the Use of Captions by Foreign Language Learners: An Eye Tracking Study.” The Modern Language Journal 97 (1):254–275.

Yarbus, Alfred L. 1967. Eye movements and vision. New York, NY: Plenum Press.

 

Notes

[1] Values are taken from a number of sources and vary depending on a number of factors (see Rayner, 1984)

 

Bios

Jan-Louis Kruger is director of translation and interpreting in the Department of Linguistics at Macquarie University in Sydney, Australia.  He holds a PhD in English on the translation of narrative point of view. His main research interests include studies on the reception and cognitive processing of audiovisual translation products including aspects such as cognitive load, comprehension, attention allocation, and psychological immersion.

Agnieszka Szarkowska, PhD, is Assistant Professor in the Institute of Applied Linguistics at the University of Warsaw, Poland. She is the founder and head of the Audiovisual Translation Lab, a research group working on media accessibility. Her main research interests lies in audiovisual translation, especially subtitling for the deaf and the hard of hearing and audio description.

Izabela Krejtz, PhD, is Assistant Professor at University of Social Sciences and Humanities, Warsaw. She is a co-founder of Eyetracking Research Center at USSH. Her research interests include neurocognitive and educational psychology. Her applied work focuses on pro-positive trainings of attention control, eye tracking studies in perception of audiovisual material and emotions regulation.


From Subtitles to SMS: Eye Tracking, Texting and Sherlock – Tessa Dwyer

$
0
0

Abstract

As we progress into the digital age, text is experiencing a resurgence and reshaping as blogging, tweeting and phone messaging establish new textual forms and frameworks. At the same time, an intrusive layer of text, obviously added in post, has started to feature on mainstream screen media – from the running subtitles of TV news broadcasts to the creative portrayals of mobile phone texting on film and TV dramas. In this paper, I examine the free-floating text used in BBC series Sherlock (2010–). While commentators laud this series for the novel way it integrates text into its narrative, aesthetic and characterisation, it requires eye tracking to unpack the cognitive implications involved. Through recourse to eye tracking data on image and textual processing, I revisit distinctions between reading and viewing, attraction and distraction, while addressing a range of issues relating to eye bias, media access and multimodal redundancy effects.

Figure 1

Figure 1: Press conference in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

Introduction

BBC’s Sherlock (2010–) has received considerable acclaim for its creative deployment of text to convey thought processes and, most notably, to depict mobile phone messaging. Receiving high-profile write-ups in The Wall Street Journal (Dodes, 2013) and Wired UK, this innovative representational strategy has been hailed an incisive reflection of our current “transhuman” reality and “a core element of the series’ identity” (McMillan 2014).[1] In the following discussion, I deploy eye tracking data to develop an alternate perspective on this phenomenon. While Sherlock’s on-screen text directly engages with the emerging modalities of digital and online technologies, it also borrows from more conventional textual tools like subtitling and captioning or SDH (subtitling for the deaf and hard-of-hearing). Most emphatically, the presence of floating text in Sherlock challenges the presumption that screen media is made to be viewed, not read. To explore this challenge in detail, I bring Sherlock’s inventive titling into contact with eye tracking research on subtitle processing, using insights from audiovisual translation (AVT) studies to investigate the complexities involved in processing dynamic text on moving-image screens. Bridging screen and translation studies via eye tracking, I consider recent on-screen text developments in relation to issues of media access and linguistic diversity, noting the gaps or blind spots that regularly infiltrate research frameworks. Discussion focuses on ‘A Study in Pink’ – the first episode of Sherlock’s initial season – which producer Sue Vertue explains was actually “written and shot last, and so could make the best use of onscreen text as additional script and plot points” (qtd in McMillan, 2014).

Texting Sherlock

Figure 2

Figure 2: Watson reads a text message in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

The phenomenon under investigation in this article is by no means easy to define. Already it has inspired neologisms, word mashes and acronyms including TELOP (television optical projection), ‘impact captioning’ (Sasamoto, 2014), ‘decotitles’ (Kofoed, 2011), ‘beyond screen text messaging’ (Zhang 2014) and ‘authorial titling’ (Pérez González, 2012). While slight differences in meaning separate such terms from one another, the on-screen text in Sherlock fits all. Hence, in this discussion, I alternate between them and often default to more general terms like ‘titling’ and ‘on-screen text’ for their wide applicability across viewing devices and subject matter. This approach preserves the terminological ambiguity that attaches to this phenomenon instead of seeking to solve it, finding it symptomatic of the rapid rate of technological development with which it engages. Whatever term is decided upon today could well be obsolete tomorrow. Additionally, as Rick Altman (2004: 16) notes in his ‘crisis historiography’ of silent and early sound film, the “apparently innocuous process of naming is actually one of culture’s most powerful forms of appropriation.” He argues that in the context of new technologies and the representational codes they engender, terminological variance and confusion signals an identity crisis “reflected in every aspect of the new technology’s socially defined existence” (19).

According to the write-ups, phone messaging is the hero of BBC’s updated and rebooted Sherlock adaptation. Almost all the press garnered around Sherlock’s on-screen text links this strategy to mobile phone ‘texting’ or SMS (short messaging service). Reporting on “the storytelling challenges of a world filled with unglamorous smartphones, texting and social media”, The Wall Street Journal’s Rachel Dodes (2013) credits Sherlock with solving this dilemma and establishing a new convention for depicting texting on the big screen, creatively capturing “the real world’s digital transformation of everyday life.” For Mariel Calloway (2013), “Sherlock is honest about the role of technology and social media in daily life and daily thought… the seamless way that text messages and internet searches integrate into our lives.” Wired’s Graeme McMillan (2014) ups the ante, naming Sherlock a “new take” on “television drama as a whole” due precisely to its on-screen texting technique that sets it apart from other “tech-savvy shows out there”. McMillan continues, that “as with so many aspects of Sherlock, there’s an element of misdirection going on here, with the fun, eye-catching slickness of the visualization distracting from a deeper commentary the show is making about its characters relationship with technology – and, by extension, our own relationship with it, as well.”

As this flurry of media attention makes clear, praise for Sherlock’s on-screen text or texting firmly anchors this strategy to technology and its newly evolving forms, most notably the iPhone or smartphone. Appearing consistently throughout the series’ three seasons to date, on-screen text in Sherlock occurs in a plain, uniform white sans-serif font that appears unadorned over the screen image, obviously added during post-production. This text is superimposed, pure and simple, relying on neither text bubbles nor coloured boxes nor sender ID’s to formally separate it from the rest of the image area. As Michele Tepper (2011) eloquently notes, by utilising text in this way, Sherlock “is capturing the viewer’s screen as part of the narrative itself”:

It’s a remarkably elegant solution from director Paul McGuigan. And it works because we, the viewing audience, have been trained to understand it by the last several years of service-driven, multi-platform, multi-screen applications. Last week’s iCloud announcement is just the latest iteration of what can happen when your data is in the cloud and can be accessed by a wide range of smart-enough devices. Your VOIP phone can show caller ID on your TV; your iPod can talk to both your car and your sneakers; Twitter is equally accessible via SMS or a desktop application. It doesn’t matter where or what the screen is, as long as it’s connected to a network device. … In this technological environment, the visual conceit that Sherlock’s text message could migrate from John Watson’s screen to ours makes complete and utter sense.

Unlike on-screen text in Glee (Fox, 2009–), for instance (see Fig. 3), that is used only occasionally in episodes like ‘Feud’ (Season 4, Ep 16, March 14, 2013), Sherlock flaunts its on-screen text as signature. Its consistently interesting textual play helps to give the series cohesion. Yet, just as it aids in characterisation, helps to progress the narrative, and binds the series as a whole, it also, necessarily, remains at somewhat of a remove, as an overtly post-production effect.

Figure 3

Figure 3: Ryder chats online in ‘Feud’, Glee (2013), Episode 16, Season 4.

While Tepper (2011) explains how Sherlock’s “disembodied” (Banks, 2014) texting ‘makes sense’ in the age of cross-platform devices and online clouds, this argument falters when the on-screen text in question is less overtly technological. The extradiegetic nature of this on-screen text – so obviously a ‘post’ effect – is brought to the fore when it is used to render thoughts and emotions rather than technological interfacing. In ‘A Study in Pink’, a large proportion of the text that pops up intermittently on-screen functions to represent Sherlock’s interiority, not his Internet prowess. In concert with camera angles and “microscopic close-ups”, it elucidates Sherlock’s forensic “mind’s eye” (Redmond, Sita and Vincs, this issue), highlighting clues and literally spelling out their significance (see Figs. 4 and 5). The fact that these human-coded moments of titling have received far less attention in the press than those that more directly index new technologies is fascinating in itself, revealing the degree to which praise for Sherlock’s on-screen text is invested in ideas of newness and technological innovation – underlined by the predilection for neologisms.

Figure 4

Figures 4: Sherlock examines the pink lady’s ring in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

Figure 5

Figures 5: Sherlock examines the pink lady’s ring in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

Of course, even when not attached to smartphones or data retrieval, Sherlock’s deployment of on-screen text remains fresh, creative and playful and still signals perceptual shifts resulting from technological transformation. Even when representing Sherlock’s thoughts, text flashes on screen manage to recall the excesses of the digital, when email, Facebook and Twitter ensconce us in streams of endlessly circulating words, and textual pop-ups are ubiquitous. Nevertheless, the blinkered way in which Sherlock’s on-screen text is repeatedly framed as, above all, a means of representing mobile phone texting functions to conceal some of its links to older, more conventional forms of titling and textual intervention, from silent-era intertitles to expository titles to subtitles. By relentlessly emphasising its newness, much discussion of Sherlock’s on-screen text overlooks links to a host of related past and present practices. Moreover, Sherlock’s textual play actually invites a rethinking of these older, ongoing text-on-screen devices.

Reading, Watching, Listening

As Szarkowska and Kruger (this issue) explain, research into subtitle processing builds upon earlier eye tracking studies on the reading of static, printed text. They proceed to detail differences between subtitle and ‘regular’ reading, in relation to factors like presentation speed, information redundancy, and sensory competition between different multimodal channels. Here, I focus on differences between saccadic or scanning movements and fixations, in order to compare data across the screen and translation fields. During ‘regular’ reading (of static texts) average saccades last 20 to 50 milliseconds (ms) while fixations range between 100 and 500ms, averaging 200 to 300ms (Rayner, 1998). Referencing pioneering studies into subtitle processing by Géry d’Ydewalle and associates, Szarkowska et al. (2013: 155) note that “when reading film subtitles, as opposed to print, viewers tend to make more regressions” and fixations tend to be shorter. Regressions occur when the eye returns to material that has already been read, and Rayner (1998: 393) finds that slower readers (of static text) make more regressions than faster readers. A study by d’Ydewalle and de Bruycker (2007: 202) found “the percentage of regressions in reading subtitles was globally, among children and adults, much higher than in normal text reading.” They also report that mean fixation durations in the subtitles was shorter, at 178 ms (for adults) and explain that subtitle regressions (where the eye travels back across words already read) can be partly explained by the “considerable information redundancy” that occurs when “[s]ubtitle, soundtrack (including the voice and additional information such as intonation, background noise, etc.), and image all provide partially overlapping information, eliciting back and forth shifts with the image and more regressive eye-movements” (202).

What happens to saccades and fixations when image processing is brought into the mix? When looking at static images, average fixations last 330 ms (Rayner, 1998). This figure is slightly longer than average fixations during regular reading and longer again than average subtitle fixations. Szarkowska and Kruger (this issue) note that “reading requires many successive fixations to extract information whereas looking at a scene requires fewer, but longer fixations” that tend to be more exploratory or ambient in nature, taking in a greater area of focus. In relation to moving-images, Smith (2013: 168) finds that viewers take in roughly 3.8% of the total screen area during an average length shot. Peripheral processing is at play but “is mostly reserved for selecting future saccade targets, tracking moving targets, and extracting gist about scene category, layout and vague object information”. In thinking about these differences in regular reading behaviour, screen viewing, and subtitle processing, it is noticeable that with subtitles, distinctions between fixations and saccades are less clear-cut. While saccades last between 20 and 50ms, Smith (2013: 169) notes that the smallest amount of time taken to perform a saccadic eye movement (taking into account saccadic reaction time) is 100-130ms. Recalling d’Ydewalle and de Bruycker’s (2007: 202) finding that fixations during subtitle processing last around 178ms, it would seem that subtitle conditions blur the boundaries somewhat between saccades and fixations, scanning and reading.

Interestingly, studies have also shown that the processing of two-line subtitles involves more regular word-by-word reading than for one-liners (D’Ydewalle and de Bruycker, 2007: 199). D’Ydewalle and de Bruycker (2007: 199) report, for instance, that more words are skipped and more regressions occur for one-line subtitles than for two-line subtitles. Two-line subtitles result in a larger proportion of time being spent in the subtitle area, and occasion more back-and-forth shifts between the subtitles and the remaining image area (201). This finding suggests that the processing of one-line subtitles differs considerably from regular reading behaviour. D’Ydewalle and de Bruycker (2007: 202) surmise that the distinct way in which one-line subtitles are processed relates to a redundancy effect caused by the multimodal nature of screen media. Noting how one-line subtitles often convey short exclamations and outcries, they suggest that a “standard one-line subtitle generally does not provide much more information than what can already be extracted from the picture and the auditory message.” They conclude that one-line subtitles occasion “less reading” than two-line subtitles (202). Extrapolating further, I posit that the routine overlapping of information that occurs in subtitled screen media blurs lines between reading and watching. One-line subtitles are ‘read’ irregularly and partly blind – that is, they are regularly skipped and processed through saccadic eye movements rather than fixations.

This suggestion is supported by data on subtitle skipping. Szarkowska and Kruger (this issue) find that longer subtitles containing frequently used words are easier and quicker to process than shorter subtitles containing low-frequency words. Hence, they conclude that cognitive load relates more to word familiarity than quantity, something that is overlooked in many professional subtitling guidelines. This finding indicates that high-frequency words are processed ‘differently’ in subtitling than in static text, in a manner more akin to visual recognition or scanning than reading. Szarkowska and Kruger find that high-frequency words in subtitles are often skipped. Hence, as with one-line subtitles, high-frequency words are, to a degree, processed blind, possibly through shape recognition and mapping more than durational focus. In relation to other types of on-screen text, such as the short, free-floating type that characterises Sherlock, it seems entirely possible that this innovative mode of titling may just challenge distinctions between text and image processing. While commentators laud this series for the way it integrates on-screen text into its narrative, style and characterisation, eye tracking is required to unpack the cognitive implications of Sherlock’s text/image morph.

The Pink Lady

Figure 6

Figure 6: Letters scratched into the floor in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

Sherlock producer Vertue refers to the pink lady scene in ‘A Study in Pink’ as particularly noteworthy for its “text all around the screen”, referring to it as the “best use” of on-screen text in the series (qtd in McMillan, 2014). In this scene, a dead woman dressed in pink lies face first on the floor of a derelict building into which she has painstakingly etched a word or series of letters (‘Rache’) with her fingernails. As Sherlock investigates the crime scene, forensics officer Anderson interrupts to explain that ‘Rache’ is the German word for ‘revenge’. The German-to-English translation pops up on screen (see Fig. 6), and this time Sherlock sees it too. This superimposed text, so obviously laid over the image, oversteps its surface positioning to enter Sherlock’s diegetic space, and we next view it backwards, from Sherlock’s point of view, not ours (see Fig. 7). After an exasperated eye roll that signals his disregard for Anderson, Sherlock dismisses this textual intervention and we watch it swirl into oblivion. Here, on-screen text is at once both inside and outside the narrative, diegetic and extra-diegetic, informative and affecting. In this way it self-reflexively draws attention to the show’s narrative framing, demonstrating its complexity as distinct diegetic levels merge.

Figure 7

Figure 7: Sherlock sees on-screen text in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

For Carol O’Sullivan (2011), when on-screen text affords this type of play between the diegetic and extra-diegetic it functions as an “extreme anti-naturalistic device” (166) that she unpacks via Gérard Genette’s notion of narrative metalepsis (164). Detailing numerous examples of humourous, formally transgressive diegetic subtitles, such as those found in Annie Hall (Woody Allen, 1977) (Fig. 8), O’Sullivan points to their metatextual function, referring to them as “metasubtitles” (166) that implicitly comment on the limits and nature of subtitling itself. When Sherlock’s on-screen titles oscillate between character and viewer point-of-view shots, they too become metatextual, demonstrating, in Genette’s terms, “the importance of the boundary they tax their ingenuity to overstep in defiance of verisimilitude – a boundary that is precisely the narrating (or the performance) itself: a shifting but sacred frontier between two worlds, the world in which one tells, the world of which one tells” (qtd in O’Sullivan 2011: 165). Moreover, for O’Sullivan, “all subtitles are metatextual” (166) necessarily foregrounding their own act of mediation and interpretation. Specifically linking such ideas to Sherlock, Luis Perez Gonzalez (2012: 18) notes how “the series creators incorporate titles that draw attention to the material apparatus of filmic production”, thereby creating an complex alienation-attraction effect “that shapes audience engagement by commenting upon the diegetic action and disrupting conventional forms of semiotic representation, making viewers consciously work as co-creators of media content.”

Figure 8

Figure 8: Subtitled thoughts in the balcony scene, Annie Hall (1977).

Eye Bias

One finding from subtitle eye tracking research particularly pertinent to Sherlock is the notion that on-screen text causes eye bias. This was established in various studies conducted by d’Ydewalle and associates, which found that subtitle processing is largely automatic and obligatory. D’Ydewalle and de Bruycker (2007: 196) state:

Paying attention to the subtitle at its presentation onset is more or less obligatory and is unaffected by major contextual factors such as the availability of the soundtrack, knowledge of the foreign language in the soundtrack, and important episodic characteristics of actions in the movie: Switching attention from the visual image to “reading” the subtitles happens effortlessly and almost automatically (196).

This point is confirmed by Bisson et al. (2014: 399) who report that participants read subtitles even in ‘reversed’ conditions – that is, when subtitles are rendered in an unfamiliar language and the screen audio is fully comprehensible (in the viewers’ first language) (413). Again, in intralingual or same-language subtitling – when titles replicate the language spoken on screen –hearing audiences still divert to the subtitle area (413). These findings indicate that viewers track subtitles irrespective of language or accessibility requirements. In fact, the tracking of subtitles overrides function. As Bisson et al. (413) surmise, “the dynamic nature of the subtitles, i.e., the appearance and disappearance of the subtitles on the screen, coupled with the fact that the subtitles contained words was enough to generate reading behavior”.

Szarkowska and Kruger (this issue) reach a similar conclusion, explaining eye bias towards subtitles in terms of both bottom-up and top-down impulses. When subtitles or other forms of text flash up on screen, they affect a change to the scene that automatically pulls our eyes. The appearance and disappearance of text on screen is registered in terms of motion contrast, which according to Smith (2013: 176), is the “critical component predicting gaze behavior”, attaching to small movements as well as big. Additionally, we are drawn to words on screen because we identify them as a ready source of relevant information, as found in Batty et al. (forthcoming). Analysing a dialogue-free montage sequence from animated feature Up (Pete Docter, 2009), Batty et al. found that on-screen text in the form of signage replicates in miniature how ‘classical’ montage functions as a condensed form of storytelling aiming for enhanced communication and exposition. They suggest that montage offers a rhetorical amplification of an implicit intertitle, thereby alluding to the historical roots of text on screen while underlining its narrative as well as visual salience. One frame from the montage sequence focuses in close-up on a basket containing picnic items and airline tickets (see Fig. 9). Eye tracking tests conducted on twelve participants indicates a high degree of attentional synchrony in relation to the text elements of the airline ticket on which Ellie’s name is printed. Here, text provides a highly expedient visual clue as to the narrative significance of the scene and viewers are drawn to it precisely for its intertitle-like, expository function, highlighting the top-down impulse also at play in the eye bias caused by on-screen text.

Figure 9

Figure 9: Heat map showing collective gaze weightings during the montage sequence in Up (2009).

In this image from Up, printed text appears in the centre of the frame and, as Smith (2013: 178) elucidates, eyes are instinctively drawn towards frame centre, a finding backed up by much subtitle research (see Skarkowska and Kruger, this issue). However, eye tracking results on Sherlock conducted by Redmond, Sita and Vincs (this issue) indicate that viewers also scan static text when it is not in the centre of the frame. In an establishing shot of 221B Baker Street from the first episode of Sherlock’s second season, ‘A Scandal in Belgravia’, viewers track static text that borders the frame across its top and right hand sides, again searching for information (See Fig. 10). Hence, the eye-pull exerted by text is noticeable even in the absence of movement, contrast and central framing. In part, viewers are attracted to text simply because it is text – identified as an efficient communication mode that facilitates speedy comprehension (see Lavaur, 2011: 457).

Figure 10

Figure 10: Single viewer gaze path for ‘A Scandal in Belgravia’, Sherlock (2012), Episode 1, Season 2.

Distraction/Attraction

What do these eye tracking results across screen and translation studies tell us about Sherlock’s innovative use of on-screen text and texting? Based on the notion that text on screen draws the eye in at least dual ways, due to both its dynamic/contrastive nature and its communicative expediency, we can surmise that for Sherlock viewers, on-screen text is highly visible and more than likely to be in that 3.8% of the screen on which they will focus at any one point in time (see Smith, 2013: 168). The marked eye bias caused by text on screen is further accentuated in Sherlock by the freshness of its textual flashes, especially for English-speaking audiences given the language hierarchies of global screen media (see Acland 2012, UNESCO 2013). The small percentage of foreign-language media imported into most English-speaking markets tends to result in a lack of familiarity with subtitling beyond niche audience segments. For those unfamiliar with subtitling or captioning, on-screen text appears particularly novel. Additionally, as explored, floating TELOPs in Sherlock attract attention due to the complex functions they fulfil, providing narrative and character clues as well as textual and stylistic cohesion. As Tepper (2011) points out, in the first episode of the series, viewers are introduced to Sherlock’s character via text, before seeing him on screen. “When he texts the word ‘Wrong!’ to DI Lestrade and all the reporters at Lestrade’s press conference,” notes Tepper, “the technological savvy and the imperiousness of tone tell you most of what you need to know about the character.”

There seems no doubt that on-screen text in Sherlock attracts eye movement, and that it therefore distracts from other parts of the image. One question then that immediately presents itself is why Sherlock’s textual distractions are tolerated – even celebrated – to a far greater extent than other, more conventional or routine forms of titling like subtitles and captions. While Sherlock’s on-screen text is praised as innovative and incisive, interlingual subtitling and SDH are criticised by detractors for the way in which they supposedly force viewers to read rather than watch, effectively transforming film into “a kind of high-class comic book with sound effects” (Canby, 1983).[2] Certainly, differences in scale affect such attitudes and the quantitative variance between post-subtitles (produced for distribution only) and authorial or diegetic titling (as seen in Sherlock) is pronounced.[3] However, eye tracking research on subtitle processing indicates that, on the whole, viewers easily accommodate the increased cognitive load it presents. Although attentional splitting occurs, leading to an increase in back-and-forth shifts between the subtitles and the rest of the image area (Skarkowska and Kruger, this issue), viewers acclimatise by making shorter fixations than in regular reading and by skipping high-frequency words and subtitles while still managing to register meaning (see d’Ydewalle and de Bruycker, 2007: 199). In this way, subtitle processing reveals many differences to reading of static text, and approximates techniques of visual scanning. Bearing these findings in mind, I propose it is more accurate to see subtitling as transforming reading into viewing and text into image, rather than vice versa.

Situating Sherlock in relation to a range of related TELOP practices across diverse TV genres (such as game shows, panel shows, news broadcasting and dramas) Ryoko Sasamoto (2014: 7) notes that the additional processing effort caused by on-screen text is offset by its editorial function.[4] TELOPs are often deployed by TV producers to guide interpretation and ensure comprehension by selecting and highlighting information deemed most relevant. This suggestion is backed up by research by Rei Matsukawa et al. (2009), which found that the information redundancy effect caused by TELOPs facilitates understanding of TV news. For Sasamoto (2014: 7), ‘impact captioning’ highlights salient information in much the same way as voice intonation or contrastive stress. It acts as a “written prop on screen” enabling “TV producers to achieve their communicative aims… in a highly economical manner” (8). Focusing on Sherlock specifically, Sasamoto suggests that its captioning provides “a route for viewers into complex narratives” (9). Moreover, as Szarkowska and Kruger (this issue) note, in static reading conditions, “longer fixations typically reflect higher cognitive load.” Consequently, the shorter fixations that characterise subtitle viewing supports the contention that on-screen text processing is eased by its expedient, editorial function and by redundancy effects resulting from its multimodality.

Switched On

Another way in which Sherlock’s text and titling innovations extend beyond mobile phone usage was exemplified in July 2013 by a promotional campaign that promised viewers a ‘sneak peak’ at a yet-to-be-released episode title, requiring them to find and piece together a series of clues. In true Sherlockian style, the clues were well hidden, only visible to viewers if they switched on closed-captioning or SDH available for deaf and hard-of-hearing audiences. With this device turned on, viewers encountered intralingual captioning along the bottom of their screen and additionally, individually boxed letters that appeared top left (see Figs. 11 and 12). Viewers needed to gather all these single letter clues in order to deduce the episode title: ‘His Last Vow’. According to the ‘I Heart Subtitles’ blog (July 16, 2013), in doing so, Sherlock once again displayed its ability to “think outside the box and consider all…options”. It also cemented its commitment to on-screen text in various guises, and effectively gave voice to an audience segment typically disregarded in screen commentary and analysis. Through this highly unusual, cryptic campaign, Sherlock alerted viewers to more overtly functional forms of titling, and intimated points of connection between language, textual intervention and access.

Figure 11

Figures 11: Boxed letter clues (top left of frame) that appeared when closed captioning was switched on, during a re-run of ‘A Scandal in Belgravia’, Sherlock (2012), Episode 1, Season 2.

Figure 12

Figures 12: Boxed letter clues (top left of frame) that appeared when closed captioning was switched on, during a re-run of ‘A Scandal in Belgravia’, Sherlock (2012), Episode 1, Season 2.

Conclusion

On-screen text invites a rethinking of the visual, expanding its borders and blurring its definitional clarity. Eye tracking research demonstrates that moving text on screens is processed differently to static text, affected by a range of factors issuing from its multimodal complexity. Sherlock subtly signals such issues through its playful, irreverent deployment of text, which enables viewers to directly access Sherlock’s thoughts and understand his reasoning, while also distancing them, asking them to marvel at his ‘millennial’ technological prowess (Stein and Busse, 2012: 11) while remaining self-consciously aware of his complex narrative framing as it flips inside out, inviting audiences to watch themselves watching. Such diegetic transgression is yet to be mapped through eye tracking, intimating a profitable direction for future studies. To date, data on text and image processing demonstrates how on-screen text attracts eye movement and hence, it can be inferred that it distracts from other parts of the image area. Yet, despite rendering more of the image effectively ‘invisible’, text in the form of TELOPs are increasingly prevalent in news broadcasts, current affairs panel shows (when audience text messages are displayed) and, most notably, in Asian TV genres where they are now a “standard editorial prop” featured in many dramas and game shows (Sasamoto, 2014: 1). In order to take up the challenge presented by such emerging modes of screen address, research needs to move beyond surface assessments of the attraction/distraction nexus. It is the very attraction to TELOP distraction that Sherlock – via eye tracking – brings to the fore.

 

References

Acland, Charles. 2012. “From International Blockbusters to National Hits: Analysis of the 2010 UIS Survey on Feature Film Statistics.” UIS Information Bulletin 8: 1-24. UNESCO Institute for Statistics.

Altman, Rick. 2004. Silent Film Sound. New York: Columbia University Press.

Banks, David. 2012. “Sherlock: A Perspective on Technology and Story Telling.” Cyborgology, January 25. Accessed October 9, 2014.

Batty, Craig, Adrian Dyer, Claire Perkins and Jodi Sita (forthcoming). “Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative.” In Making Sense of Cinema: Empirical Studies into Film Spectators and Spectatorship, edited by Carrie Lynn D. Reinhard and Christopher J. Olson. London and New York: Bloomsbury.

Bennet, Alannah. 2014. “From Sherlock to House of Cards: Who’s Figured Out How to Translate Texting to Film.” Bustle, August 18. Entertainment. Accessed October 9. http://www.bustle.com/articles/36115-from-sherlock-to-house-of-cards-whos-figured-out-how-to-translate-texting-to-film/image/36115.

Biedenharn, Isabella. 2014. “A Brief Visual History of On-Screen Text Messages in Movies and TV.Flavorwire, April 24. Accessed October 13.

Bisson, Marie-Jos´ee, Walter J. B. Van Heuven, Kathy Conklin And Richard J. Tunney. 2014. “Processing of native and foreign language subtitles in films: An eye tracking study.” Applied Psycholinguistics 35: 399–418. Accessed October 13, 2014. doi: 10.1017/S0142716412000434.

Calloway, Mariel. 2013. “The Game is On(line): BBC’s ‘Sherlock’ in the Age of Social MediaMariel Calloway, March 8. Accessed October 14, 2014.

Canby, Vincent. 1983. “A Rebel Lion Breaks Out.” New York Times, March 27, 21.

Dodes, Rachel. 2013. “From Talkies to Texties.” Wall Street Journal, April 4, Arts and Entertainment Section. Accessed October 13, 2014.

d’Ydewalle, Géry and Wim De Bruycker, 2007. “Eye movements of children and adults while reading television subtitles.” European Psychologist 12 (3): 196-205.

Kofoed, D. T. 2011. “Decotitles, the Animated Discourse of Fox’s Recent Anglophonic Internationalism.” Reconstruction 11 (1). Accessed October 5, 2012.

Lavaur, Jean-Marc and Dominic Bairstow. 2011. “Languages on the screen: Is film comprehension related to the viewers’ fluency level and to the language in the subtitles?” International Journal of Psychology 46 (6): 455-462. doi: 10.1080/00207594.2011.565343.

McMillan, Graeme. 2014. “Sherlock’s Text Messages Reveal Our TranshumanismWired UK, February 3. Accessed October 14.

Matsukawa, Rei, Yosuke Miyata and Shuichi Ueda. 2009. “Information Redundancy Effect on Watching TV News: Analysis of Eye Tracking Data and Examination of the Contents.” Literary and Information Science 62: 193-205.

O’Sullivan, Carol. 2011. Translating Popular Film. Basingstoke and New York: Palgrave Macmillan.

Pérez González, Luis. 2013. “Co-Creational Subtitling in the Digital Media: Transformative and Authorial Practices.” International Journal of Cultural Studies 16 (1): 3-21. Accessed September 25, 2014. doi: 10.1177/1367877912459145.

Rayner, K. 1998. “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 124: 372-422.

Redmond, Sean, Jodi Sita and Kim Vincs. 2015. “Our Sherlockian Eyes: The Surveillance of VisionRefractory: a Journal of Entertainment Media, 25.

Romero-Fresco, Pablo. 2013. “Accessible filmmaking: Joining the dots between audiovisual translation, accessibility and filmmaking.” JoSTrans: The Journal of Specialised Translation 20: 201-23. Accessed September 20, 2014.

Sasamoto, Ryoko. 2014. “Impact caption as a highlighting device: Attempts at viewer manipulation on TV.” Discourse, Context and Media 6: 1-10. Accessed September 18 (Article in Press). doi: 10.1016/j.dcm.2014.03.003.

Schrodt, Paul. 2013. “This is How to Shoot Text MessagingEsquire, February 4. The Culture Blog. Accessed October 13, 2014.

Smith, Tim J. 2013. “Watching You Watch Movies: Using Eye Tracking to Inform

Cognitive Film Theory” in Psychocinematics: Exploring Cognition at the Movies, edited by Arthur P. Shimamura, 165-91. Oxford and New York: Oxford University Press. Accessed October 7, 2014. doi: http://dx.doi.org/10.1093/acprof:oso/9780199862139.001.0001.

Stein, Louise Ellen and Kristina Busse. 2012. “Introduction: The Literary, Televisual and Digital Adventures of the Beloved Detective.” In Sherlock and Transmedia Fandom: Essays on the BBC Series, edited by Louise Ellen Stein and Kristina Busse, 9-24. Jefferson: McFarland and Company.

Szarkowska, Agnieszka et. al. 2013. “Harnessing the Potential of Eye-Tracking for Media Accessibility.” in Translation Studies and Eye-Tracking Analysis, edited by Sambor Grucza, Monika Płużyczka and Justyna Zając, 153-83. Frankfurt am Mein: Peter Lang.

Szarkowska, Agnieszka and Jan Louis Kruger. 2015. “Subtitles on the Moving Image: An Overview of Eye Tracking Studies.” Refractory: a Journal of Entertainment Media, 25.

Tepper, Michele. 2011. “The Case of the Travelling Text Message.” Interactions Everywhere, June 14. Accessed October 14, 2014.

UNESCO. 2013. “Feature Film Diversity”, UIS Fact Sheet 24, May. Accessed October 3, 2014.

Zhang, Sarah. 2014. “How Hollywood Figured Out A Way To Make Texting In Movies Look Less Dumb.Gizmodo, August 18. Accessed August 19.

Zhou, Tony. 2014. “A Brief Look at Texting and the Internet in Film”. Video Essay, Every Frame a Painting, August 15. Accessed August 19.

 

List of Figures

 

 

Notes

[1] While some commentators point out that Sherlock was by no means the first to depict text messaging in this way – as floating text on screen – it is this series more than any other that has brought this phenomenon into the limelight. Other notable uses of on-screen text to depict mobile phone messaging occur in films All About Lily Chou-Chou (Iwai, 2001), Disconnect (Rubin, 2013), The Fault in Our Stars (Boone, 2014), LOL (Azuelos, 2012), Non-Stop (Collet-Serra, 2014), Wall Street: Money Never Sleeps (Stone, 2010), and in TV series Glee (Fox, 2009–), House of Cards (Netflix, 2013–), Hollyoaks (Channel 4, 1995–), Married Single Other (ITV, 2010) and Slide (Fox8, 2011). For discussion of some ‘early adopters’, see Biendenharn 2014.

 

Notes

[2] Notably, in this New York Times piece, Canby (1983) actually defends subtitling against this charge, and advocates for subtitling over dubbing.

[3] On distinctions between post-subtitling and pre-subtitling (including diegetic subtitling), see O’Sullivan (2011).

[4] According to Sasamoto (2014: 1), “the use of OCT [Open Caption Telop] as an aid for enhanced viewing experience originated in Japan in 1990.”

 

Bio

Dr Tessa Dwyer teaches Screen Studies at the University of Melbourne, specialising in language politics and issues of screen translation. Her publications have appeared in journals such as The Velvet Light Trap, The Translator and The South Atlantic Quarterly and in a range of anthologies including B is for Bad Cinema (2014), Words, Images and Performances in Translation (2012) and the forthcoming Locating the Voice in Film (2016), Contemporary Publics (2016) and the Routledge Handbook of Audiovisual Translation (2017). In 2008, she co-edited a special issue of Refractory on split screens. She is a member of the ETMI research group and is currently writing a book on error and screen translation.

Our Sherlockian Eyes: the Surveillance of Vision – Sean Redmond, Jodi Sita and Kim Vincs

$
0
0

Abstract

For this inter-disciplinary article, we undertook a pilot case study that eye-tracked the ‘Holmes Saves Mrs. Hudson’ sequence from the episode, A Scandal in Belgravia (Sherlock, BBC, 2012). This small-scale empirical study involved a total of 13 participants (3 males and 10 females, mean age was: 27 years), comprised of a mixture of academics and undergraduate students at La Trobe University in Melbourne, Australia. The article examines its findings through a range of threaded frames – neuroscience, forensics, surveillance, haptics, memory, performance-movement, and relationality – and uniquely draws upon the interests of the authors to set the examination in context. The article is both a reading of Sherlock and a dialogue between its authors. We discover that the codes and conventions of Sherlock have a direct impact on where viewers look but we also discover eyes emerging in the periphery of the frame, and we account for these ways of seeing in different ways.

My Sherlockian Eyes

Sean Redmond

I have always been fascinated, perhaps even obsessed, with my eyes. I have often felt them looking into things, as if they had their own embodied consciousness that I was entirely, simultaneously, conscious of. It was as if we, my eyes and I, saw the world separately and together, possessing a double vision, one set within the meaty windows of my sockets, and the other looking outside, grasping the world with a replete hapticity, sending shivers across my pupils and retinas as they did so.

I have found myself trying to catch my eyes out, to second guess their movements, their sightlines, and their interests. I must be a sight for sore eyes on the rush hour train, wrestling with what I will allow my eyes to see. I often try to resist my conforming eyes, to make them look towards the cultural periphery, to the aesthetic margins, and to the haphazard shards of broken, refracted light on oily windows that few others see as they go about their busy, and sometimes dreary lives. I have a deep yearning to see my eyes politicised, to turn them completely into organs of touch (Marks, 2000), and to feel them wander freely across the intricate layers of the film and television screen. I want Sherlockian eyes.

I have held a rather romantic notion about my viewing eyes, and the eyes of some viewers: that they sometimes wander freely across the spaces, objects, lights, colours, bodies, movements and sounds of the diegetic world they are presented with. Narrative action may be centre frame, and all the elements of the mise en scène may be attempting to draw one’s eyes to this interaction, but I will catch myself looking to the far left of the screen, to hold my sight on an obscure pattern on a wall, or to search for the origins of a distant minor or insignificant sound just off-screen. I want to see inside and outside the narrative simultaneously. I imagine my eyes as Sherlock-like, searching for narrative clues, new plot developments, and for the sensuous expression of character, mood and feeling. But I also see them loosing or freeing themselves; my eyes (unconsciously) float within all the elements of filmic or televisual material as they happen on the screen.

I see in Sherlock’s eyes this double vision: the ability to have foresight, to see into the margins of things, and to be consciously aware of the vision within and all around him. As Sherlock sees into the finest grain of things, so do my eyes and I. My Sherlockian eyes are forensic, haptic, self-processing and are blessed with twenty twenty vision – they have the power to see into all things clearly. Sherlock mirrors or rather embodies the very qualities of the cinema machine (Metz, 1982), and of the surveillance regimes (Foucault, 1977), that emerged at the time the first Sherlock Holmes books were written (1887-1927). Sherlock is a text that already embodies the eye tracking experience.

But is this so, or just a fictive longing? What evidence do we have that our eyes do what we say they do? What evidence do we have that viewers possess a double vision? This romantic, phenomenological notion of the viewing, carnal, haptic eyes, then, we wanted to test, to explore, to see in action and interaction…

The Science of the Sherlockian Eye

Jodi Sita

I can often be found staring off into space, deep in thought, looking at nothing in particular. If I were being eye tracked it would look like I was staring at something. Where people look and, more particularly, why they are looking there, are questions that fascinate me and make me think about the phenomenon of blank stares. Human beings have fascinating eyes that, because they are housed independently, with their own localized environments, need to and can move about quite a lot. A tiny spot on the back of the eye, the retina, houses the receptors for high visual acuity, and this spot must be directed at the object we want to see, in order for us to see it clearly, and see its fine detail. People tend to move their eyes to aspects of a scene which are interesting or useful. The visual system in the brain directs these movements; they are not random. Bottom–up control processes (see; Itti & Koch, 2000) help direct some shifts in visual gaze and involve features that are thought to attract attention due to their ability to be noticeable. These include salient features such as luminosity, colour and movement. More importantly, in the human gaze we know that top-down processes are at play when viewing complex or meaningful scenes; our eyes employ feature selection which are based on our understanding of the scene and our internal expectations about where important things are or are likely to occur (Torralba et al., 2006; Birmingham et al., 2008, and Vincent et al., 2009).

What I am curious to learn more about is how our viewing behavior is shaped by what we are doing, by how we are interacting with the world and how our brains are responding to that and shaping that encounter. Thus, my involvement in this work comes from these curiosities, however, it also stems from my own forensic tendencies. It develops from my own need to ask the Sherlockian questions about viewers viewing Sherlock.

My early research had me investigate a branch of forensic science; handwriting and signature examination. At the time the area had a lot of practitioners, many quite experienced and successful, and there had been a substantial amount written about it, yet very little objective evidence for the fields’ claims had been produced. What the field needed were studies which produced hard evidence to support or dispute its original claims. My work was part of a large and ongoing body of work, where the field’s ideas and claims are tested objectively, and whose results can be used as evidence to support existing notions or derive new ones. It was within this area of research that I started using eye tracking, yet it has also led me to want to bring eye tracking to this moving-image field; where the focus of the eye tracker, shining like an objective lens over some of the theories of the area, can help bring to it some other method for its practitioners to use – to examine how viewers watch and are involved with what they are watching.

The Optometry of Sherlock

Kim Vincs

The science of the gaze—of how eyes fixate or fail to fixate—has always been of great interest to me, firstly in my original career as an optometrist, and more latterly as a choreographer and then a transmedia dance artist. As an optometrist, I was less concerned with where people looked than with whether they could look, and with the accuracy and resolution of the sensory information they received and interpreted when they did look. What do I mean by this? In considering whether people could look, I am referring to whether they were able to accurately fixate the static and moving targets they wanted and needed to. Fixating, that is, aiming one’s eyes at a static or moving target, is a function of attention, and integrated as action by the sensory and muscular systems of the eye and brain. There are many pathological conditions that interfere with the capacity to fixate a static visual stimulus quickly, accurately and efficiently. As an optometrist, I was primarily concerned with detecting these conditions and referring patients who had them for appropriate treatment. I was, in a very real sense, perfectly happy to allow my patients to decide for themselves what to fixate on. My job was simply to ensure that, should they wish to, they would be capable of locating and tracking something. This willingness to allow dissociation between capacity and will, between ability and decision, is something I consider foundational to the ways in which I have pursued my subsequent research into creative practices. I have never, as a choreographer or an interactive / transmedia artist, wished to dictate to people where they should look or what they should perceive. I consider my job to be to place appropriate objects / events / movements within a context in which they can be perceived should people so choose.

This outlook has had some specific implications for my art practice. As a choreographer, I have never thought to ask what someone watches when they observe a dancer moving. Cognitive psychologist Kate Stevens’ seminal work on eye movements in dance has demonstrated a classic novice/expert shift in the way that observers view dance. As with many other fields of expertise such as airline pilots, and driving instructors, experts make significantly fewer saccades, that is, changes in fixation, watching a dance performance than do novices, where experts are people with professional experience in dance and novices are people with no particular prior experience of the artform (Stevens et al., 2010). The implication of these results is that experts do not need to change fixation as many times as novices because they are able, to some extent, to predict where the dancing body will move. In essence, they understand what they are looking for, and are therefore able to maximize the efficiency of their fixation choices.

What Steven’s work does tell us is which movement features most attract fixation when watching a dancing body. My own work in motion capture analysis of dance movement provides me with a theory about why this might be a difficult thing to measure. Dance, at the movement level, comprises movement of some 33 major joints, each of which may make movements of entirely different velocity, acceleration and magnitude to achieve an overall aesthetic effect. The dancing body essentially has no ‘centre of focus’ that can be interpolated from movement data such as the speed, momentum or even position of specific body parts, because the semantics of dance movement are only meaningful in relation to the composition across the body. As I have argued previously, (Vincs, 2014) the semantic significance of a movement bears no relationship to its metrics, such as amplitude or speed. In some aesthetic contexts, tiny movements of the fingers may be essential to the meaning and feeling tone of the movement. In others, such as large virtuosic or acrobatic movement forms, hand gestures may contribute relatively little to a movement’s significance.

Dance grammars are aesthetically and culturally, rather than anatomically determined. I think that this fact has also contributed to my attraction to the notion of Sherlockian eyes. As a choreographer, I am always a detective, seeking potential significances in movement rather than predetermined ones. I value the opportunity to go looking for the dancing body, browsing, shuffling, wandering through the multiple and complex joint actions that comprise a single ‘step,’ looking for something of newness and emotional value rather than assuming I know what it is and where I will find it. Yet I am always aware that my aesthetic search is underpinned by a neurosensory apparatus that is primed to respond selectively to human movement (Hagendoorn, 2004, Vincs, 2009). I am therefore armed, at least potentially, with an inherent ‘grammar’ that is defined by the morphology and physical capacity of the human body, and I am curious as to what predilections and biases my visual sensory system imposes on my seemingly adventurous gaze.

What now follows is an exploration of our different approaches to the eye tracking data that we generated. Jodi is first off, outlining our empirical method and undertaking a close reading of the preliminary results. Jodi shows how the results begin to tell us that the viewers’ gaze patterns and fixations are closely clustered together, and she situates these findings in relation to the science of the eye, the importance of the face in human communication, and to the visual and narrative codes and conventions of Sherlock. Sean then explores the results in terms of haptic visuality and the surveillance gaze, drawing upon phenomenology and the discourses of conspiracy to argue that the vision in Sherlock is marked by touch, texture, and control. Kim examines the results in terms of movement and relationality, examining the eye tracking data in terms of the way it supports and confirms the necessary nature of vision in seeing into moving things. Kim shows that even though there is a high degree of direction in terms of where viewers are being asked to look, visual perception allows or enable the eyes to wander. Finally, we conclude our article together, drawing together our voices to offer an interdisciplinary way forward.

Eye Tracking Sherlock (the objective viewing): Methods and Preliminary Results

Jodi Sita

We undertook a pilot case study that eye-tracked the ‘Holmes Saves Mrs. Hudson’ sequence from the episode, A Scandal in Belgravia (Sherlock, BBC, 2012). This small-scale empirical study involved a total of 13 participants (3 males and 10 females, mean age was: 27 years), comprised of a mixture of academics and undergraduate students at La Trobe University in Melbourne, Australia.

A Tobii X-120 remote eye tracker (Tobii Technology, Stockholm, Sweden) was used to record participants eye movements which has an accuracy of 0.5ºof visual angle and allows a moderate amount of free head movement (30 x 22 x 30 cm at 70 cm (Width x Height x Depth)). This data collection technique uses reflected infrared light from the eye to determine participants viewer gaze positioning and allows for natural head movements and natural human responses to screened material. The eye tracker was connected to a PC running an Intel ® core ™ i7 CPU ‘Cool Master’ hard-drive. The eye tracker used Tobii Studio 2.3.2 professional edition software for the presentation of the movie scene stimuli and recording eye movements. The eye tracker was set up on a desk, situated below a Dell PC monitor (1680×1050), which was utilised by the participants to view the Sherlock sequence. Participants were seated on a sturdy chair between 55-65cm away from the eye tracker and between 65-75cm from the viewing screen. A second screen (Dell; 1920×1080) was utilised by the researchers to view, in real time, the eye movements of the participants as they were being tracked and calibrated, although all computer analyses and statistics reported here was based on stored data.

Participants were recruited via posters advertising the study at La Trobe University, with ethics approval (Ethics approval number: FHEC13/101). Participants were required to be at least 18 years of age to be considered eligible. People whom expressed interest in taking part in the study were contacted via email to attend a single recording session. In preliminary tests participants were introduced to the study and screened for exclusion criteria such as taking medications (e.g. benzodiazepines) that may potentially affect their eye movement, known neurological conditions, disorders or injuries that could potentially affect their eye movement. All participants were screened for normal or corrected to normal near visual acuity of N8 or better on the Designs for Vision near sighted visual acuity test, and with a pen-torch eye movement excursion test to screen for symmetrical movement of the eyes. Participants who were ametropic were allowed to wear their glasses to watch the stimuli.

Prior to eye movement data being collected, the eye tracker was configured for each participant using a 9-point on screen calibration test within the Tobii Studio recording software. Participants were told only that they would watch short segments from a variety of films. Recording sessions typically lasted between 15-25 minutes, and each participant was tested individually.

First, we found that our viewers’ eyes were strongly drawn to follow movement and directional cues and signs. This included camera and character movement. In the opening scene, where Mrs. Hudson’s fingers scrape along the wall, followed by Sherlock’s fingers retracing her steps (03-010 seconds), we see all viewers making strings of successive fixations – each following these finger movements (see Figures 1 and 2). The sound of these fingers scraping along the wall was heavily amplified, and fully sychronised, and we suggest, then, that sound was also an aesthetic device being employed to direct where viewer’s looked. These results confirm previous findings where camera movement, sound, character behavior, and editing patterns are seen to inform gaze patterns and fixations (see Smith, forthcoming, Smith and Mital, 2011).

In one brief shot in the middle of this scene, we cut to a close-up of Mrs. Hudson’s face, full of anguish. All the subjects discern this face in amongst the movement and chaos of the surrounding action, as seen by their fixating to its features. Her face is captured in the center of the screen, making it central to the scene’s visualisation. However, the face is known to be a strong attractor of what the eye attends to (Treuting (2006)), and with it being such an important narrative component in this scene, would have been a strong attentional cue.

Figure 1: Finger drag: 2 subjects

Figure 1: Finger drag: 2 subjects

Figure 2: Finger drag: 13 subjects

Figure 2: Finger drag: 13 subjects

Second, we observed an alignment in vision with regards to where Sherlock was looking. This sight co-proximation is referred to as ‘joint attention,’ in which what one attends to seems to shift automatically to where another is looking (Birmingham et al, 2009). Interestingly, this is a common misdirection trick used by magicians (Kuhn, et al, 2009).

In particular, we observed that Sherlock’s’ point of view in the scene very often produced a close proximity in viewers focus and attention (participant’s looked where Sherlock looked, and with the same overall gaze patterning, see from 1.05 to 1.14). This also supports the findings reported by other film scholars using eye tracking methods, such as Rassell et al. (forthcoming, 2015) who found that a character’s point of view and subjective experiences have an influence on where  viewers look.

The trends for this short sequence support the idea that Sherlock is a character-driven drama in which his vision is not only foregrounded but given omnipotent and omniscient power. Thus, viewers are not only being positioned to observe from his authorial position but to trust where he looks and what he discovers there. There are recognizable genre codes and conventions also in play, structuring the looking patterns we have observed. This is a detective-thriller series that repeats a series of camera and editing motifs that become familiar to audiences (Neale, 1990).

Sherlock Image 3

Figure 3: A heap map showing the hot spots where viewer’s gazed; red indicating longer dwelling time

Figure 4: The gaze plots showing the sequence of looks that viewers made over Sherlock’s face

Figure 4: The gaze plots showing the sequence of looks that viewers made over Sherlock’s face

Third, we found that viewers focused heavily on the characters faces, both in scenes with dialogue and those without. In scenes where Sherlock was clearly putting together the evidence, viewers focused heavily on his eyes, dwelling there for almost the entire shot. (Figures 3 and 4).

Viewers fixated back and forth between the eyes, face, and mouth of the central characters. These viewing patterns are characteristic of the movements made in facial and emotional recognition (Ekman & Friesen, 1971; Hernandez, et al, 2009) and show some indication that viewers were paying attention to the different character’s in the scene, working out the role of each character and what their intentions and emotions might be. These patterns of eye movements suggest that viewers are engaging with the scene as they would in a normal face-to-face encounter, using eye movements to verify who people are and what they are feeling. It is interesting to note that people who are not able to perform these socially informative tasks, such as those with the disorder schizophrenia, and with some traumatic brain injuries, do not show the same eye movement behaviors (Watt & Douglas, 2006; Loughland et al, 2002; Williams,et al, 1999).

Our viewers clearly followed narrative cues in line with the dialogue exchanges, looking back and forth between the character’s interpersonal relays (Figures 5, 6 and 7). These results are similar to those of Treuting (2006), who eye-tracked 14 participants viewing short clips from such films as Shawshank Redemption (1994) and Harry Potter and the Philosopher’s Stone (2001). Treuting found that gaze clusters emerged in and around the central character’s faces involved in dialogue and moments of heightened drama (see also, Redmond, Sita, 2013).

Sherlock-5.-Single-viewer-charecter-alighnment

Figure 5: Single viewer character alignment

Figure 6: 6 subjects and the relay of looks on eyes, mouths and faces.

Figure 6: 6 subjects and the relay of looks on eyes, mouths and faces.

Figure 7: Final scene, last 12 seconds, searching for information: 13 subjects (most of the fixations are falling over the faces of the 2 central character’s in dialogue)

Figure 7: Final scene, last 12 seconds, searching for information: 13 subjects (most of the fixations are falling over the faces of the 2 central character’s in dialogue)

Fourth, we saw evidence that viewers searched for narrative information and cues: this included fixating on aspects of the background wall before Sherlock first enters the scene (from 0.33-0.36 seconds), then moving between the image of a smile seen on the wall and Holmes’ face, spending time ‘reading’ the shop window signs and the note on the front door (Figure 8) as Watson arrives at the scene (from 2.22 to 2.37). One can understand such scanning as influenced by the meticulous work of the mise en scène: where all the elements have been carefully placed to enact this type of searching for narrative cues (see Smith, forthcoming).

Clip-8

Figure 8: Searching for narrative information

Finally, albeit in relation to our last point, we observed that certain viewers looked at more elements of the mise en scène (Figure 9, shows gaze patterns for 4 of the 13 viewers), including the interior lights, the computer, and furniture, even as the more dramatic moments of the scene were taking place.

These findings were interesting but not totally unexpected; we would hope that not all people viewing the same scene would watch it in the same way (this is something that is discussed further below). Insights like this allow us to see that even though there are some aspects that are strong attention grabbers, such as faces and movement within a scene, other aspects can captivate and draw attention away from those areas of interest. For example, the scene shown below (Figure 9) involves a particularly emotive exchange between two key characters, Mrs. Hudson and Dr. Watson. The fact that 4 of the viewers were attending elsewhere helps us to see these aspects of interest outside the main narrative at play. Why certain viewers look to the margins of the screen, to the more ‘insignificant’ elements of the mise en scène remains of great interest. One possibility would be that these viewers were not fully engaged with the exchange between the characters, and their attention therefore drifted to other elements in the scene. Another possibility is that these scenic elements drew particular interest because of their pattern, colour, etc. Further testing is needed to begin to tease out whether this response is scene-dependent, or a characteristic of these particular observers.

Figure 9: A slightly different patterning – 4 subjects – and wider viewing

Figure 9: A slightly different patterning – 4 subjects – and wider viewing

It should be noted, nonetheless, that these observations come from only a very small sample (13 participants to date), which will be increased, and which still needs to undergo further data analysis and interpretation.

In summary, what have we see in these results so far? Evidence of the eyes being held to attention by narrative cues, by camera and character movement, faces, dialogue, point of view and performance. These were elements to be expected and add to the growing body of eye tracking evidence that supports much of current film and television theory, particularly those working in the cognitive tradition such as David Bordwell (2007) and Noël Carroll (1996). The results equally support the results of other studies into; narrative centered visual texts (see Batty et al, forthcoming, 2015); and how sound and movement affect gaze patterns (Rassell et al, forthcoming, 2015) Further, they speak to the way viewers are pulled seamlessly into the diegetic worlds they believe and invest in.

In this article we would now like to apply two different theoretical filters to the results just summarized; the first will be an examination of the gaze, by Sean, and the second will be an examination of the physiological and perceptual processes of the eye in relation to movement, by Kim. Both filters will attempt to make deeper sense of the results from the traditions in which the scholars operate from. Following this analysis, a summary conclusion will draw their approaches together to make further inter-disciplinary sense of Sherlock’s eyes.

Sherlock’s Gaze

Sean Redmond

The concept of the gaze has a long and contentious history in Film Studies if much less so in the study of television. In fact, John Ellis has suggested that the domestic context in which television viewing has historically taken place, with a host of likely distractions, and in a context of constant programme flow and segmentation, produces a glance aesthetic whereby the image isn’t looked into deeply or for a sustained period of time (1982: 138). Sherlock, of course, contests this idea since the programme is heavily built around the details of forensic gazing.

Most notably, the idea of the gaze has been employed in psychoanalytical film theory to make the argument that the cinema looking apparatus is patriarchal and heterosexual, and viewers are positioned as ‘male’ subjects through which masculine identifications emerge (Mulvey, 1975, 1989). Its main male characters, and male writers and directors of course control the vision regime in Sherlock, although its objects of focus are rarely to-be-looked-at female characters.

Critical race theory, by contrast, has employed the concept of the gaze to demonstrate how the racial Other is fixed in inferior, marginal and fetishized subject positions (Hall, 2001). Sherlock can be read as a post-colonial text enacting a present England that centres whiteness and ‘invisibly’ marginalizes the Other from its panopticon empowered centre (Cuningham, 2004). The Other in Sherlock of course extends to those who sit outside the bourgeois social centre; there are particular class dimensions to the way crime is surveyed and defined (Jann, 1990).

In terms of surveillance discourse, film has been read as a vision machine set within a invasive visual culture that promotes:

the normalizing gaze, a surveillance that makes it possible to qualify, to classify and to punish. It establishes over individuals a visibility through which one differentiates and judges them (Foucault, 1977: 25).

Sherlock can be read as a text that carries out this normalizing gaze, defining the parameters of law and order and the way the criminal can be discovered, classified, and ultimately disciplined. That is not to say that the visual excesses of the programme do not at times undermine its simple binaries. To the contrary, Sherlock is constantly troubled by its own dominant discourses particularly through the way Sherlock is also a maverick outsider.

Finally, film phenomenology has made use of the gaze to demonstrate how looking and seeing is always embodied, experiential, and depending on the text, haptic and synesthetic – where ‘the eyes themselves function like organs of touch’ (Marks, 2010, 162). Sherlock creates the conditions of both embodied presence and haptic visuality through the way the gaze is employed to see deeply into things, while the programmes textural mise en scène ‘demands’ to be attended to.

What I would now like to do is analyse two particular aspects of the way the gaze can be understood in Sherlock, relating my reading back to the eye tracking results that we have, and to eye tracking technology itself. First, I will explore Sherlock through its forensic gazing and the way this creates the particular conditions for the way viewers become locked into particular viewing patters and relations. Second, I will explore Sherlock through its haptic elements whereby the viewer is understood to gaze at and touch (things) simultaneously.

The Forensic Gaze

In Sherlock one can argue that camera movement and position are motivated by the following factors. First, to reveal narrative information such as a new location, or setting; character relations and their relative physical proximity; time, and temporal detail; and moments of revelation where a new angle or focus reveals something previously hidden or a new ‘enigma’ emerges. Second, as a dramatic device: the camera is re-positioned to signal and cue moments of narrative development, crisis, reaction, and activation. Third, there are repeated and recognised televisual conventions of the programme: one can locate and expect certain camera motifs to function in Sherlock, such as the way we enter Sherlock’s mind’s eye to see what he is unearthing in microscopic close-up. Finally, camera movement and position signals certain emotional states and modes of feeling. The cut to a close-up, for example, a moment of affecting intensity, such as is the case with the fingers being scraped along the wall in the scene under analysis in this article.

When one takes these Sherlockian codes and conventions into consideration one can make better sense of the eye tracking results that we have gotten. The eyes of the viewer seem to be relentlessly led and directed. Viewers familiar with the programme’s codes can be expected to have expectations of its visual tapestry, and to make predictions about where to look (see Rassell et al. forthcoming). This would explain both the way that viewers seem closely aligned with the looking operations of the scene (figures 1-7), and the way that viewers scan shots for narrative information (figures 8-10).

However, I also think there is something more telling to discover here – one around a consistent forensic looking regime where the text and the viewer align. This is the ‘double vision’ we refer to in my introduction to this article. Viewers come to embody the gazing powers that Sherlock possess and look at the diegetic world through his eyes even where no direct or imagined point of view is in operation. Viewers experience their very own form of social surveillance becoming detectives and snoopers in the process. Sherlock, then, can be read as a text of and for paranoid surveillance, fuelled by the constant search for facts, omissions, falsehoods, and half-truths. At a more general cultural level, trust is at issue here in what is perceived to be an age of ‘faithless’ activity and widespread corruption, where politicians are regarded to be as corrupt as the criminals they covertly support. Sherlockians ultimately become part of this age of conspiracy (Knight, 2002).

As does, in a very real sense, eye tracking technology and the data it produces. Sherlock is his very own eye tracker – he creates his own heat maps and relays and through this inbuilt biotechnology he sees into everything. Eye tracking technology is Sherlocklian and the data it produces allows us to see into everything the viewer sees. Or, at least mostly…

The Haptic Gaze

Laura U Marks (2010) has written that haptic visuality is a more intimate form of looking, where the eyes, ‘move over the surface of its object rather than plunge into illusionist depth, not to distinguish form so much as to discern texture” (162). For Marks, film and video may be “thought of as impressionable and conductive, like skin.” (2000: xi-xii) and this sensory materiality is heightened by it containing:

Grainy, unclear images; sensuous imagery that evokes memory of the senses (i.e. water, nature); the depiction of characters in acute states of sensory activity (smelling, sniffing, tasting, etc.); close-to-the-body camera positions and panning across the surface of objects; changes in focus, under- and overexposure, decaying film and video imagery; optical printing; scratching on the emulsion; densely textured, effects and formats such as Pixelvision… and alternating between film/video.  (Totaro, 2002)

The gaze found in Sherlock is very often a haptic one. The programme’s entire mise en scène evokes the activity and memory of sensation. Lights, objects, clothes, furniture, exteriors are given deep and layered textures. Sherlockian environments are populated with objects and qualities that are themselves sensory driven (poison, oil, tactile fabrics, beads of sweat, cigarette smoke, wet soil). The camera very often dwells on these, picks them out, and tracks and pans over them in close and proximate detail. Sherlock of course is a master of haptic visuality – his eyes touches the things that he observes or that he conjures up in his imagination. In many respects, then, the viewer is also invited to see Sherlock through a haptic lens.

If one was to return to the eye tracking results on the scene what we might be observing is not just an alignment in vision, and the search for narrative information, but eyes that have been turned into organs of touch and deep sensual appreciation. For example, in image 1 and 2 viewers are not just following the fingers that scrape along the wall but touching (with) them, and in touching them feeling them as if it is their fingers suffering this pain. In figures 5-7 viewers are not just following the relay of looks between the two characters but ‘touching’ faces, eyes and mouths. In figures 8-10, viewers are not just searching for narrative information and clues but are actively seeing into the textures, lights, objects, items that populate those scenes. The heat maps that eye tracking technology can generate may be more apt than we imagine since the suggestion of temperature, of body-heat, may well give truth to the embodied and carnal nature of vision. This is one of the limitations of eye tracking technology, however, since it cannot tell us what people are feeling when watching a film or television text.

I would like to make one final observation about the eye that searches the mise en scène for narrative information or clues, as in figures 8-10. This is a point about the privacy or individualism of watching a screen text so co-dependent it is on personal memory, biography, and the contexts one finds one self-viewing something. After the viewing of the scene one of our subjects remarked that they had actually spent much of the time trying to figure out whom the actor was playing Dr Watson. Any number of ‘personal’ factors might get in the way of the looking regime of the text and for why we might scan a particular text.

Roland Barthes (1981) has usefully employed the concept of the punctum (a Latin word derived from the Greek word for trauma) to viewing photographs. He argues that the still image inspires an intensely private meaning, one in which an affecting ‘partial object’ emerges from its centre to ‘prick’ or ‘wound’ the viewer. The punctum is personal and as soon as it emerges it holds the viewer’s gaze. Although Barthes is singularly writing about the photograph I think the idea of the punctum can be applied to the moving image text, to Sherlock. Although a dynamic media, television and film still settle on images and representations that reach out into the private realm of the viewer; and the viewer still finds their memories, traumas, life events activated and mobilised in the fictive worlds constructed. The wandering eyes of figures 8-10 are caught in their own biographical exploration, looking for objects that may ultimately wound them, their haptic eyes. In Sherlock we are not just being positioned as objects and subjects of surveillance but as carnal beings.

The Eyes of Sherlock; the Eyes of the Viewer

Kim Vincs

Coming, as I do, from the perspective of a dancer and choreographer, I read these eye tracking results not exactly as touch, but as a search for relationality. Erin Manning, in her seminal work on the philosophy of movement (Manning, 2009) emphasizes the incipiency of movement—movement as something in the act of becoming something, of reaching towards something or someone—over its positionality. That is to say, movement, in Manning’s terms, is always a process of relationality or reaching towards the world, rather than simply a series of coordinates on a grid. Bodies, or, more precisely “bodies-in-the-making,” are a means of thought rather than simply of action because they define, by the possibilities embedded in the moment of pre-articulation, a relationship or set of relationships within the world (Manning, 2009: 78).

Our eye movement data defines very clearly the relationships between the protagonists. In figures 1 and 2, ‘finger drag,’ the fixations map the pathway of Mrs. Hudson’s fingers along the wall. The relationships between Mrs. Hudson’s fingers and the texture of the wall are, in fact, the only potential human relationships within the scene. These relationships are not ‘positional’— that is to say, fingers defining points on the wall—but, by virtue of their spatial distribution, an articulation of the trajectory of Mrs. Hudson’s hand in relation to the wall.

Figures 3—6 reveal fixation patterns that are concentrated around the face, and in particular the eyes. These fixations address the origin of relationality in the scenes within the eyes, as if the eyes are understood to reveal the incipient thought of the characters. In Manning’s terms, incipiency the moment in which a movement is in a state of ‘pre-acceleration,’ organizing and mobilizing itself, yet still capable of any number of actual outcomes, is the most potent aspect of a movement. The predominance of fixations on the eyes and face, while perfectly interpretable in as simply a biological reflex designed to respond to and recognize human faces, also speaks of the process by which relationality is thought into being.

Figures 7—10, with their fixations distributed between human-to-human gaze and gesture, (Figure 7) non-human scenic elements that lend cognitive elements or ‘clues,’ (Figure 8) and poetic elements such as the look and texture of surrounding objects in Figures 8 (curtain texture) and 10 (lens flare), articulate an expanded notion of relationality in which human and non-human elements are implicated within a web of actions.

For me, these results suggest a mutability between human and non-human elements that is reminiscent of Manning’s understanding of relationality as something that can have both interpersonal and person-world dimensions, and also points to the kind of ‘Sherlockian body’ I seek as a dance artist. A purely ‘narrative’ approach to the filmic bodies analysed here would suggest that only human factors would feature in the gaze analysis. Similarly, a biologically driven notion of human movement perception as pre-wired to detect human shapes and actions would not predict migrations to and from the bodies to surrounding objects.

I read these results as revealing a semantic ambiguity that can form the basis for a Sherlockian search for relationality that is produced and constructed by the viewer as much as it is dictated by the film-maker. These scenes offer relatively few visual details from which to construct such a relationality, and this, no doubt, is indicative of a film-making style designed to direct the eye, and hence the mind, to very specifically and deliberately arranged narrative events. However, despite the seeming prescriptiveness of these images, they offer the viewer an opportunity, to construct relational scenarios across like and non-like (human and non-human) scenic elements. They therefore demonstrate at least the potential for a Sherlockian approach to the visual perception of movement.

Conclusion

So, what have we seen with your eyes? On the one hand, we have demonstrated how Sherlock’s narrative and mise en scène pulls the viewer into taking certain (emotive, forensic, relational) viewing positions. Sherlock is tightly bound by a number of codes and conventions and its palette and composition are highly constructed, creating spaces and interactions that focus our eyes. On the other hand, we have looked at the way haptics and relationality open up the possibility for better understanding the synaesthetic and organic/inorganic ways through which the vision of movement takes place. A televisual text such as Sherlock intends to marshal our viewing experience but as we have also seen, eyes wander; they search, their own movement and the poetics of movement opening up textual encounters not always pre-determined by the scenes deliberate operations. Our eyes escape themselves. Finally, we have noted how memory and biography can impact on where and why we look, or why we might look away. One of our respondents searched Watson’s face in the hope of remembering the actor’s name. The brutal violence metered out on Mrs. Holmes may be looked at (felt) differently by someone who has themselves undergone such misery. This is ultimately about our-being-in-the-world, and about how we make sense of beings-in-the-fictive world. Vision is never disembodied but full of the drink, food and love of life itself.

Vision needs to be understood as something that can only be fully understood through combining different theoretical and methodological frameworks. What we have found in this article, and in the broader work of the Eye Tracking the Moving Image Research group – see the introduction to this special edition for more on this group – is that it is in the conversations and deliberations, provocations and discussions between vision scientists, neuroscientists, anatomists, choreographers, film makers, ethnographers, screen theorists, and screen writers where insights are best made, conclusions thickened, and arguments enriched and extended. What we have found in this article is that Sherlock exists at the eye of the ArtsScience nexus, and it is at this nexus where the authors would like to situate their work.

The language of film and television constantly creates these spaces of vision and for seeing to take place, whether this be; the embodied point-of-view shot which allows us to become the character; aerial cinematography that brings wide open exteriors and bejeweled cityscapes into view; the furtive camera that glimpses into dark corners, allowing us to happen on to what is supposedly hidden; or the interiorized gaze that expressionistically captures the nightmare visions of the lost, the hunted, and the alien. With a close-up shot one can trace the undulating valleys of emotion on a character’s face and feel their affecting eyes reaching out and into yours. In Sherlock, the very force of his vision, mobilized through special effects and the power of digital photography, enables his/our eyes to create mathematical formulations out of thin air and to re-visit crime scenes as if one is witnessing, experiencing it all over again. Sherlock establishes the sense that all vision is embodied and personally engineered, and to the wider conceit that television is an artform that gives a miraculous, omnipotent and omnipresent vision to its ever watchful viewers.

 

References

Barthes, Roland. 1981. Camera lucida: Reflections on photography. Macmillan.

Batty, Craig, Dyer, Adrian, Perkins, Claire and Sita, Jodi. 2015. “Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative.” In Making Sense of Cinema, edited by CarrieLynn D. Reinhard and Christopher Olson, New York: Bloomsbury, forthcoming.

Birmingham, Elina, Bischof, Walter. F., and Kingstone, Alan. 2008. “Gaze selection in complex social scene.” Visual Cognition 16 (2–3): 341–355

Bordwell, David. 2007. Poetics of Cinema. London: Routledge.

Carroll, Noël. 1996. Theorizing The Moving Image, Cambridge, Cambridge University Press.

Cuningham, Henry. 1994. “Sherlock Holmes and the Case of Race”. The Journal of Popular Culture, 28: 113–125. doi: 10.1111/j.0022-3840.1994.2802_113.x

Ellis, John. 1982. Visible Fictions, London: Routledge.

Ekman, Paul and Friesen, Wallace, V. 1971. “Constants across cultures in the face and emotion.” Journal of Personality and Social Psychology Vol 17 (2): 124-129.

Foucault, Michel. 1977. Discipline and Punish: the Birth of the Prison, New York: Random House.

Itti, Laurent and Koch, Christof. 2000. “A saliency-based search mechanism for overt and covert shifts of visual attention.” Vision Research 40 (10–12): 1489–1506.

Jann, Rosemary. 1990. “Sherlock Holmes codes the social body”. ELH: 685-708.

Knight, Peter. (Ed.). 2002. Conspiracy nation: The politics of paranoia in postwar America. New York: New York University Press.

Kuhn, Gustav, Tatler, Ben, and Cole, Geoff. 2009. “You look where I look! Effect of gaze cues on overt and covert attention misdirection.” Visual Cognition 17 (6/7): 925-944.

Hagendoorn, Ivar. 2004. “Some speculative hypotheses about the nature and perception of dance and choreography”. Journal of consciousness studies, 11, 79 – 110.

Hall, Stuart. 2001. :The Spectacle of the Other”. Discourse theory and practice: A reader, Routledge: 324-344.

Hernandez, Nadia, Metzger, Aude, Magné, Rémy, Bonnet-Brilhault, Frédérique, Roux, Sylvie, Barthelemy, Catherine, and Martineau, Joëlle. 2009. “Exploration of core features of a human face by healthy and autistic adults analyzed by visual scanning.” Neuropsychologia 47(4): 1004-1012.

Loughland, Carmel, M., Williams, Leanne. M., and Gordon, Evian. 2002. “Visual scanpaths to positive and negative facial emotions in an outpatient schizophrenia sample.” Schizophrenia Research, 55: 159–170.

Manning, Erin. 2009. Relationscapes: movement, art, philosophy. Kindle edition Cambridge, Mass: MIT Press.

Marks, Laura U. 2000. The Skin of the Film: Intercultural Cinema, Embodiment and the Senses, Duke University Press.

Metz, Christian. 1982. The imaginary signifier: Psychoanalysis and the cinema. Indiana University Press.

Mulvey, Laura. 1975. “Visual Pleasure and Narrative Cinema”. Screen, 16(3), 6-18.

Mulvey, Laura. 1989. Visual and Other Pleasures. London: Macmillan.

Neale, Steve. “Questions of Genre”. Screen: 31(1) (1990): 45-65.

Rassell, Andrea, Redmond, Sean, Robinson, Jenny, Stadler, Jane, Verhagen, Darrin and Pink, Sarah. 2015. “Seeing, Sensing Sound: Eye Tracking Soundscapes in Saving Private Ryan and Monsters, Inc.”. In Making Sense of Cinema, edited by CarrieLynn D. Reinhard and Christopher Olson, New York: Bloomsbury, forthcoming.

Redmond, Sean and Sita, Jodi. 2013. “What eye tracking tells us about the way we watch films”, 5th December, The Conversation. Accessed 5th September 2014.

Smith, Tim, J. 2013. “Watching You Watch Movies: Using Eye Tracking to Inform Cognitive Film Theory.” In Psychocinematics: Exploring Cognition at the Movies, edited by Arthur P. Shimamura, Oxford University Press.

Smith, Tim. J. and Mital, Parag. K. 2011. “Watching the world go by: Attentional prioritization of social motion during dynamic scene viewing.” [conference abstract]. Journal of Vision, 11(11): 478.

Stevens, Catherine, Winskel, Heather, Howell, Claire, Vidal, Lyne, Latimer, Cyril, and Milne-Home, Josephine. 2010. “Perceiving Dance: Schematic Expectations Guide Experts’ Scanning of a Contemporary Dance Film. Journal of Dance Medicine & Science, 14(1), 19 – 25.

Torralba, Antonio, Oliva, Aude., Castelhano, Monica. S., Henderson, John, M. 2006. “Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search.” Psychological Review, 113 (4): 766–786.

Totaro, Donato. 2002. “Deleuzian Film Analysis: The Skin of the Film”. Off-Screen,  (accessed August 1st, 2011).

Treuting, Jennifer. 2006. “Eye tracking and cinema: A study of film theory and visual perception.” Society of Motion Picture and Television Engineers, 115(1): 31-40.

Vincent, Benjamin.T., Baddeley, Roland, Correani, Alessia, Troscianko, Tom, and Leonards, Ute. 2009. “Do we look at lights? Using mixture modelling to distinguish between low- and high-level factors in natural image viewing”. Visual Cognition, 17 (6–7), 856–879.

Vincs, Kim, and Barbour, Kim. 2014. “Snapshots of Complexity: using motion capture and principal component analysis to reconceptualise dance”. Digital Creativity, 25(1) 62-78.

Vincs, Kim, Schubert, Emery, and Stevens, Catherine. 2009. “Measuring responses to dance: is there a ‘grammar’ of dance?” Proceedings of the World Dance Alliance Global Summit, Brisbane, July 14-16, 2008.

Watts, Amber, J., and Douglas, Jacinta, M. 2006. “Interpreting facial expression and communication competence following severe traumatic brain injury”. Aphasiology, 20, 707–722.

Williams, Leanne, M., Loughland, Carmel, M., Gordon, Evian. and Davidson, Dean. 1999. “Visual scanpaths in schizophrenia: Is there a deficit in face recognition?”. Schizophrenia Research, 40: 189–199.

 

Bios

Sean Redmond is an Associate Professor in Media and Communication at Deakin University. He has research interests in film and television aesthetics, film and television genre, film authorship, film sound, stardom and celebrity, and film phenomenology. He has published nine books including The Cinema of Takeshi Kitano: Flowering Blood (Columbia, 2013), and Celebrity and the Media (Palgrave, 2014), and with Su Holmes he edits the journal Celebrity Studies. Sean Redmond and Jodi Sita set up the Eye Tracking the Moving Image Research group in 2011.

Jodi Sita is an academic working within the areas of neuroscience and anatomy with expertise in eye tracking research. She has extensive experience, with multiple project types using eye tracking technologies and other biophysical data. Her current research uses eye tracking to study viewers gaze patterns while watching moving images; to examine expertise in Australian Rules Football League coaches and players and to examine the signature forgery process.

Professor Kim Vincs is Director and founder of Deakin Motion.Lab, at Deakin University. Kim integrates scientific and artistic approaches through research. She is currently working on a three-year project, supported by the Australian Research Council’s Discovery program. Her collaborations with mathematicians, biomechanists and cognitive psychologists span Deakin and the Universities of Sydney, Western Sydney and New South Wales.

Politicizing Eye tracking Studies of Film – William Brown

$
0
0

Abstract

This essay puts eye tracking studies of cinema into contact with film theory, or what I term film-philosophy, so as to distinguish film theory from specifically cognitive film theory. Looking at the concept of attention, the essay explains how winning and keeping viewers’ attention in a synchronous fashion is understood by eye tracking studies of cinema as key to success in filmmaking, while film-philosophy considers the winning and keeping of attention by cinema to be a political issue driven by economics and underscored by issues of control. As such, film-philosophy understands cinema as political, even if eye tracking studies of film tend to avoid engagement in political debate. Nonetheless, the essay identifies political dimensions in eye tracking film studies: the legitimization of the approach, its emphasis on mainstream cinema as an object of study and its emphasis on statistical significance all potentially have political connotations/ramifications. Invoking the concept of cinephilia, the essay then suggests that idiosyncratic viewer responses, as well as films that do not synchronously capture attention, might yield important results/play an important role in life in an attention-driven society.

In this essay, I wish to put eye tracking studies of film into dialogue with a more political approach to film, drawn from film theory, or what, for the benefit of distinguishing film theory from cognitive film theory, I shall term film-philosophy. In doing so, I shall draw out what for film-philosophy are some of the limitations of eye tracking, including its emphasis on statistical significance, or what most viewers look at when they watch films. I shall argue that we might learn as much, if not more, about cinema by paying attention not only to statistically significant and shared responses to films (what most viewers look at), but also to those viewers whose responses to a film do not form part of the statistically significant group, and/or to films that may not induce in viewers statistically significant and shared responses. In effect, we may find that there are insights to be derived from those who look at the margins of the cinematic image, rather than at the centre, even if those viewers are themselves ‘marginal’ in the sense that they are pushed to the margins of most/all eye tracking studies of film viewers. There is perhaps also value to be found in looking at ‘marginal’ films. In this way, we might find that idiosyncratic responses to a film or films is as important as the shared response. I shall also argue that there is a politics to the idiosyncratic response, especially when it is put into dialogue with film theoretical/film-philosophical work on cinephilia, and that as a result there is also a politics to eye tracking and its emphasis on statistical significance. I shall start, however, by looking at the state of eye tracking film research today.

On 29 and 30 July 2014, the Academy of Motion Picture Arts and Sciences (AMPAS) – the same American academy that distributes the so-called Oscars – held two events under the combined title of ‘Movies in Your Brain: The Science of Cinematic Perception’. The events included contributions from neuroscientists Uri Hasson, Talma Hendler and Jeffrey M. Zacks, psychologist James E. Cutting, directors Darren Aronofsky and Jon Favreau, editor Walter Murch and writer-producer Ari Handel. The host of the first evening was psychologist Tim J. Smith, whose eye tracking studies of cinema have arguably become the best known and most influential over recent years (see, inter alia, Smith 2012a; Smith 2013; Smith 2014). Through these events, as well as through coverage of these events in fashionable journals like Wired (Miller 2014a; Miller 2014b), we can see how eye tracking – together with the study of film using brain scanning technologies such as functional Magnetic Resonance Imaging (fMRI) – is clearly becoming important for our understanding of how films work. This in turn means that such studies are surely important to film studies.

For a detailed history and overview of eye tracking, explaining how it works and what it tells us about film, I cannot do better than to guide readers to the afore-mentioned work by Smith. Smith has soundly demonstrated, and with great clarity, how the human eye moves via small movements called saccades, and that in between saccades the human eye fixates. It is during fixations that humans take in visual information, with fixations being linked therefore to attention and to working memory; we tend to remember objects from our visual field upon which we have fixated, or to which we have paid attention. Clearly this is important to the study of film, since viewers typically attend only to parts of the movie screen at any given time, and not necessarily to others or to the whole of the screen (and the surrounding auditorium). Can/do filmmakers exert influence over where we look, for how long, and thus what we remember about a film – with those memories themselves lasting for greater or lesser periods of time? And if filmmakers do influence such things, how much influence do they exert and through which techniques? These are the questions that eye tracking technology can help to answer – and scholars like Smith do so with great skill and eloquence.

My aim, however, is not simply to reproduce findings by Smith and others who have used eye tracking devices to study film. In order to construct a theoretical argument concerning the importance of the idiosyncratic, or ‘cinephilic’, response to a film or films in general, as well as the importance of a filmmaker not necessarily ‘controlling’ where a viewer looks, but instead allowing/encouraging viewers precisely to look idiosyncratically, cinephilically, or where they wish, I need instead to bring the scientific and ‘apolitical’ use of eye tracking devices into a political discourse concerning the nature of cinema, power, hegemony and the issue of cinematic homogeneity and/or heterogeneity. This is a controversial maneuver – in that it will bring together two areas of film studies that often seem to stand in ‘opposition’ to each other, namely cognitive film theory and a film theory that still plies its trade using Continental philosophy, or what for the sake of simplicity I shall term film-philosophy. My desire is not simply to be controversial, however. Rather it is to engage with what eye tracking means to film studies, both currently and potentially in the future.

To begin to bring eye tracking studies of film into the ‘political discourse’ mentioned above, I shall relate an anecdote. A semi-regular response from colleagues in film studies, when I tell them about eye tracking studies of film viewing, is that eye tracking doesn’t tell us anything about films that we didn’t already know. Is it a surprise that we tend to look more often at the center of the screen? Is it a surprise that we typically attend more to brightly illuminated parts of the screen than to dimly lit ones? Is it a surprise that we tend to direct our attention towards human faces when watching a film that features human characters? Anyone who has consciously thought about what they do while watching a film will be able to tell from memory alone that these things are all true. As a result, eye tracking studies of film can sometimes be filled with what, at least to the film student/scholar, are truisms. By way of an example, Paul Marchant and colleagues say that ‘these strategies and techniques… [capture] the audience’s visual attention: focus, camera movement, eye line match, color and contrast, motion of elements within the shot, graphic matching’ (Marchant et al. 2009, 158). On my print-out of Marchant et al.’s essay, my own apostil next to this assertion reads as follows: ‘Do we not know this already (otherwise cinema would not have developed these techniques)?’ Many, if not all, film viewers will know simply from experience that these techniques help to guide their attention, even if they are blissfully unaware of the relationship between eye fixations, attention and memory. Of course, it is pleasing to have our introspective responses to/our intuitive knowledge about cinema ‘scientifically’ confirmed (to a large extent, but not entirely – about which, more later); but essentially, so my colleagues’ argument goes, eye tracking studies tell us what we already know.

Now, even if I myself find some eye tracking studies of film to be ‘truistic’, I nonetheless believe that eye tracking studies of film are of great importance. However, their importance is perhaps in playing a role that is different from the one that eye tracking studies of film seem to give to themselves, which is as a key component of cognitive film theory. Instead, I think that eye tracking studies of film are important for film theory, or what today is termed film-philosophy. I shall explain the distinction between cognitive film theory and film-philosophy presently.

Little in this world is uniform, and so by definition I generalize when I say that the basic tenet of cognitive film theory is, with David Bordwell and Noël Carroll’s Post Theory: Reconstructing Film Studies (1996) serving as its figurehead, for film studies to move towards a theory of cinema based on the analysis of films themselves, and away from a film theory that uses cinema as a means of confirming or denying a Lacanian understanding of the human and/or an Althusserian/Marxist conception of contemporary capital. In spite of cognitive film theory’s lack of uniformity, eye tracking studies of film are nonetheless part of cognitive film theory’s project to help us to look at cinema ‘as it is’, and not to use cinema as a political football. Conversely, film-philosophy is in general informed by the kinds of Continental philosophers, often though not limited to Gilles Deleuze, that cognitive film theorists reject, and it engages not just with films ‘as they are’, but with the politics of films.

Now, to claim that we can isolate films and film viewing from a human world that is perhaps always political, and to claim that we can then analyse films ‘as they are’, is perhaps absurd: films ‘as they are’ are part of a political world, and cognitive film theorists are not unaware of this, just as film-philosophers are not incapable of scientific analysis. However, how much politics is allowed into the analysis of films perhaps informs the broad distinction between cognitive film theory and film-philosophy, as I hope to clarify by looking briefly at the role of attention in the work of two scholars, Tim J. Smith and Jonathan Beller. In his ‘Attentional Theory of Cinematic Continuity’ (AToCC), Smith (2012a) uses eye tracking studies to demonstrate how filmmakers capture and maintain viewers’ attention, with certain techniques, mainly those associated with continuity editing, being more successful than others. Meanwhile, in his Cinematic Mode of Production: Attention Economy and the Society of the Spectacle, Beller (2006) suggests that capturing attention is not necessarily an aesthetic, but rather a political project: the more attention a film garners, the more success one will have in monetizing that film, with the making of money becoming the bottom line of cinema. Beller does not appeal to some early cinema that did not attempt to elicit viewers’ attention and thus make money; such an early cinema did not necessarily exist. Rather, Beller argues that cinema has always been part of an economy that is based on attention; indeed, cinema plays a key role in naturalizing this attention economy, meaning that cinema has not always been necessarily capitalist, but that the capitalist world endeavors as much as possible to become cinematic, to capture our attention as much as possible in order to ‘win’ the economic race, since capturing eyeballs means making money. Smith explains how attention is captured; Beller offers an explanation as to why. Even though filmmakers rely on natural processes in order to capture attention (Smith), the process of consistently trying to capture our attention (‘cinema’) is not natural, but political and economic (Beller).

James E. Cutting, in commenting on an earlier draft of this paper, says that the results of eye tracking studies of film, which reveal how filmmakers capture attention, are

big news… because almost nothing else does this – not static pictures (photographs, artworks), not class room behavior by teachers, not leaders of business meetings, and often not even spectacles of various kinds (sporting events, rock concerts, etc); even TV is typically not as good as the average narrative, popular movie. (Cutting, signed peer review 2014)

If cinema is indeed better at capturing our attention than these other media, and if in some senses it is better at capturing our attention than those parts of the world that do not feature such media – i.e. if cinema is better at capturing our attention than reality – then cinema, and the making-cinematic of reality in a bid to capture attention, to make money and/or to influence people (Cutting compares cinema in particular to teachers and to business leaders) is profoundly political. It is profoundly political because learning about how to capture attention – learning about how cinema works – is tied to the shaping of our material reality (putting screens everywhere) and to controlling attention (encouraging us to look at those screens, and not at the rest of reality). Cognitive film theory is apolitical; film-philosophy, meanwhile, engages in the very political dimensions of cinema. Eye tracking studies of film tend to position themselves as part of the former; my aim here is to bring them into dialogue with the latter.

If eye tracking studies of film tend to position themselves as part of a would-be apolitical approach to cinema, then in their investigation into cinema, they are nonetheless conducting an investigation into politics, as per Beller’s equation of cinema with politics highlighted above. However, while eye tracking studies of film position themselves as apolitical, politics do creep into eye tracking studies, especially through what I shall call their absences. What is more, these politics do relate to film-philosophy’s ‘political’ approach to film. In order to demonstrate this, I shall begin by analyzing how eye tracking studies of film have sought historically to legitimate themselves.

Early in an essay that gives an overview of eye tracking studies of film, Smith asserts, without naming any, that the hypotheses of film theory ‘generally remain untested’ (Smith 2013, 165). In this almost throwaway comment, we perhaps find important information. For in asserting that eye tracking is what can help us to ‘test’ out some theories of film, as Smith goes on to do in relation to Sergei M. Eisenstein’s writing about his own film, Alexander Nevsky (USSR, 1938), he perhaps overlooks how film theorists often (but perhaps not always) try (though not always with success) to construct their theories based on the films that they have seen, studied and perhaps made, and not the other way around. That is, Smith seems not to consider that watching films is itself a means of testing our theories about films – without the need for eye tracking devices. On a related note, while he does consider filmmakers like Eisenstein, D.W. Griffith, Edward Dmytryk and others as ‘experimentalists’ of sorts (who have tested their own theories), Smith also does not fully acknowledge that the history of cinema can itself be seen as a prolonged ‘test’ in what ‘works’ or ‘does not work’ with audiences – with that which ‘works’ being regularly adopted as either a short- or a long-term strategy by the film industry, be that in terms of re-using storylines, adopting a specific cinematic style, employing bankable film stars, using topical settings, engaging with zeitgeist themes and so on. Instead, it is Smith’s intervention that will validate or otherwise that history of theory and practice, and which will confirm what filmmakers, and perhaps also many audience members, have probably known for a long time, even if putting their knowledge into practice sometimes proves harder than we might imagine (because otherwise films would presumably not have ‘mistakes’ in them).

Now, it’s natural that a (relatively) new approach to studying film would need to legitimize itself in order to gain credibility and following – and Smith clearly charts the c30 year trajectory of eye tracking in film studies since the 1980s onwards (Smith 2014: 90). Nonetheless, if the history of cinema is not ‘test’ enough for Smith, then implicitly a claim is being made here about what constitutes a ‘real’ test, and, by extension, what sort of person can carry out a ‘real’ test. In other words, eye tracking, and the cognitive framework more generally, here legitimizes itself as being a tool for verifying (scientifically) what previously were ‘mere’ and speculative theories (these are my terms) – with the people qualified to carry out these tests being neither filmmakers nor audience members, but psychologists. By justifying eye tracking in this way, Smith is not just making a statement of fact (eye tracking demonstrates that viewers look at the same things at the same time during films made using the continuity editing style), but he is also – I assume unintentionally – making an implicit value judgment that carries political assumptions regarding what constitutes a/the most legitimate framework for learning and knowing about film. If, as per my anecdote above, I can and do know the same things via introspection that eye tracking tells me, then why is introspection not equally legitimate as a framework, even if the former involves less visible labor, and certainly less sexy imagery, and thus does not seem to involve any real ‘testing’?

Eye tracking thus seeks ‘politically’ to legitimate itself as a tool for film analysis. To be clear: eye tracking is legitimate, but it is also always already making claims about what constitutes knowledge: introspection is not knowledge, while science is – even if both can lead to the same understanding. Importantly, in producing visible evidence (the afore-mentioned ‘sexy imagery’ of colored clusters of eye-gaze on scenes from films), then eye tracking studies are also always already cinematic, by which I mean to say that they affirm a system whereby the visual/the cinematic (here are pictures of attention being captured) are validated above invisible (here, introspective) approaches to the same knowledge. This in turn always already affirms the process of cinema and attention-grabbing as being the (political) system that is most powerful.

If eye tracking affirms a politically cinematic world, in that cinematic forms of knowledge are more valid than invisible, i.e. uncinematic, ones, then within that cinematic world eye tracking might also, and in some respects implicitly does, legitimate some forms of cinema over others. This is suggested by the way in which eye tracking studies look predominantly at Hollywood/mainstream cinema in their analyses of film. For example, in his AToCC, Smith (2012a) cites a diverse range of movies, including L’escamotage d’une dame au théâtre Robert Houdin/The Vanishing Lady (Georges Méliès, France, 1896) and L’année dernière à Marienbad/Last Year at Marienbad (Alain Resnais, France/Italy, 1961), but eye tracking data are given mainly for contemporary Hollywood films, including Blade Runner (Ridley Scott, USA/Hong Kong/UK, 1982), Requiem for a Dream (Darren Aronofsky, USA, 2000) and There Will Be Blood (Paul Thomas Anderson, USA, 2007), with Smith suggesting that continuity editing is the form of cinema best suited to capturing attention.1

The absence of eye tracking data on those other, non-Hollywood films is perhaps telling, as suggested by two respondents to Smith’s essay, who query how his theories would apply to different cinemas, including the avant garde (Freeland 2012, 40-41) and, at least by implication, Japanese cinema (Rogers 2012, 47-48). Eye tracking would of course yield important insights into avant-garde and other forms of cinema, but that information is not offered here.

Furthermore, Smith’s suggestion that continuity editing is the form best suited to capturing attention, also prompts Paul Messaris and Greg M. Smith to argue that continuity editing violations, in particular jump cuts, are quite regular and not particularly detrimental to the continuity of the film viewing experience (Messaris 2012, 28-29; Greg M. Smith 2012, 57). Malcolm Turvey, meanwhile, argues that the film viewing experience is always continuous, meaning that the ‘continuity’ of continuity editing ‘is not continuity of viewer attention per se… but rather the manner in which films engage and manage that attention’ (Turvey 2012, 52-53; for Smith’s riposte to these responses and more, see Smith 2012b).

These responses highlight how filmmaking ‘perfection’ (an absence of continuity errors) need not be fetishized too much; audiences are quite happy to watch films with continuity errors (many of which they will not notice). Furthermore, many audiences love what Jeffrey Sconce (1995) might term ‘paracinema’ – i.e. ‘trash’ cinema and ‘bad’ movies – be they intentionally ‘bad’ or otherwise. In other words, it would seem that as long as audiences are primed regarding how they should receive a film (or, in Turvey’s language, as long as their attention is managed and then engaged in the right way), then you don’t need to care about and can even love the stylised acting, the ropey mise-en-scène, the unmotivated camera movements, the strange edits and the story loopholes of, say, The Room (Tommy Wiseau, USA, 2003), supposedly the worst film in history. Under the right circumstances (with the right management/ preparation), it would seem that audiences can like pretty much anything, including a 485-minute film of the Empire State Building (Empire, Andy Warhol, USA, 1964). In other words, while in his AToCC Smith mentions Méliès and Resnais, and while he engages with Eisenstein and other filmmakers elsewhere, the AToCC puts an emphasis on mainstream Hollywood cinema and its predominant system of continuity editing, since this cinema elicits a synchronicity of response, or control over attention, in that viewers attend to the same parts of the screen at the same time – while also often failing to detect edits done in the continuity editing style (see Smith and Henderson 2008). There is a seeming bias here towards mainstream, narrative filmmaking, the engrossing nature of which is lauded at the expense of other cinemas.

Let us move away from Smith in order to demonstrate how this bias is not his alone. Jennifer Treuting suggests that ‘[t]he use of eye tracking… can help filmmakers and other visual artists refine their craft’ (Treuting 2006: 31). In some respects, this is an innocent comment; I have no doubt that eye tracking can help filmmakers and other visual artists to refine their craft. But suggested in this ‘refinement’ is also the move towards validating the mainstream/continuity style at the expense of its alternatives. A combined eye tracking and fMRI study carried out by Uri Hasson and colleagues also makes this clear: much fuss is made over how work by Alfred Hitchcock elicits greater synchrony (‘inter-subject correlation’) in viewers than does an ‘unstructured’ shot of a concert in Washington Square Park, a film that is simply a ‘point of reference’ and which ‘fails to direct viewers’ gaze’ (Hasson et al. 2008, 13-14; emphasis added). My reference above to Warhol’s Empire here becomes apposite: what Hasson and colleagues dismiss as a ‘point of reference’ and as a ‘failure’ in various respects defines one of the great experimental films. Perhaps ‘marginal’ films like Empire should also be considered successful – but at achieving something different to the work of Hitchcock, and perhaps Hasson’s film is not a ‘point of reference’, but an experimental work that equally inhabits the totality of films in the world that we shall call cinema.

If Hitchcock ‘succeeds’ in controlling viewers’ attention, while Warhol by implication ‘fails’, then eye tracking becomes implicitly/inevitably embroiled in not just what film is, but in what film could or should be – as Treuting’s suggestion that eye tracking might feed back into filmmaking also makes clear. This suggests that there is a politics to eye tracking film studies, particularly in the UK where universities are increasingly relying on ‘impact’, particularly on the economy, in order to survive: they don’t just observe films, but feed back into how films are, or should be, made, by exploring what is ‘successful’ in terms of eliciting attention, getting bums on seats and thus making money. In some respects, eye tracking in particular and cognitive film theory in general are now dragged back towards the Marxist approach to cinema that cognitive film theory initially sought to reject: it, too, shapes/seeks to shape cinema just as Marxist film theory in effect lobbied for alternatives to the mainstream. However, where Marxist film theory lobbied for a rejection of mainstream cinematic techniques, eye tracking studies seem to validate them – and to suggest that filmmakers might ‘refine their craft’ by adopting/intensifying them. Saving the thorny issue of ‘control’ and ‘influence’ for later, there is still a political dimension to this potential validation of mainstream cinema techniques, because it reaffirms the economic hegemony of one style over others and it also validates in some degree a homogeneity of product (and of audience?) – all within a ‘cinematic’ economic system that is itself predicated upon gaining attention. Cinema is both business and art, but if art is one thing it is unique/different, and so a move towards homogeneity is a move towards the reduction of art in favor of business. If it requires an artist rather than an academic to make this clear, then Darren Aronofsky’s apprehensive response to Hasson’s work at the AMPAS events hopefully serves this purpose: ‘“It’s a scary tool for the studios to have,” Aronofsky said. “Soon they’ll do test screenings with people in MRIs.” The audience laughed, but it didn’t seem like he was joking, at least not entirely’ (Miller 2014b).

I have so far argued that cinema is political, that eye tracking studies have required some political maneuvering in order to legitimate themselves, and that the focus on continuity editing/mainstream cinema by eye tracking studies may also have a political dimension. However, are eye tracking studies themselves without methodological politics, in that they simply report findings? I wish presently to suggest that eye tracking research does have methodological limitations – which is why I asserted above that eye tracking film studies are only to a large extent and not entirely reliable – and that these limitations also have a political dimension. The methodological limitations are not simply a case of potential inaccuracies regarding the type of eye-tracker used, determining how long the eye needs to be still for a fixation to take place, what algorithm is used to measure this, or how accurate is the eye-tracker in determining where exactly the eye is looking – all ongoing issues with eye tracking technologies (see, inter alia, Wass et al. 2013; Saez de Urabain et al. 2014). It is also a case of issues of statistical significance and the politics thereof, particularly what I shall call the temporal politics, and to a lesser extent the social politics, of eye tracking. In relation to the latter, many eye tracking studies involve students in order to carry out their research (e.g. Tatler et al. 2010; Võ et al. 2012). As a result, the findings might pertain not universally, but to population members who are of a certain age and, if we can say that university students tend to be from more affluent backgrounds, a certain socioeconomic status. In relation to statistical significance, meanwhile, all studies tend to discount those viewers who do not look where the researchers want them to look; for example, in a study of where people look when viewing moving faces, only 87 per cent of fixations targeted the face region when shown a moving face with sound, with that figure dropping to 82 per cent when shown a moving face without sound (Võ et al. 2012, 7). Of course, when what one is investigating is where people look when they look at faces, it is correct to discount those 13-18 per cent of fixations that were not directed at the face. But the point is that similar discounts happen all the time, not least in the process of averaging that we see in various experiments, including those mentioned by Marchant et al., Hasson et al., and Smith. And yet, where neuroscience is based in large part upon the study of anomalous brains – from autists to damage sufferers to perceived geniuses – psychologists engaged in eye tracking tend to go with force majeure and report the average, or what most people do. There may however be in human populations a ‘long tail’ (to use the terminology of Chris Anderson, 2006) that may not in any one experiment be statistically significant, but which over a number of experiments might begin to show patterns that could help us to understand vision and attention in a more ‘holistic’ fashion.

To continue by way of another anecdote: a film scholar took part in an eye tracking film study at a leading European university. Upon completion, the colleague conducting the study told the scholar that they looked in completely different places – generally at the margins of the screen – to where most of the other participants looked, and that their participation was therefore useless to the study. If we can say that the film scholar looked (perhaps deliberately) where others do not look, then to what degree is film viewing a matter of, to use Turvey’s language, management and engagement? That is, do film scholars look differently at films, perhaps even at the world? And if so, what can we make of this?

The Russian ‘godfather’ of eye tracking studies, Alfred Yarbus. famously published in the 1960s that setting viewers different tasks will completely modify where they look at an image (Yarbus 1967; see also Tatler et al. 2010). There is much to extrapolate from this. For while eye tracking studies will use terms like ‘naïve’ to define how participants are unaware of the aims of the study, when it comes to film viewing, humans are rarely naïve at all. Advertising, reviews and other publicity materials are always – at least on an implicit level – telling us how and where to look at films, just as the media and our conspecifics are telling us how and where to look in the real world. Now, it may well be that humans who have never before seen a movie have little trouble understanding Hollywood cinema, as affirmed, inter alia, by both Messaris (2012, 31-33) and Smith (2012b, 74). Nonetheless, our attention is not just managed and engaged in the cinema, but it is also managed and engaged for the cinema, and I have not read any studies where psychologists showed a non-Hollywood film to first-time audiences and in which those audiences had trouble understanding the film; that is, these studies affirm nothing about the comprehension of continuity editing per se, although they might affirm that humans can understand cinema without training – as is presumably affirmed worldwide everyday as the first film shown to children is not a Hollywood film but a Bollywood, Nollywood, Filipino, Chinese or other movie; what is more, the studies perhaps only affirm the cultural hegemony enjoyed by Hollywood, in that psychologists present a Hollywood and not another film to those first-time viewers – and then use that research to affirm Hollywood’s economic primacy as being a result of its filmmaking style and not also as a result of historical and other factors. As Cynthia Freeland reminds us in her response to Smith’s AToCC, James Peterson in Post Theory argued that

a common feature of avant-garde film viewing – one that usually passes without comment: viewers initially have difficulty comprehending avant-garde films, but they learn to make sense of them. Students who take my course in the avant-garde cinema are at first completely confused by the films I show; by the end of term, they can speak intelligently about the films they see. (Peterson 1996, 110; quoted in Freeland 2012, 41)

In other words, as per my assertions re: The Room above, it is quite possible that humans would quite easily watch – and enjoy – all manner of different films, but that they do not because their attention is not ‘managed and engaged’. Again, this is a political issue, because if it is true, then it is about who can afford to use the mass media to manage and engage the attention of the most people in the quest for profit – meaning that alternative approaches to filmmaking are forced either to adopt the same system of filmmaking to compete, or they are pushed to the margins where the struggle to find audiences – because people are not prepped to watch them. The scholar at the European university has had a long education in film, and this potentially helps to manage and engage differently how they attend to them; their ‘statistically insignificant’ response might well be important in helping to demonstrate how we can not just view different/marginal films, but also view mainstream films differently.

Cutting and colleagues suggest that film editing correlates with a 1/f pattern, with 1/f (one over frequency) referring to the ‘natural’ amount of time that humans attend to objects in the real world (Cutting et al. 2010). In other words, the suggestion is that Hollywood editing rhythms reflect human attention spans – ‘evolving toward 1/f spectra… [meaning that] the mind can be “lost”… most easily in a temporal art form with that structure’ (Cutting et al. 2010, 7). Now, since David L Gilden only came up with the 1/f structure in 1995 (Gilden et al. 1995), it remains untested, and untestable without a time machine, as to whether the human attention span itself changes over time, or according to culture. That said, if cinema has always been going at about the pace that human attention was working, and if cinema cutting rates have accelerated since the 1930s and through to the present era, then attention spans may well interact with culture, and even be shaped by our media.

I often ask my students how long they should look at a painting for. It’s a trick question, because of course there is no right or wrong answer. It is my (untested) hypothesis, however, that the amount of time humans look at paintings has been shaped by the media, including films; that is, in galleries, I see people look at paintings for about the average duration of a film shot (four to five seconds) – although recently they have begun to look at a painting for about the amount of time that it takes them to take a photo of that painting with their mobile handheld device.2 Smith, citing Cutting’s work, suggests that

[i]n an average movie theatre with a 40-foot screen viewed at a distance of 35 feet, this region at the centre of our gaze will only cover about 0.19 per cent of the total screen area. Given that the average shot length of most films produced today is less than 4 seconds… viewers will only be able to make at most 20 fixations covering only 3.8 per cent of the screen area. (Smith 2013: 168)

Given that paintings vary in size, one cannot rightly say how long it would take to see a ‘whole’ painting. But if one looks at a cinema-screen sized painting for 4 seconds, then one would, after Smith, fixate on about 4 per cent of that painting. In order to see the whole painting, then more time is needed, just as more time is needed to take in our natural, rather than cinematic, environment, since we also only ever see a small proportion of that at any one time.

Relating to film the foregoing foray into painting, we might add that, given that we do not take in visual information while saccading, and given that saccades have a duration of 20-50 miliseconds (Smith 2013, 168), this means that we do not take in visual information for 0.7 seconds during every four-second shot. At 90 minutes in length, there are on average 1,350 shots per film, meaning that we do not take in visual information for 15 minutes and 45 seconds per film – blinks and turning away from the screen for snogging and toilet breaks not included. If spatially we only see 3.8 per cent of the screen during a shot, and if we only see 82.5 per cent of a film’s duration, this means that we see around 3.14 per cent of the average (Hollywood) film (no spooky π references intended).3 To be clear, these statistics apply not just to Hollywood: I would only see 3.14 per cent of Empire if I were to watch it at the cinema, too. But since it is a film comprised of a single-seeming shot and a static frame, Empire clearly encourages viewers to look for longer at the space within the frame, while Hollywood arguably does not give viewers the time to do so, since the content and duration of images is concerned uniquely with story-telling, and not with anything else. This in turn affects for how long we think that we are supposed to look at objects in our everyday lives, if for the sake of argument my gallery hypothesis be allowed to stand. Neither paintings, nor Empire, nor the world itself is organized to be seen ‘cinematically’, even if Empire is undoubtedly a work of cinema. That is, they all invite contemplation, but what they often receive is a shot-length of attention before they become boring (Empire perhaps deliberately so). Neither paintings, nor Empire, nor large swathes of the world itself controls our attention in the way cinema does; there would be much more idiosyncrasy and less synchrony of attention when looking at Empire than at a mainstream film. If the proliferation of screens featuring cinematic techniques is the making-cinematic of reality in the services of capital, then the refusal to attend to paintings, Empire and the world itself suggests not just that our attention is controlled while watching a film, but that our attention is working at a ‘cinematic’ rhythm – a rhythm that Empire uses the very apparatus of cinema in order to try to break.

The ‘temporal politics’ that I mentioned above, then, is to do with the management and engagement of attention rhythms/patterns not just in cinema, and not just for cinema (we are prepped to be movie viewers), but also by cinema for the world (people pay attention to paintings in galleries about as long as they would attend to a film shot/as long as a film shot would allow them to attend to it, before ‘cutting’, or turning away, likely getting out one’s phone, the screen of which one can also cut across with the swipe of a thumb). Politics rear their head again as homogeneity of attention span, perhaps even of life rhythm, jump into bed with the political and economic concerns that govern the structures of our society. Almost certainly in an unwitting fashion (this is not a conspiracy), validating certain cutting rates and attentions spans over others becomes an issue linked to social control, and the economic bottom line of both cinema and perhaps society as a whole. Eye tracking studies of film are part of this political ecology.

A final throw of the dice. Those of us engaged in education are of course part of a system that prepares our students for the real world. But I am personally also committed to encouraging my students sociably and communicatively to develop their individuality, to become ‘idiosyncratic’, to look at the world differently and various other notions that have long since been corporatized disingenuously as advertising slogans. Being a film teacher, I do this through encouraging my students to look differently at films. Hollywood films employ techniques that do not encourage us to look differently at movies; instead, our attention (and our brain activity) are synchronized. What is more, the idiosyncratic viewers that do look at films differently (the European film scholar) are discounted from eye tracking studies for not conforming to the norm (for not confirming to us what we already know, even if not through a scientific framework). Not only might we encourage our students to look at the world differently (to become the idiosyncratic, perhaps ‘educated’ viewer), but we might also encourage our students to make films differently, since films can also play a role in encouraging us to see the world differently, to become ‘idiosyncratic’ individuals (Hasson’s research involved the production of an interesting avant garde work, regardless of his own thoughts on the matter). Perhaps eye tracking (and fMRI) studies can help in this by turning their attention not to the majority, but to the minority, to the marginal people who look, both figuratively and literally, at the margins of the screen, and at marginal films. And this perhaps involves slowing attention down, and making it (willfully?) deeper rather than rapid and superficial. I know that the longer I look at a painting, the more the power of its creation comes to my mind, the more I marvel at it and also at the world that sustains it. In other words, it brings me joy. As I repeat often to those students who do not seem committed to participating in my classes: the more you put in, the more you get out.

Would to educate (to manage and engage attention) both in the classroom and through making and showing different sorts of (slower?) films not simply replace one trend with another, and itself be prey to political issues regarding what type of ‘idiosyncrasy’ is best? Of course, such questions are going to be of ongoing importance and would need constant attention. In relation to eye tracking film studies, though, the introduction of a ‘temporal’ dimension might help enrich our understanding of idiosyncrasy. The spatial information that idiosyncratic eye-tracks give to us is chaotic and without pattern – and thus of not much use to the psychologist; however, there may well be temporal patterns that emerge when we consider ‘idiosyncrasy’ as a shared process (to be encouraged?), rather than as a reified thing to be commoditized.

Paul Willemen has written about cinephilia as being the search for/paying attention to otherwise overlooked details in movies (Willemen 1994, 223-57). Meanwhile, Laura Mulvey has argued that DVD technology allows the film viewer to develop a deeper, cinephilic relationship with movies, since she can now pause and really analyse a film – by ‘delaying’ it/slowing it down (Mulvey 2006, 144-60). To look idiosyncratically at a movie is thus to look ‘cinephilically’; it is to look at cinema with love, perhaps to look with love tout court – but in this instance at cinema. My argument comes full circle, then, as we bring cognitive film theory, via eye tracking film studies, into contact with film theory/film-philosophy, exemplified here by Mulvey as a major figure from the Screen movement/moment. There is no I in eye tracking – but if we can accept that eye tracking studies of cinema are embroiled in a political discourse (and a political reality) concerning which films are validated as better than others and why, then perhaps by putting an ‘I’ into eye tracking, by looking at the idiosyncratic in addition to the statistically significant, then we may be able to bring about different ways of seeing and making films.

 Notes

  1. The exception is Dancer in the Dark (Lars von Trier, Spain/Argentina/ Denmark/Germany/Netherlands/Italy/USA/UK/France/Sweden/Finland/ Iceland/Norway, 2000).
  2. One of my peer reviewers took issue with the speculative nature of this suggestion. The other agreed with it.
  3. Note that I insist on the term ‘visual information’ – since film does not just engage us visually, but also aurally and via other senses (as Freeland, 2012, also reminds Smith in her response to his AToCC essay).

 

References

Anderson, Chris. 2006. The Long Tail: Why the Future of Business is Selling Less of More. New York: Hyperion.

Beller, Jonathan. 2006. The Cinematic Mode of Production: Attention Economy and the Society of the Spectacle. Lebanon, N.H.: Dartmouth College Press.

Bordwell, David. 2010. “Now you see it, now you can’t.” Observations on Film Art: Kristin Thompson and David Bordwell, June 21. http://www.davidbordwell.net/blog/?p=8509.

Bordwell, David, and Noël Carroll. 1996. Post-Theory: Reconstructing Film Studies. Madison: University of Wisconsin Press.

Cutting, James E.. 2014. Peer Reviewer’s Comments. Received October 1.

Cutting, James E., Jordan E. DeLong and Christine E. Nothelfer. 2010. “Attention and the Evolution of Hollywood Film.” Psychological Science 20:10, 1-8.

Freeland, Cynthia. 2012. “Continuity, Narrative, and Cross-Modal Cuing of Attention.” Projections: The Journal for Movies and Mind 6:1, 34-42.

Gilden, D.L., T. Thornton, and M.W. Mallon. 1995. “1/f Noise in Human Cognition.” Science 267:1837-39.

Hasson, Uri, Ohad Landesman, Barbara Knappmeyer, Ignacio Vallines, Nava Rubin and David J. Heeger. 2008. “Neurocinematics: The Neuroscience of Film.” Projections: The Journal for Movies and Mind 2:1, 1-26.

Marchant, Paul, David Raybould, Tony Renshaw and Richard Stevens. 2009. “Are you seeing what I’m seeing? An eye tracking evaluation of dynamic scenes.” Digital Creativity 20:3, 153-163.

Messaris, Paul. 2012. “Continuity and Its Discontents.” Projections: The Journal for Movies and Mind 6:1, 28-33.

Miller, Greg. 2014a. “How Movies Manipulate Your Brain to Keep You Entertained.Wired, August 26.

Miller, Greg. 2014b. “How Movies Synchronize the Brains of an Audience.” Wired, August 28.

Mulvey, Laura. 2006. Death 24x a Second: Stillness and the Moving Image. London: Reaktion.

Peterson, James. 1996. “Is a Cognitive Approach to the Avant-Garde Cinema Perverse?” In Post Theory: Reconstructing Film Studies, edited by David Bordwell and Noël Carroll, 108-129. Madison: University of Wisconsin Press.

Rogers, Sheena. 2012. “Auteur of Attention: The Filmmaker as a Cognitive Scientist.” Projections: The Journal for Movies and Mind 6:1, 42-49.

Saez de Urabaín, Irati R., Mark H. Johnson and Tim J. Smith. 2014. “GraFIX: A semiautomatic approach for parsing low- and high-quality eye tracking data.” Behavior Research Methods, March 27, pp. 1-20.

Sconce, Jeffrey. 1995. “‘Trashing’ the academy: taste, excess, and an emerging politics of cinematic style.” Screen 36:4, 371-393.

Smith, Greg M. 2012. “Continuity Is Not Continuous.” Projections: The Journal for Movies and Mind 6:1, 56-61.

Smith, Tim J. 2012a. “The Attentional Theory of Continuity Editing.” Projections: The Journal for Movies and Mind 6:1, 1-27.

Smith, Tim J. 2012b. “Extending AToCC: A Reply.” Projections: The Journal for Movies and Mind 6:1, 71-78.

Smith, Tim J. 2013. “Watching You Watch Movies: Using Eye Tracking to Inform Cognitive Film Theory.” In Psychocinematics: Exploring Cognition at the Movies, edited by Art P. Shimamura, 165-191. New York: Oxford University Press.

Smith, Tim J. 2014. “Audiovisual Correspondences in Sergei Eisenstein’s Alexander Nevsky: A Case Study in Viewer Attention.” In Cognitive Media Theory, edited by Ted Nannicelli and Paul Taberham, 85-105. London: Routledge/American Film Institute.

Smith, Tim J, and John M. Henderson. 2008. “Edit Blindness: The relationship between attention and global change blindness in dynamic scenes.” Journal of Eye Movement Research 2(2):6, 1-17.

Tatler, Benjamin W., Nicholas J. Wade, Hoi Kwan, John M. Findlay and Boris M. Velichkovsky. 2010. “Yarbus, eye movements, and vision.” i-Perception 1:7-27.

Treuting, Jennifer. 2006. “Eye Tracking and the Cinema: A Study of Film Theory and Visual Perception.” SMPTE Motion Imaging Journal 115:1, 31-40.

Turvey, Malcolm. 2012. “The Continuity of Narrative Comprehension.” Projections: The Journal for Movies and Mind 6:1, 49-56.

Võ, Melissa L.-H., Tim J. Smith, Parag K. Mital and John M. Henderson. 2012. “Do the eyes really have it? Dynamic allocation og attention when viewing moving faces.” Journal of Vision 12(13):3, 1-14.

Wass, Sam V., Tim J. Smith and Mark H. Johnson. 2013. “Parsing eye tracking data of variable quality to provide accurate fixation duration estimates in infants and adults.” Behavior Research Methods 45:1, 229-250.

Willemen, Paul. 1994. Looks and Frictions: Essays in Cultural Studies and Film Theory. Bloomington: Indiana University Press.

Yarbus, Alfred L. 1967. Eye Movements and Vision. Translated by Basil Haigh. New York: Plenum Press.

 

William Brown is a Senior Lecturer in Film at the University of Roehampton, London. He is the author of Supercinema: Film-Philosophy for the Digital Age (Berghahn, 2013) and, with Dina Iordanova and Leshu Torchin, of Moving People, Moving Images: Cinema and Trafficking in the New Europe (St Andrews Film Studies, 2010). He is the co-editor, with David Martin-Jones, of Deleuze and Film (Edinburgh University Press, 2012). He is also a filmmaker.

Read, Watch, Listen: A commentary on eye tracking and moving images – Tim J. Smith

$
0
0

Abstract

Eye tracking is a research tool that has great potential for advancing our understanding of how we watch movies. Questions such as how differences in the movie influences where we look and how individual differences between viewers alters what we see can be operationalised and empirically tested using a variety of eye tracking measures. This special issue collects together an inspiring interdisciplinary range of opinions on what eye tracking can (and cannot) bring to film and television studies and practice. In this article I will reflect on each of these contributions with specific focus on three aspects: how subtitling and digital effects can reinvigorate visual attention, how audio can guide and alter our visual experience of film, and how methodological, theoretical and statistical considerations are paramount when trying to derive conclusions from eye tracking data.

 

Introduction

I have been obsessed with how people watch movies since I was a child. All you have to do is turn and look at an audience member’s face at the movies or at home in front of the TV to see the power the medium holds over them. We sit enraptured, transfixed and immersed in the sensory patterns of light and sound projected back at us from the screen. As our physical activity diminishes our mental activity takes over. We piece together minimal audiovisual cues to perceive rich otherworldly spaces, believable characters and complex narratives that engage us mentally and move us emotionally. As I progressed through my education in Cognitive Science and Psychology I was struck by how little science understood about cinema and the mechanisms filmmakers used to create this powerful experience.[i] Reading the film literature, listening to filmmakers discuss their craft and excavating gems of their craft knowledge I started to realise that film was a medium ripe for psychological investigation. The empirical study of film would further our understanding of how films work and how we experience them but it would also serve as a test bed for investigating complex aspects of real-world cognition that were often considered beyond the realms of experimentation. As I (Smith, Levin & Cutting, 2010) and others (Anderson, 2006) have argued elsewhere, film evolved to “piggy back” normal cognitive development and use basic cognitive tendencies such as attentional preferences, theory of mind, empathy and narrative structuring of memory to make the perception of film as enjoyable and effortless as possible. By investigating film cognition we can, in turn advance our understanding of general cognition. But to do so we need to step outside of traditional disciplinary boundaries concerning the study of film and approach the topic from an interdisciplinary perspective. This special issue represents a highly commendable attempt to do just that.

By bringing together psychologists, film theorists, philosophers, vision scientists, neuroscientists and screenwriters this special issue (and the Melbourne research group that most contributors belong to) provides a unique perspective on film viewing. The authors included in this special issue share my passion for understanding the relationship between viewers and film but this interest manifests in very different ways depending on their perspectives (see Redmond, Sita, and Vincs, this issue; for a similar personal journey into eye tracking as that presented above). By focussing on viewer eye movements the articles in this special issue provide readers from a range of disciplines a way into the eye tracking investigation of film viewing. Eye tracking (as comprehensively introduced and discussed by Dyer and Pink, this issue) is a powerful tool for quantifying a viewer’s experience of a film, comparing viewing behaviour across different viewing conditions and groups as well as testing hypotheses about how certain cinematic techniques impact where we look. But, as is rightly highlighted by several of the authors in this special issue eye tracking is not a panacea for all questions about film spectatorship.

Like all experimental techniques it can only measure a limited range of psychological states and behaviours and the data it produces does not say anything in and of itself. Data requires interpretation. Interpretation can take many forms[ii] but if conclusions are to be drawn about how the data relates to psychological states of the viewer this interpretation must be based on theories of psychology and ideally confirmed using secondary/supporting measures. For example, the affective experience of a movie is a critical aspect which cognitive approaches to film are often wrongly accused of ignoring. Although, cognitive approaches to film often focus on how we comprehend narratives (Magliano and Zacks, 2011), attend to the image (Smith, 2013) or follow formal patterns within a film (Cutting, DeLong and Nothelfer, 2010) several cognitivists have focussed in depth on emotional aspects (see the work of Carl Plantinga, Torben Grodal or Murray Smith). Eye tracking is the perfect tool for investigating the impact of immediate audiovisual information on visual attention but it is less suitable for measuring viewer affect. Psychophysiological measures such as heart rate and skin conductance, neuroimaging methods such as fMRI or EEG, or even self-report ratings may be better for capturing a viewer’s emotional responses to a film as has been demonstrated by several research teams (Suckfull, 2000; Raz et al, 2014). Unless the emotional state of the viewer changed where they looked or how quickly they moved their eyes the eye tracker may not detect any differences between two viewers with different emotional states.[iii]

As such, a researcher interested in studying the emotional impact of a film should either choose a different measurement technique or combine eye tracking with another more suitable technique (Dyer and Pink, this issue). This does not mean that eye tracking is unsuitable for studying the cinematic experience. It simply means that you should always choose the right tool for the job and often this means combining multiple tools that are strong in different ways. As Murray Smith (the current President of the Society for the Cognitive Study of the Moving Images; SCSMI) has argued, a fully rounded investigation of the cinematic experience requires “triangulation” through the combination of multiple perspectives including psychological, neuroscientific and phenomenological/philosophical theory and methods (Smith, 2011) – an approach taken proudly across this special issue.

For the remainder of my commentary I would like to focus on certain themes that struck me as most personally relevant and interesting when reading the other articles in this special issue. This is by no means an exhaustive list of the themes raised by the other articles or even an assessment of the importance of the particular themes I chose to select. There are many other interesting observations made in the articles I do not focus on below but given my perspective as a cognitive scientist and current interests I decided to focus my commentary on these specific themes rather than make a comprehensive review of the special issues or tackle topics I am unqualified to comment on. Also, I wanted to take the opportunity to dispel some common misconceptions about eye tracking (see the section ‘Listening to the data’) and empirical methods in general.

Reading an image

One area of film cognition that has received considerable empirical investigation is subtitling. As Kruger, Szarkowska and Krejtz (this issue) so comprehensively review, they and I believe eye tracking is the perfect tool for investigating how we watch subtitled films. The presentation of subtitles divides the film viewing experience into a dual- task: reading and watching. Given that the media was originally designed to communicate critical information through two channels, the image and soundtrack introducing text as a third channel of communication places extra demands on the viewer’s visual system. However, for most competent readers serially shifting attention between these two tasks does not lead to difficulties in comprehension (Kruger, Szarkowska and Krejtz, this issue). Immediately following the presentation of the subtitles gaze will shift to the beginning of the text, saccade across the text and return to the centre of interest within a couple of seconds. Gaze heatmaps comparing the same scenes with and without subtitles (Kruger, Szarkowska and Krejtz, this issue; Fig. 3) show that the areas of the image fixated are very similar (ignoring the area of the screen occupied by the subtitles themselves) and rather than distracting from the visual content the presence of subtitles seems to actually condense the gaze behaviour on the areas of central interest in an image, e.g. faces and the centre of the image. This illustrates the redundancy of a lot of the visual information presented in films and the fact that under non-subtitle conditions viewers rarely explore the periphery of the image (Smith, 2013).

My colleague Anna Vilaró and I recently demonstrated this similarity in an eye tracking study in which the gaze behaviour of viewers was compared across versions of an animated film, Disney’s Bolt (Howard & Williams, 2008) either in the original English audio condition, a Spanish language version with English subtitles, an English language version with Spanish subtitles and a Spanish language version without subtitles (Vilaró, & Smith, 2011). Given that our participants were English speakers who did not know Spanish these conditions allowed us to investigate both where they looked under the different audio and subtitle conditions but also what they comprehended. Using cued recall tests of memory for verbal and visual content we found no significant differences in recall for either types of content across the viewing conditions except for verbal recall in the Spanish-only condition (not surprisingly given that our English participants couldn’t understand the Spanish dialogue). Analysis of the gaze behaviour showed clear evidence of subtitle reading, even in the Spanish subtitle condition (see Figure 1) but no differences in the degree to which peripheral objects were explored. This indicates that even when participants are watching film sequences without subtitles and know that their memory will be tested for the visual content their gaze still remains focussed on central features of a traditionally composed film. This supports arguments for subtitling movies over dubbing as, whilst placing greater demands on viewer gaze and a heightened cognitive load there is no evidence that subtitling leads to poorer comprehension.

Figure 1: Figure from Vilaró & Smith (2011) showing the gaze behaviour of multiple viewers directed to own language subtitles (A) and foreign language/uninterpretable subtitles (B).

Figure 1: Figure from Vilaró & Smith (2011) showing the gaze behaviour of multiple viewers directed to own language subtitles (A) and foreign language/uninterpretable subtitles (B).

The high degree of attentional synchrony (Smith and Mital, 2013) observed in the above experiment and during most film sequences indicates that all visual features in the image and areas of semantic significance (e.g. social information and objects relevant to the narrative) tend to point to the same part of the image (Mital, Smith, Hill and Henderson, 2011). Only when areas of the image are placed in conflict through image composition (e.g. depth of field, lighting, colour or motion contrast) or staging (e.g. multiple actors) does attentional synchrony break down and viewer gaze divide between multiple locations. Such shots are relatively rare in mainstream Hollywood cinema or TV (Salt, 2009; Smith, 2013) and when used the depicted action tends to be highly choreographed so attention shifts between the multiple centres of image in a predictable fashion (Smith, 2012). If such choreographing of action is not used the viewer can quickly exhaust the information in the image and start craving either new action or a cut to a new shot.

Hochberg and Brooks (1978) referred to this as the visual momentum of the image: the pace at which visual information is acquired. This momentum is directly observable in the saccadic behaviour during an images presentation with frequent short duration fixations at the beginning of a scene’s presentation interspersed by large amplitude saccades (known as the ambient phase of viewing; Velichovsky, Dornhoefer, Pannasch and Unema, 2000) and less frequent, longer duration fixations separated by smaller amplitude saccades as the presentation duration increases (known as the focal phase of viewing; Velichovsky et al., 2000). I have recently demonstrated the same pattern of fixations during viewing of dynamic scenes (Smith and Mital, 2013) and shown how this pattern gives rise to more central fixations at shot onset and greater exploration of the image and decreased attentional synchrony as the shot duration increases (Mital, Smith, Hill and Henderson, 2011). Interestingly, the introduction of subtitles to a movie may have the unintended consequence of sustaining visual momentum throughout a shot. The viewer is less likely to exhaust the information in the image because their eyes are busy saccading across the text to acquire the information that would otherwise be presented in parallel to the image via the soundtrack. This increased saccadic activity may increase the cognitive load experienced by viewers of subtitled films and change their affective experience, producing greater arousal and an increased sense of pace.

For some filmmakers and producers of dynamic visual media, increasing the visual momentum of an image sequence may be desirable as it maintains interest and attention on the screen (e.g. Michael Bay’s use of rapidly edited extreme Close-Ups and intense camera movements in the Transformer movies). In this modern age of multiple screens fighting for our attention when we are consuming moving images (e.g. mobile phones and computer screens in our living rooms and even, sadly increasingly at the cinema) if the designers of this media are to ensure that our visual attention is focussed on their screen over the other competing screens they need to design the visual display in a way that makes comprehension impossible without visual attention. Feature Films and Television dramas often rely heavily on dialogue for narrative communication and the information communicated through the image may be of secondary narrative importance to the dialogue so viewers can generally follow the story just by listening to the film rather than watching it. If producers of dynamic visual media are to draw visual attention back to the screen and away from secondary devices they need to increase the ratio of visual to verbal information. A simple way of accomplishing this is to present the critical audio information through subtitling. The more visually attentive mode of viewing afforded by watching subtitled film and TV may partly explain the growing interest in foreign TV series (at least in the UK) such as the popularity of Nordic Noir series such as The Bridge (2011) and The Killing (2007).

Another way of drawing attention back to the screen is to constantly “refresh” the visual content of the image by either increasing the editing rate or creatively using digital composition.[iv] The latter technique is wonderfully exploited by Sherlock (2010) as discussed brilliantly by Dwyer (this issue). Sherlock contemporised the detective techniques of Sherlock Holmes and John Watson by incorporating modern technologies such as the Internet and mobile phones and simultaneously updated the visual narrative techniques used to portray this information by using digital composition to playfully superimpose this information onto the photographic image. In a similar way to how the sudden appearance of traditional subtitles involuntarily captures visual attention and draws our eyes down to the start of the text, the digital inserts used in Sherlock overtly capture our eyes and encourage reading within the viewing of the image.

If Dwyer (this issue) had eyetracked viewers watching these excerpts she would have likely observed this interesting shifting between phases of reading and dynamic scene perception. Given that the appearance of the digital inserts produce sudden visual transients and are highly incongruous with the visual features of the background scene they are likely to involuntarily attract attention (Mital, Smith, Hill & Henderson, 2012). As such, they can be creatively used to reinvigorate the pace of viewing and strategically direct visual attention to parts of the image away from the screen centre. Traditionally, the same content may have been presented either verbally as narration, heavy handed dialogue exposition (e.g. “Oh my! I have just received a text message stating….”) or as a slow and laboured cut to close-up of the actual mobile phone so we can read it from the perspective of the character. Neither takes full advantage of the communicative potential of the whole screen space or our ability to rapidly attend to and comprehend visual information and audio information in parallel.

Such intermixing of text, digital inserts and filmed footage is common in advertisements, music videos, and documentaries (see Figure 2) but is still surprisingly rare in mainstream Western film and TV. Short-form audiovisual messages have recently experienced a massive increase in popularity due to the internet and direct streaming to smartphones and mobile devices. To maximise their communicative potential and increase their likelihood of being “shared” these videos use all audiovisual tricks available to them. Text, animations, digital effects, audio and classic filmed footage all mix together on the screen, packing every frame with as much info as possible (Figure 2), essentially maximising the visual momentum of each video and maintaining interest for as long as possible.[v] Such videos are so effective at grabbing attention and delivering satisfying/entertaining/informative experiences in a short period of time that they often compete directly with TV and film for our attention. Once we click play, the audiovisual bombardment ensures that our attention remains latched on to the second screen (i.e., the tablet or smartphone) for its duration and away from the primary screen, i.e., the TV set. Whilst distressing for producers of TV and Film who wish our experience of their material to be undistracted, the ease with which we pick up a handheld device and seek other stimulation in parallel to the primary experience may indicate that the primary material does not require our full attention for us to follow what is going on. As attention has a natural ebb-and-flow (Cutting, DeLong and Nothelfer, 2010) and “There is no such thing as voluntary attention sustained for more than a few seconds at a time” (p. 421; James, 1890) if modern producers of Film and TV want to maintain a high level of audience attention and ensure it is directed to the screen they must either rely on viewer self-discipline to inhibit distraction, reward attention to the screen with rich and nuanced visual information (as fans of “slow cinema” would argue of films like those of Bela Tarr) or utilise the full range of postproduction effects to keep visual interest high and maintained on the image, as Sherlock so masterfully demonstrates.

Figure 2: Gaze Heatmaps of participants’ free-viewing a trailer for Lego Indiana Jones computer game (left column) and the Video Republic documentary (right column). Notice how both make copious use of text within the image, as intertitles and as extra sources of information in the image (such as the head-up display in A3). Data and images were taken from the Dynamic Images and Eye Movement project (DIEM; Mital, Smith, Hill & Henderson, 2010). Videos can be found here (http://vimeo.com/6628451) and here (http://vimeo.com/2883321).

Figure 2: Gaze Heatmaps of participants’ free-viewing a trailer for Lego Indiana Jones computer game (left column) and the Video Republic documentary (right column). Notice how both make copious use of text within the image, as intertitles and as extra sources of information in the image (such as the head-up display in A3). Data and images were taken from the Dynamic Images and Eye Movement project (DIEM; Mital, Smith, Hill & Henderson, 2010). Videos can be found here (http://vimeo.com/6628451) and here (http://vimeo.com/2883321).

A number of modern filmmakers are beginning to experiment with the language of visual storytelling by questioning our assumptions of how we perceive moving images. Forefront in this movement are Ang Lee and Andy and Lana Wachowski. In Ang Lee’s Hulk (2003), Lee worked very closely with editor Tim Squyers to use non-linear digital editing and after effects to break apart the traditional frame and shot boundaries and create an approximation of a comic book style within film. This chaotic unpredictable style polarised viewers and was partly blamed for the film’s poor reception. However, it cannot be argued that this experiment was wholly unsuccessful. Several sequences within the film used multiple frames, split screens, and digital transformation of images to increase the amount of centres of interest on the screen and, as a consequence increase pace of viewing and the arousal experienced by viewers. In the sequence depicted below (Figure 3) two parallel scenes depicting Hulk’s escape from a containment chamber (A1) and this action being watched from a control room by General Ross (B1) were presented simultaneously by presenting elements of both scenes on the screen at the same time. Instead of using a point of view (POV) shot to show Ross looking off screen (known as the glance shot; Branigan, 1984) followed by a cut to what he was looking at (the object shot) both shots were combined into one image (F1 and F2) with the latter shot sliding into from behind Ross’ head (E2). These digital inserts float within the frame, often gliding behind objects or suddenly enlarging to fill the screen (A2-B2). Such visual activity and use of shots-within-shots makes viewer gaze highly active (notice how the gaze heatmap is rarely clustered in one place; Figure 3). Note that this method of embedding a POV object shot within a glance shot is similar to Sherlock’s method of displaying text messages as both the glance, i.e., Watson looking at his phone, and the object, i.e., the message, are shown in one image. Both uses take full advantage of our ability to rapidly switch from watching action to reading text without having to wait for a cut to give us the information.

Figure 3: Gaze heatmap of eight participants watching a series of shots and digital inserts from Hulk (Ang Lee, 2003). Full heatmap video is available at http://youtu.be/tErdurgN8Yg.

Figure 3: Gaze heatmap of eight participants watching a series of shots and digital inserts from Hulk (Ang Lee, 2003). Full heatmap video is available at http://youtu.be/tErdurgN8Yg.

Similar techniques have been used Andy and Lana Wachowski’s films including most audaciously in Speed Racer (2008). Interestingly, both sets of filmmakers seem to intuitively understand that packing an image with as much visual and textual information as possible can lead to viewer fatigue and so they limit such intense periods to only a few minutes and separate them with more traditionally composed sequences (typically shot/reverse-shot dialogue sequences). These filmmakers have also demonstrated similar respect for viewer attention and the difficulty in actively locating and encoding visual information in a complex visual composition in their more recent 3D movies. Ang Lee’s Life of Pi (2012) uses the visual volume created by stereoscopic presentation to its full potential. Characters inhabit layers within the volume as foreground and background objects fluidly slide around each other within this space. The lessons Lee and his editor Tim Squyers learned on Hulk (2003) clearly informed the decisions they made when tackling their first 3D film and allowed them to avoid some of the issues most 3D films experience such as eye strain, sudden unexpected shifts in depth and an inability to ensure viewers are attending to the part of the image easiest to fuse across the two eye images (Banks, Read, Allison & Watt, 2012).

Watching Audio

I now turn to another topic featured in this special issue, the influence of audio on gaze (Robinson, Stadler and Rassell, this issue). Film and TV are inherently multimodal. Both media have always existed as a combination of visual and audio information. Even early silent film was almost always presented with either live musical accompaniment or a narrator. As such, the relative lack of empirical investigation into how the combination of audio and visual input influences how we perceive movies and, specifically how we attend to them is surprising. Robinson, Stadler and Rassell (this issue) have attempted to address this omission by comparing eye movements for participants either watching the original version of the Omaha beach sequence from Steven Spielberg’s Saving Private Ryan (1998) or the same sequence with the sound removed. This film sequence is a great choice for investigating AV influences on viewer experience as the intensity of the action, the hand-held cinematography and the immersive soundscape all work together to create a disorientating embodied experience for the viewer. The authors could have approached this question by simply showing a set of participants the sequence with audio and qualitatively describing the gaze behaviour at interesting AV moments during the sequence. Such description of the data would have served as inspiration for further investigation but in itself can’t say anything about the causal contribution of audio to this behaviour as there would be nothing to compare the behaviour to. Thankfully, the authors avoided this problem by choosing to manipulate the audio.

In order to identify the causal contribution of any factor you need to design an experiment in which that factor (known as the Independent Variable) is either removed or manipulated and the significant impact of this manipulation on the behaviour of interest (known as the Dependent Variable) is tested using appropriate inferential statistics. I commend Robinson, Stadler and Rassell’s experimental design as they present such an manipulation and are therefore able to produce data that will allow them to test their hypotheses about the causal impact of audio on viewer gaze behaviour. Several other papers in this special issue (Redmond, Sita and Vincs; Batty, Perkins and Sita) discuss gaze data (typically in the form of scanpaths or heatmaps) from one viewing condition without quantifying its difference to another viewing condition. As such, they are only able to describe the gaze data, not use it to test hypotheses. There is always a temptation to attribute too much meaning to a gaze heatmap (I too am guilty of this; Smith, 2013) due to their seeming intuitive nature (i.e., they looked here and not there) but, as in all psychological measures they are only as good as the experimental design within which there are employed.[vi]

Qualitative interpretation of individual fixation locations, scanpaths or group heatmaps are useful for informing initial interpretation of which visual details are most likely to make it into later visual processing (e.g. perception, encoding and long term memory representations) but care has to be taken in falsely assuming that fixation equals awareness (Smith, Lamont and Henderson, 2012). Also, the visual form of gaze heatmaps vary widely depending on how many participants contribute to the heatmap, which parameters you choose to generate the heatmaps and which oculomotor measures the heatmap represent (Holmqvist, et al., 2011). For example, I have demonstrated that unlike during reading visual encoding during scene perception requires over 150ms during each fixation (Rayner, Smith, Malcolm and Henderson, 2009). This means that if fixations with durations less than 150ms are included in a heatmap it may suggest parts of the image have been processed which in actual fact were fixated too briefly to be processed adequately. Similarly, heatmaps representing fixation duration instead of just fixation location have been shown to be a better representation of visual processing (Henderson, 2003). Heatmaps have an immediate allure but care has to be taken about imposing too much meaning on them especially when the gaze and the image are changing over time (see Smith and Mital, 2013; and Sawahata et al, 2008 for further discussion). As eye tracking hardware becomes more available to researchers from across a range of disciplines we need to work harder to ensure that it is not used inappropriately and that the conclusions that are drawn from eye tracking data are theoretically and statistically motivated (see Rayner, 1998; and Holmqvist et al, 2013 for clear guidance on how to conduct sound eye tracking studies).

Given that Robinson, Stadler and Rassell (this issue) manipulated the critical factor, i.e., the presence of audio the question now is whether their study tells us anything new about the AV influences on gaze during film viewing. To examine the influence of audio they chose two traditional methods for expressing the gaze data: area of interest (AOI) analysis and dispersal. By using nine static (relative to the screen) AOIs they were able to quantify how much time the gaze spent in each AOI and utilise this measure to work out how distributed gaze was across all AOIs. Using these measures they reported a trend towards greater dispersal in the mute condition compared to the audio condition and a small number of significant differences in the amount of time spent in some regions across the audio conditions.

However, the conclusions we can draw from these findings are seriously hindered by the low sample size (only four participants were tested, meaning that any statistical test is unlikely to reveal significant differences) and the static AOIs that did not move with the image content. By locking the AOIs to static screen coordinates their AOI measures express the deviation of gaze relative to these coordinates, not to the image content. This approach can be informative for quantifying gaze exploration away from the screen centre (Mital, Smith, Hill and Henderson, 2011) but in order to draw conclusions about what was being fixated the gaze needs to be quantified relative to dynamic AOIs that track objects of interest on the screen (see Smith an Mital, 2013). For example, their question about whether we fixate a speaker’s mouth more in scenes where the clarity of the speech is difficult due to background noise (i.e., their “Indistinct Dialogue” scene) has previously been investigated in studies that have manipulated the presence of audio (Võ, Smith, Mital and Henderson, 2012) or the level of background noise (Buchan, Paré and Munhall, 2007) and measured gaze to dynamic mouth regions. As Robinson, Stadler and Rassell correctly predicted, lip reading increases as speech becomes less distinct or the listener’s linguistic competence in the spoken language decreases (see Võ et al, 2012 for review).

Similarly, by measuring gaze dispersal using a limited number of static AOIs they are losing considerable nuance in the gaze data and have to resort to qualitative description of unintuitive bar charts (figure 4). There exist several methods for quantifying gaze dispersal (see Smith and Mital, 2013, for review) and even open-source tools for calculating this measure and comparing dispersal across groups (Le Meur and Baccino, 2013). Some methods are as easy, if not easier to calculate than the static AOIs used in the present study. For example, the Euclidean distance between the screen centre and the x/y gaze coordinates at each frame of the movie provides a rough measure of how spread out the gaze is from the screen centre (typically the default viewing location; Mital et al, 2011) and a similar calculation can be performed between the gaze position of all participants within a viewing condition to get a measure of group dispersal.

Using such measures, Coutrot and colleagues (2012) showed that gaze dispersal is greater when you remove audio from dialogue film sequences and they have also observed shorter amplitude saccades and marginally shorter fixation durations. Although, I have recently shown that a non-dialogue sequence from Sergei Eisenstein’s Alexander Nevsky (1938) does not show significant differences in eye movement metrics when the accompanying music is removed (Smith, 2014). This difference in findings points towards interesting differences in the impact diegetic (within the depicted scene, e.g. dialogue) and non-diegetic (outside of the depicted scene, e.g. the musical score) may have on gaze guidance. It also highlights how some cinematic features may have a greater impact on other aspects of a viewer’s experience than those measureable by eye tracking such as physiological markers of arousal and emotional states. This is also the conclusion that Robinson, Stadler and Rassell come to.    

Listening to the Data (aka, What is Eye Tracking Good For?)

The methodological concerns I have raised in the previous section lead nicely to the article by William Brown, entitled There’s no I in Eye Tracking: How useful is Eye Tracking to Film Studies (this issue). I have known William Brown for several years through our attendance of the Society for Cognitive Studies of the Moving Image (SCSMI) annual conference and I have a deep respect for his philosophical approach to film and his ability to incorporate empirical findings from the cognitive neurosciences, including some references to my own work into his theories. Therefore, it comes somewhat as a surprise that his article openly attacks the application of eye tracking to film studies. However, I welcome Brown’s criticisms as it provides me with an opportunity to address some general assumptions about the scientific investigation of film and hopefully suggest future directions in which eye tracking research can avoid falling into some of the pitfalls Brown identifies.

Brown’s main criticisms of current eye tracking research are: 1) eye tracking studies neglect “marginal” viewers or marginal ways of watching movies; 2) studies so far have neglected “marginal” films; 3) they only provide “truisms”, i.e., already known facts; and 4) they have an implicit political agenda to argue that the only “true” way to study film is a scientific approach and the “best” way to make a film is to ensure homogeneity of viewer experience. I will address these criticisms in turn but before I do so I would like to state that a lot of Brown’s arguments could generally be recast as an argument against science in general and are built upon a misunderstanding of how scientific studies should be conducted and what they mean.

To respond to Brown’s first criticism that eye tracking “has up until now been limited somewhat by its emphasis on statistical significance – or, put simply, by its emphasis on telling us what most viewers look at when they watch films” (Brown, this issue; 1), I first have to subdivide the criticism into ‘the search for significance’ and ‘attentional synchrony’, i.e., how similar gaze is across viewers (Smith and Mital, 2013). Brown tells an anecdote about a Dutch film scholar who’s data had to be excluded from an eye tracking study because they did not look where the experimenter wanted them to look. I wholeheartedly agree with Brown that this sounds like a bad study as data should never be excluded for subjective reasons such as not supporting the hypothesis, i.e., looking as predicted. However, exclusion due to statistical reasons is valid if the research question being tested relates to how representative the behaviour of a small set of participants (known as the sample) are to the overall population. To explain when such a decision is valid and to respond to Brown’s criticism about only ‘searching for significance’ I will first need to provide a brief overview of how empirical eye tracking studies are designed and why significance testing is important.

For example, if we were interested in the impact sound had on the probability of fixating an actor’s mouth (e.g., Robinson, Stadler and Rassell, this issue) we would need to compare the gaze behaviour of a sample of participants who watch a sequence with the sound turned on to a sample who watched it with the sound turned off. By comparing the behaviour between these two groups using inferential statistics we are testing the likelihood that these two viewing conditions would differ in a population of all viewers given the variation within and between these two groups. In actual fact we do this by performing the opposite test: testing the probability that that the two groups belong to a single statistically indistinguishable group. This is known as the null hypothesis. By showing that there is less than a 5% chance that the null hypothesis is true we can conclude that there is a statistically significant chance that another sample of participants presented with the same two viewing conditions would show similar differences in viewing behaviour.

In order to test whether our two viewing conditions belong to one or two distributions we need to be able to express this distribution. This is typically done by identifying the mean score for each participant on the dependent variable of interest, in this case the probability of fixating a dynamic mouth AOI then calculating the mean for this measure across all participants within a group and their variation in scores (known as the standard deviation). Most natural measures produce a distribution of scores looking somewhat like a bell curve (known as the normal distribution) with most observations near the centre of the distribution and an ever decreasing number of observations as you move away from this central score. Each observation (in our case, participants) can be expressed relative to this distribution by subtracting the mean of the distribution from its score and dividing by the standard deviation. This converts a raw score into a normalized or z-score. Roughly ninety-five percent of all observations will fall within two standard deviations of the mean for normally distributed data. This means that observations with a z-score greater than two are highly unrepresentative of that distribution and may be considered outliers.

However, being unrepresentative of the group mean is insufficient motivation to exclude a participant. The outlier still belongs to the group distribution and should be included unless there is a supporting reason for exclusion such as measurement error, e.g. poor calibration of the eye tracker. If an extreme outlier is not excluded it can often have a disproportionate impact on the group mean and make statistical comparison of groups difficult. However, if this is the case it suggests that the sample size is too small and not representative of the overall population. Correct choice of sample size given an estimate of the predicted effect size combined with minimising measurement error should mean that subjective decisions do not have to be made about who’s data is “right” and who should be included or excluded.

Brown also believes that eye tracking research has so far marginalised viewers who have atypical ways of watching film, such as film scholars either by not studying them or treating them as statistical outliers and excluding them from analyses. However, I would argue that the only way to know if their way of watching a film is atypical is to first map out the distribution of how viewers typically watch films. If a viewer attended more to the screen edge than the majority of other viewers in a random sample of the population (as was the case with Brown’s film scholar colleague) this should show up as a large z-score when their gaze data is expressed relative to the group on a suitable measure such as Euclidean distance from the screen centre. Similarly, a non-native speaker of English may have appeared as an outlier in terms of how much time they spent looking at the speaker’s mouth in Robinson, Stadler and Rassell’s (this issue) study. Such idiosyncrasies may be of interest to researchers and there are statistical methods for expressing emergent groupings within the data (e.g. cluster analysis) or seeing whether group membership predicts behaviour (e.g. regression). These approaches may have not previously been applied to questions of film viewing but this is simply due to the immaturity of the field and the limited availability of the equipment or expertise to conduct such studies.

In my own recent work I have shown how viewing task influences how we watch unedited video clips (Smith and Mital, 2013), how infants watch TV (Wass and Smith, in press), how infant gaze differs to adult gaze (Smith, Dekker, Mital, Saez De Urabain and Karmiloff-Smith, in prep) and even how film scholars attend to and remember a short film compared to non-expert film viewers (Smith and Smith, in prep). Such group viewing differences are of great interest to me and I hope these studies illustrate how eye tracking has a lot to offer to such research questions if the right statistics and experimental designs are employed.

Brown’s second main criticism is that the field of eye tracking neglects “marginal” films. I agree that the majority of films that have so far been used in eye tracking studies could be considered mainstream. For example, the film/TV clips used in this special issue include Sherlock (2010), Up (2009) and Saving Private Ryan (1998). However, this limit is simply a sign of how few eye tracking studies of moving images there have been. All research areas take time to fully explore the range of possible research questions within that area.

I have always employed a range of films from diverse film traditions, cultures, and languages. My first published eye tracking study (Smith and Henderson, 2008) used film clips from Citizen Kane (1941), Dogville (2003), October (1928), Requiem for a Dream (2000), Dancer in the Dark (2000), Koyaanisqatsi (1982) and Blade Runner (1982). Several of these films may be considered “marginal” relative to the mainstream. If I have chosen to focus most of my analyses on mainstream Hollywood cinema this is only because they were the most suitable exemplars of the phenomena I was investigating such as continuity editing and its creation of a universal pattern of viewing (Smith, 2006; 2012). This interest is not because, as Brown argues, I have a hidden political agenda or an implicit belief that this style of filmmaking is the “right” way to make films. I am interested in this style because it is the dominant style and, as a cognitive scientist I wish to use film as a way of understanding how most people process audiovisual dynamic scenes.

Hollywood film stands as a wonderfully rich example of what filmmakers think “fits” human cognition. By testing filmmaker intuitions and seeing what impact particular compositional decisions have on viewer eye movements and behavioural responses I hope to gain greater insight into how audiovisual perception operates in non-mediated situations (Smith, Levin and Cutting, 2012). But, just as a neuropsychologist can learn about typical brain function by studying patients with pathologies such as lesions and strokes, I can also learn about how we perceive a “typical” film by studying how we watch experimental or innovative films. My previous work is testament to this interest (Smith, 2006; 2012a; 2012b; 2014; Smith & Henderson, 2008) and I hope to continue finding intriguing films to study and further my understanding of film cognition.

One practical reason why eye tracking studies rarely use foreign language films is the presence of subtitles. As has been comprehensively demonstrated by other authors in this special issue (Kruger, Szarkowska and Krejtz, this issue) and earlier in this article, the sudden appearance of text on the screen, even if it is incomprehensible leads to differences in eye movement behaviour. This invalidates the use of eye tracking as a way to measure how the filmmaker intended to shape viewer attention and perception. The alternatives would be to either use silent film (an approach I employed with October; Smith and Henderson, 2008), remove the audio (which changes gaze behaviour and awareness of editing; Smith & Martin-Portugues Santacreau, under review) or use dubbing (which can bias the gaze down to the poorly synched lips; Smith, Batten, and Bedford, 2014). None of these options are ideal for investigating foreign language sound film and until there is a suitable methodological solution this will restrict eye tracking studies to experimental films in a participant’s native language.

Finally, I would like to counter Brown’s assertion that eye tracking investigations of film have so far only generated “truisms”. I admit that there is often a temptation to reduce empirical findings to simplified take-home messages that only seem to confirm previous intuitions such as a bias of gaze towards the screen centre, towards speaking faces, moving objects or subtitles. However, I would argue that such messages fail to appreciate the nuance in the data. Empirical data correctly measured and analysed can provide subtle insights into a phenomenon that subjective introspection could never supply.

For example, film editors believe that an impression of continuous action can be created across a cut by overlapping somewhere between two (Anderson, 1996) and four frames (Dmytryk, 1986) of the action. However, psychological investigations of time perception revealed that our judgements of duration depend on how attention is allocated during the estimated period (Zakay and Block, 1996) and will vary depending on whether our eyes remain still or saccade during the period (Yarrow et al, 2001). In my thesis (Smith, 2006) I used simplified film stimuli to investigate the role that visual attention played in estimation of temporal continuity across a cut and found that participants experienced an overlap of 58.44ms as continuous when an unexpected cut occurred during fixation and an omission of 43.63ms as continuous when they performed a saccade in response to the cut. As different cuts may result in different degrees of overt (i.e., eye movements) and covert attentional shifts these empirical findings both support editor intuitions that temporal continuity varies between cuts (Dmytryk, 1986) whilst also explaining the factors that are important in influencing time perception at a level of precision not possible through introspection.

Reflecting on our own experience of a film suffers from the fact that it relies on our own senses and cognitive abilities to identify, interpret and express what we experience. I may feel that my experience of a dialogue sequence from Antichrist (2010) differs radically from a similar sequence from Secrets & Lies (1996) but I would be unable to attribute these differences to different aspects of the two scenes without quantifying both the cinematic features and my responses to them. Without isolating individual features I cannot know their causal contribution to my experience. Was it the rapid camera movements in Antichrist, the temporally incongruous editing, the emotionally extreme dialogue or the combination of these features that made me feel so unsettled whilst watching the scene? If one is not interested in understanding the causal contributions of each cinematic decision to an audience member’s response then one may be content with informed introspection and not find empirical hypothesis testing the right method. I make no judgement about the validity of either approach as long as each researcher understands the limits of their approach.

Introspection utilises the imprecise measurement tool that is the human brain and is therefore subject to distortion, human bias and an inability to extrapolate the subjective experience of one person to another. Empirical hypothesis testing also has its limitations: research questions have to be clearly formulated so that hypotheses can be stated in a way that allows them to be statistically tested using appropriate observable and reliable measurements. A failure at any of these stages can invalidate the conclusions that can be drawn from the data. For example, an eye tracker may be poorly calibrated resulting in an inaccurate record of where somebody was looking or it could be used to test an ill formed hypothesis such as how a particular film sequence caused attentional synchrony without having another film sequence to compare the gaze data to. Each approach has its strength and weaknesses and no single approach should be considered “better” than any other, just as no film should be considered “better” than any other film.

Conclusion

The articles collected here constitute the first attempt to bring together interdisciplinary perspectives on the application of eye tracking to film studies. I fully commend the intention of this special issue and hope that it encourages future researchers to conduct further studies using these methods to investigate research questions and film experiences we have not even conceived of. However, given that the recent release of low-cost eye tracking peripherals such as the EyeTribe[vii] tracker and the Tobii EyeX[viii] has moved eye tracking from a niche and highly expensive research tool to an accessible option for researchers in a range of disciplines, I need to take this opportunity to issue a word of warning. As I have outlined in this article, eye tracking is like any other research tool in that it is only useful if used correctly, its limitations are respected, its data is interpreted through the appropriate application of statistics and conclusions are only drawn that are based on the data in combination with a sound theoretical base. Eye tracking is not the “saviour” of film studies , nor is science the only “valid” way to investigate somebody’s experience of a film. Hopefully, the articles in this special issue and the ideas I have put forward here suggest how eye tracking can function within an interdisciplinary approach to film analysis that furthers our appreciation of film in previously unfathomed ways.

 

Acknowledgements

Thanks to Rachael Bedford, Sean Redmond and Craig Batty for comments on earlier drafts of this article. Thank you to John Henderson, Parag Mital and Robin Hill for help in gathering and visualising the eye movement data used in the Figures presented here. Their work was part of the DIEM Leverhulme Trust funded project (https://thediemproject.wordpress.com/). The author, Tim Smith is funded by EPSRC (EP/K012428/1), Leverhulme Trust (PLP-2013-028) and BIAL Foundation grant (224/12).

 

References

Anderson, Joseph. 1996. The Reality of Illusion: An Ecological Approach to Cognitive Film Theory. Southern. Illinois University Press.

Batty, Craig, Claire Perkins and Jodi Sita. 2015. “How We Came To Eye Tracking Animation: A Cross-Disciplinary Approach to Researching the Moving Image”, Refractory: a Journal of Entertainment Media, 25.

Banks, Martin S., Jenny R. Read, Robert S. Allison and Simon J. Watt. 2012. “Stereoscopy and the human visual system.” SMPTE Mot. Imag. J., 121 (4), 24-43

Bradley, Margaret M., Laura Miccoli, Miguel A. Escrig and Peter J. Lang. 2008. “The pupil as a measure of emotional arousal and autonomic activation.” Psychophysiology, 45(4), 602-607.

Branigan, Edward R. 1984. Point of View in the Cinema: A Theory of Narration and Subjectivity in Classical Film. Berlin: Mouton.

Brown, William. 2015. “There’s no I in Eye Tacking: How Useful is Eye Tracking to Film Studies?”, Refractory: a Journal of Entertainment Media, 25.

Buchan, Julie N., Martin Paré and Kevin G. Munhall. 2007. “Spatial statistics of gaze fixations during dynamic face processing.” Social Neuroscience, 2, 1–13.

Coutrot, Antoine, Nathalie Guyader, Gelu Ionesc and Alice Caplier. 2012. “Influence of Soundtrack on Eye Movements During Video Exploration”, Journal of Eye Movement Research 5, no. 4.2: 1-10.

Cutting, James. E., Jordan E. DeLong and Christine E. Nothelfer. 2010. “Attention and the evolution of Hollywood film.” Psychological Science, 21, 440-447.

Dwyer, Tessa. 2015. “From Subtitles to SMS: Eye Tracking, Texting and Sherlock”, Refractory: a Journal of Entertainment Media, 25.

Dyer, Adrian. G and Sarah Pink. 2015. “Movement, attention and movies: the possibilities and limitations of eye tracking?”, Refractory: a Journal of Entertainment Media, 25.

Dmytryk, Edward. 1986. On Filmmaking. London, UK: Focal Press.

Henderson, John. M., 2003. “Human gaze control during real-world scene perception.” Trends in Cognitive Sciences, 7, 498-504.

Hochberg, Julian and Virginia Brooks. 1978). “Film Cutting and Visual Momentum”. In John W. Senders, Dennis F. Fisher and Richard A. Monty (Eds.), Eye Movements and the Higher Psychological Functions (pp. 293-317). Hillsdale, NJ: Lawrence Erlbaum.

Holmqvist, Kenneth, Marcus Nyström, Richard Andersson, Richard Dewhurst, Halszka Jarodzka and Joost van de Weijer. 2011. Eye Tracking: A comprehensive guide to methods and measures. Oxford, UK: OUP Press.

James, William. 1890. The principles of psychology (Vol.1). New York: Holt

Kruger, Jan Louis, Agnieszka Szarkowska and Izabela Krejtz. 2015. “Subtitles on the Moving Image: An Overview of Eye Tracking Studies”, Refractory: a Journal of Entertainment Media, 25.

Le Meur, Olivier and Baccino, Thierry. 2013. “Methods for comparing scanpaths and saliency maps: strengths and weaknesses.” Behavior research methods, 45(1), 251-266.

Magliano, Joseph P. and Jeffrey M. Zacks. 2011. “The Impact of Continuity Editing in Narrative Film on Event Segmentation.” Cognitive Science, 35(8), 1-29.

Mital, Parag K., Tim J. Smith, Robin Hill. and John M. Henderson. 2011. “Clustering of gaze during dynamic scene viewing is predicted by motion.” Cognitive Computation, 3(1), 5-24

Rayner, Keith. 1998. “Eye movements in reading and information processing: 20 years of research”. Psychological Bulletin, 124(3), 372-422.

Rayner, Keith, Tim J. Smith, George Malcolm and John M. Henderson, J.M. 2009. “Eye movements and visual encoding during scene perception.” Psychological Science, 20, 6-10.

Raz, Gal, Yael Jacob, Tal Gonen, Yonatan Winetraub, Tamar Flash, Eyal Soreq and Talma Hendler. 2014. “Cry for her or cry with her: context-dependent dissociation of two modes of cinematic empathy reflected in network cohesion dynamics.” Social cognitive and affective neuroscience, 9(1), 30-38.

Redmond, Sean, Jodi Sita and Kim Vincs. 2015. “Our Sherlockian Eyes: the Surveillance of Vision”, Refractory: a Journal of Entertainment Media, 25.

Robinson, Jennifer, Jane Stadler and Andrea Rassell. 2015. “Sound and Sight: An Exploratory Look at Saving Private Ryan through the Eye-tracking Lens”, Refractory: a Journal of Entertainment Media, 25.

Salt, Barry. 2009. Film Style and Technology: History and Analysis (Vol. 3rd). Totton, Hampshire, UK: Starword.

Sawahata, Yasuhito, Rajiv Khosla, Kazuteru Komine, Nobuyuki Hiruma, Takayuki Itou, Seiji Watanabe, Yuji Suzuki, Yumiko Hara and Nobuo Issiki. 2008. “Determining comprehension and quality of TV programs using eye-gaze tracking.” Pattern Recognition, 41(5), 1610-1626.

Smith, Murray. 2011. “Triangulating Aesthetic Experience”, paper presented at the annual Society for Cognitive Studies of the Moving Image conference, Budapest, June 8–11, 201

Smith, Tim J. 2006. An Attentional Theory of Continuity Editing. Ph.D., University of Edinburgh, Edinburgh, UK.

Smith, Tim J. 2012a. “The Attentional Theory of Cinematic Continuity”, Projections: The Journal for Movies and the Mind. 6(1), 1-27.

Smith, Tim J. 2012b. “Extending AToCC: a reply,” Projections: The Journal for Movies and the Mind. 6(1), 71-78

Smith, Tim J. 2013. “Watching you watch movies: Using eye tracking to inform cognitive film theory.” In A. P. Shimamura (Ed.), Psychocinematics: Exploring Cognition at the Movies. New York: Oxford University Press. pages 165-191

Smith, Tim J. 2014. “Audiovisual correspondences in Sergei Eisenstein’s Alexander Nevsky: a case study in viewer attention”. Cognitive Media Theory (AFI Film Reader), Eds. P. Taberham & T. Nannicelli.

Smith, Tim J., Jonathan Batten and Rachael Bedford. 2014. “Implicit detection of asynchronous audiovisual speech by eye movements.” Journal of Vision,14(10), 440-440.

Smith, Tim J., Dekker, T., Mital, Parag K., Saez De Urabain, I. R. & Karmiloff-Smith, A., In Prep. “Watch like mother: Motion and faces make infant gaze indistinguishable from adult gaze during Tot TV.”

Smith, Tim J. and John M. Henderson. 2008. “Edit Blindness: The relationship between attention and global change blindness in dynamic scenes”. Journal of Eye Movement Research, 2(2):6, 1-17.

Smith Tim J., Peter Lamont and John M. Henderson. 2012. “The penny drops: Change blindness at fixation.” Perception 41(4) 489 – 492

Smith, Tim J., Daniel Levin and James E. Cutting. 2012. “A Window on Reality: Perceiving Edited Moving Images.” Current Directions in Psychological Science. 21: 101-106

Smith, Tim J. and Parag K. Mital. 2013. “Attentional synchrony and the influence of viewing task on gaze behaviour in static and dynamic scenes”. Journal of Vision 13(8): 16.

Smith, Tim J. and Janet Y. Martin-Portugues Santacreu. Under Review. “Match-Action: The role of motion and audio in limiting awareness of global change blindness in film.”

Smith, Tim. J. and Murray Smith. In Prep. “The impact of expertise on eye movements during film viewing.”

Suckfull, Monika. 2000. “Film Analysis and Psychophysiology Effects of Moments of Impact and Protagonists”. Media Psychology2(3), 269-301.

Vilaro, Anna and Tim J. Smith. 2011. “Subtitle reading effects on visual and verbal information processing in films.” Published abstract In Perception. ECVP abstract supplement, 40. (p. 153). European Conference on Visual Perception. Toulousse, France.

Velichkovsky, Boris M., Sascha M. Dornhoefer, Sebastian Pannasch and Pieter J. A. Unema. 2001. “Visual fixations and level of attentional processing”. In Andrew T. Duhowski (Ed.), Proceedings of the International Conference Eye Tracking Research & Applications, Palm Beach Gardens, FL, November 6-8, ACM Press.

Wass, Sam V. and Tim J. Smith. In Press. “Visual motherese? Signal-to-noise ratios in toddler-directed television,” Developmental Science

Yarrow, Kielan, Patrick Haggard, Ron Heal, Peter Brown and John C. Rothwell. 2001. “Illusory perceptions of space and time preserve cross-saccadic perceptual continuity”. Nature, 414.

Zakay, Dan and Richard A. Block. 1996. Role of Attention in Time Estimation Processes. Time, Internal Clocks, and Movement. Elsevier Science.

 

Notes

[ii] An alternative take on eye tracking data is to divorce the data itself from psychological interpretation. Instead of viewing a gaze point as an index of where a viewer’s overt attention is focussed and a record of the visual input most likely to be encoded into the viewer’s long-term experience of the media, researchers can instead take a qualitative, or even aesthetic approach to the data. The gaze point becomes a trace of some aspect of the viewer’s engagement with the film. The patterns of gaze, its movements across the screen and the coordination/disagreement between viewers can inform qualitative interpretation without recourse to visual cognition. Such an approach is evident in several of the articles in this special issue (including Redmond, Sita, and Vincs, this issue; Batty, Perkins, and Sita, this issue). This approach can be interesting and important for stimulating hypotheses about how such patterns of viewing have come about and may be a satisfying endpoint for some disciplinary approaches to film. However, if researchers are interested in testing these hypotheses further empirical manipulation of the factors that are believed to be important and statistical testing would be required. During such investigation current theories about what eye movements are and how they relate to cognition must also be respected.

[iii] Although, one promising area of research is the use of pupil diameter changes as an index of arousal (Bradley, Miccoli, Escrig and Lang, 2008).

[iv] This technique has been used for decades by producers of TV advertisements and by some “pop” serials such as Hollyoaks in the UK (Thanks for Craig Batty for this observation).

[v] This trend in increasing pace and visual complexity of film is confirmed by statistical analyses of film corpora over time (Cutting, DeLong and Nothelfer, 2010) and has resulted in a backlash and increasing interest in “slow cinema”.

[vi] Other authors in this special issue may argue that taking a critical approach to gaze heatmaps without recourse to psychology allows them to embed eye tracking within their existing theoretical framework (such as hermeneutics). However, I would warn that eye tracking data is simply a record of how a relatively arbitrary piece of machinery (the eye tracking hardware) and associated software decided to represent the centre of a viewer’s gaze. There are numerous parameters that can be tweaked to massively alter how such gaze traces and heatmaps appear. Without understanding the psychology and the physiology of the human eye a researcher cannot know how to set these parameters, how much to trust the equipment they are using, or the data it is recording and as a consequence may over attribute interpretation to a representation that is not reliable.

[vii] https://theeyetribe.com/ (accessed 13/12/14). The EyeTribe tracker is $99 and is as spatially and temporally accurate (up to 60Hz sampling rate) as some science-grade trackers.

[viii] http://www.tobii.com/eye-experience/ (accessed 13/12/14). The Tobii EyeX tracker is $139, samples at 30Hz and is as spatially accurate as the EyeTribe although the EyeX does not give you as much access to the raw gaze data (e.g., pupil size and binocular gaze coordinates) as the EyeTribe.

 

Bio

Dr Tim J. Smith is a senior lecturer in the Department of Psychological Sciences at Birkbeck, University of London. He applies empirical Cognitive Psychology methods including eye tracking to questions of Film Cognition and has published extensively on the subject both in Psychology and Film journals.

 

“Children should play with dead things”: transforming Frankenstein in Tim Burton’s Frankenweenie – Erin Hawley

$
0
0

Abstract: In this paper, I explore the possibility of retelling Mary Shelley’s novel Frankenstein in a children’s media text.  Like most material within the horror genre, Frankenstein is not immediately accessible to children and its key themes and tropes have traditionally been read as articulations of “adult” concerns.  Yet Frankenstein is also a tale with surprisingly child-centric themes.  With this in mind, I consider how the Frankenstein tale has been transformed within the constructed space of a child’s worldview in Tim Burton’s 2012 animated film Frankenweenie.  I argue that the film neither simplifies nor expresses great fidelity to Shelley’s novel, but instead cultivates a sense of curiosity and cultural literacy regarding the Frankenstein tale and the horror genre itself.

Sparky the dog. Frankenweenie (Tim Burton, 2012)

Sparky the dog. Frankenweenie (Tim Burton, 2012)

The horror genre has long been considered “off limits” to children.  From the rewriting of fairytales to erase their violent and scary content (Zipes 1993) to the literal defacement of eighteenth century children’s literature to remove traces of the Gothic (Townshend 2008), efforts to disentangle children’s texts from horror have given rise to the notion that children cannot derive the same sort of pleasure from “being scared” that adults can.  Recent scholarship has suggested, however, that children can and do take pleasure in horror material.  In her work on child cinema audiences in Britain, Sarah Smith has found that horror films in the 1930s were “extremely popular with children” due to the “mixed feelings of fear and fun” they evoked (2005, 58).  Writing of James Whale’s film Frankenstein (1931), Smith observes that children were “fascinated by its appeal and attended in droves” (2005, 70).  Similarly, David Buckingham’s research into children as horror viewers reveals that, while fright reactions to horror material can be powerful and long-lived, child audiences also take pleasure in the conventions of the horror text – they enjoy watching “evil destroyed” but also watching it “triumph”; they enjoy the feeling of fear itself and, like adult viewers, find pleasure in horror’s momentary destabilisation of societal norms (1996, 112-116).

The pleasures of horror from a child’s perspective have also been explored by Neil Gaiman (2006), who tells an interesting story about his daughter’s fascination with James Whale’s The Bride of Frankenstein (1935).  “My daughter Maddy loves the idea of The Bride of Frankenstein,” he writes: “she’s ten”.  Such fascination leads to dress-ups and play, and eventually to young Maddy and her friend watching the horror classic under Gaiman’s supervision.  When confronted with the movie itself, however, the enthusiasm wanes: the kids don’t get it.  As Gaiman observes, “They enjoyed it, wriggling and squealing in all the right places. But once it was done, the girls had an identical reaction. ‘Is it over?’ asked one. ‘That was weird,’ said the other, flatly. They were as unsatisfied as an audience could be”.

To some extent, this reaction is not surprising.  The Bride of Frankenstein is based on Mary Shelley’s Gothic novel Frankenstein, a text that – like most material within the horror genre – is usually read as an articulation of decidedly adult concerns.  From the original novel to its more recent manifestations in popular media, the Frankenstein tale is peppered with depictions of violence and violation, murder and misogyny; across the long history of its remaking in popular culture it has been interpreted as a story about genetic manipulation (Waldby 2002, 29), sexual transgression (Mellor 2003, 12-13), and post-partum depression (Johnson 1982, 6), to name just a few of its more adult-centric resonances.

Yet Frankenstein is also a tale with surprisingly child-centric themes.  At its heart, it is a story about what it means to be an outsider and what it means to encounter, experience, and negotiate otherness; these are themes that have more recently been explored by writers of children’s and young adult fiction from Roald Dahl to Stephenie Meyer.  As Barbara Johnson has pointed out, Frankenstein is also essentially a story about parent/child relationships: with its themes of monstrosity and technology, Johnson tells us, Shelley’s novel explores “the love-hate relation we have toward our children” (1982, 6).  Building on Johnson we can suggest that by offering us a glimpse of the world through the monster’s eyes the novel also briefly presents this “love-hate relation” from the child’s perspective, and that decades of Frankenstein movies continue this by offering the misunderstood monster as an icon of all that is unruly, confused, and frightening about childhood itself.

The story Gaiman tells about his daughter’s fascination with The Bride of Frankenstein and her reaction – “that was weird” – to the movie itself is a lovely articulation of the way children may be simultaneously drawn to and locked out of the Frankenstein tale.  It is interesting to note that Gaiman’s daughter and her friend were not frightened by the film or put off by its horror elements (indeed, they seemed to enjoy this aspect of the movie, “wriggling and squealing in all the right places”); instead, it was a certain indefinable strangeness that informed their ultimately “unsatisfied” reaction.  All this suggests that children can engage meaningfully and pleasurably with material in the horror genre, especially if that material is rewritten with a child’s perspective in mind.

In this article, I explore the relationship between Frankenstein and young audiences and consider the possibility of retelling Shelley’s novel in a children’s media text.  My analysis is inspired by the recent appearance of characters from Shelley’s novel and its various adaptations in three children’s animated films: Frankenweenie (Tim Burton, 2012), in which a boy named Victor Frankenstein reanimates his dog Sparky after a tragic car accident; Igor (Anthony Leondis, 2008), in which a hunch-backed laboratory assistant brings a female monster to life; and Hotel Transylvania (Genndy Tartakovsky, 2012), in which the Frankenstein monster and his Bride join Dracula and a host of other characters from the horror genre.  This trend towards engaging the Frankenstein myth in children’s media begs the question: how have such texts made Shelley’s tale accessible to young audiences, and with what degree of success?

Below, I take up this question with specific reference to Tim Burton’s Frankenweenie.  Not only is Burton’s film (as we shall see) the most highly regarded and in some senses the most successful of these three texts, it is also the most complex and arguably does not “dumb down” its source material.  My analysis of Frankenweenie will examine how the film constructs a “child’s eye view” and transforms the Frankenstein tale so that its characters, themes, and narratives make sense within the imagined space of a child’s world.  I will demonstrate that Burton’s film captures the spirit of its source text without necessarily striving for fidelity.  I will also consider some of Frankenweenie’s extra-textual material, exploring how reviews, product tie-ins, and even the film’s intertextual references contribute to its overall project of transforming but not simplifying the Frankenstein tale for children.

Adaptation, simplification, and transformation

Victor and his dog Sparky from Tim Burton's homage to the Frankenstein story, Frankenweenie (2012).

Victor and his dog Sparky from Tim Burton’s homage to the Frankenstein story, Frankenweenie (2012).

Frankenweenie is a stop-motion animation inspired by Burton’s earlier live-action film of the same name.  Here, the Frankenstein tale is relocated to one of Burton’s characteristic suburbia-scapes (“New Holland”), complete with manicured lawns, hedge sculptures, and monstrously mediocre residents.  Within this new narrative space, “Victor Frankenstein” is a child: a troubled, creative loner who spends his time tinkering in the attic, playing with his beloved dog Sparky, and making movies.  Tragedy enters Victor’s life when Sparky is killed in a car accident.  Inspired by his science teacher, the delightfully dour Mr Rzykruski, Victor steals Sparky’s body from the pet cemetery, drags the corpse back to the family home, and reanimates him in the attic.  When Victor’s classmates learn his secret, they try to replicate the experiment.  Chaos ensues as pets both living and dead are transformed into monsters who descend on New Holland, leading to a climactic showdown at the windmill overlooking the town.

The relationship between Frankenweenie and its source text, Mary Shelley’s Frankenstein, is complex.  Burton’s film both diverges from and intersects with Shelley’s novel, defining itself through patterns of fleeting fidelity and moments of spectacular transformation.  At the same time, the film makes reference to a plethora of other texts both within and beyond the Frankenstein mythos, thereby demonstrating the ways in which “adaptation” approaches and merges with “intertextuality” (see Elliott 2014; Martin 2009; Leitch 2003).  In other words, Frankenweenie is by no means a “faithful” retelling of Shelley’s Frankenstein.  It should be noted, however, that Shelley’s novel – despite being adapted many times in the centuries since its publication, across different media and in different genres – has not tended to inspire fierce fidelity in adapting authors.  As Albert Lavalley points out, Frankenstein tends to be “viewed by the playwright or the screenwriter as a mythic text, an occasion for the writer to let loose his own fantasies or to stage what he feels is dramatically effective, to remain true to the central core of the myth, [but] often to let it interact with fears and tensions of the current time” (1979, 245).

The notion of “fidelity” to an original text as the means of measuring an adaptation’s success, strength, and value has itself been thoroughly contested and problematised in recent years.  Fuelled particularly by the work of adaptation theorists such as Robert Stam (2005), Thomas Leitch (2003, 2007), and Imelda Whelehan (1999), this problematisation of the fidelity model has been an intervention in established ways of thinking about the relationship between an adaptive text and its source material.  As Will Brooker observes, though, fidelity criticism may be “outmoded and discredited within academia” but it has managed to “retain its currency within popular discourse” (2012, 45); in particular, it still informs the critical reception of films that adapt well-known novels or works of literature.  Even within academia, moreover, fidelity criticism has tended to linger in discussions of children’s media texts, particularly when the texts in question are retellings of classic or literary works.  It is often assumed that such adaptations carry some degree of responsibility for encouraging children to read and connect with the source material (Napolitano 2009, 81); in this way, the issue of fidelity becomes more urgent in the context of children’s media.

Concerns about fidelity in children’s adaptations are compounded by the issue of simplification.  Frankenweenie, for instance, is both an adaptation of a literary text and a reworking of classic horror films within the space of a child’s animation: the question we may immediately wish to ask, then, is “what has been lost in this process?”.  Both Shelley’s novel and the films of James Whale are today held in high regard as cultural classics, while the Frankenstein myth itself is a repository of ideas and cultural conversations about selfhood, embodiment, subjectivity, life, and death.  Potentially, the simplification of this myth for children would involve more than just a strategic removal of violent and sexual content in order to achieve a PG rating: it would be a process of dumbing down, a cleaning up of a story that works best when it is not “clean”.  It would also be a form of commercialisation, a reduction of a complex tale so that it can be packaged and marketed to young audiences.

These problems of simplification, commodification, and the dumbing down of source material are frequently mentioned by analysts of children’s adaptations, especially when the adaptation in question is a Disney product (as is Frankenweenie).  Writing in 1965 for the journal Horn Book, Frances Sayers refers to the “sweet” and “saccharine” nature of Disney adaptations and argues that, in order to both address and construct a child or family audience, Disney texts present life as lacking “any conflict except the obvious conflict of violence” (609).  Her concerns have been echoed by Hastings, who writes of the “conscious effort [by Disney] to produce children’s movies with no alarming moral ambiguities” (1993, 84).  Zipes, in turn, laments the way Disney has “‘violated’ the literary genre of the fairytale and packaged his versions in his name through the merchandising of books, toys, clothing, and records” (1995, 38).  Marc Napolitano’s work on the “Disneyfication of Dickens” is particularly relevant here because it explores the intersection between adult literature and children’s media.  Napolitano argues that the films Oliver & Company and The Muppet Christmas Carol – both Disney texts that retell canonical works by Charles Dickens – are “simplified and sanitized adaptations of Dickens that were marketed to families by the Walt Disney Company” (2009, 80).  In Oliver & Company in particular, Napolitano argues, Disney “lightens the material significantly and uses cute, cuddly animal characters, all of whom would be reproduced as stuffed toys, McDonald’s Happy Meal prizes… and countless other types of child-friendly merchandise to market the film to kids” (2009, 82).

All such criticism of Disney’s treatment of literary material is important, and functions as part of a wider interrogation of the seemingly “apolitical” and “critically untouchable” world of children’s animated film (Bell, Haas, and Sells 1995, 2).  As analysts of Disney products, it is essential that we disentangle ourselves from our own enjoyment of the Disney “magic”; this is part of what Zipes has called “Breaking the Disney Spell” (1995).  At the same time, however, claims about “Disneyfication” can be problematic when they make sweeping assumptions about young audiences, their levels of media literacy, and the ways in which they engage with media texts.  In other words, when accusing a children’s text of simplification we ourselves risk making an overly simplified reading of the child audience.  The charge of simplification becomes especially problematic when it lapses into what Semenza (2008) has termed the “dumbing down cliché”: the notion that adapting a literary text for children must always and automatically involve a process of reduction and commodification.

In the context of these concerns, Frankenweenie provides us with an interesting example because it resists the simplification process and simultaneously encourages its young audience to reconnect with the source material through means other than fidelity.  The film’s refusal to “Disneyfy” the Frankenstein tale is signified by the transformation of the Disney logo in the opening sequence: lightning strikes the familiar Disney castle and the picture turns black and white, a suggestion that there will be no fairies or cute, singing animals in the film that follows; that there will be no attempts to render the Frankenstein tale “safe” and “simple” even as it is opened up for young viewers.  While this transformation of the Disney logo is indicative of the mediating presence of Burton in the adaptation process, and of the supposed clash between the Disney and Burton brands, it can also be read as a resistance to simplification – a suggestion that the film will be “Frankenstein for kids” but not “Frankenstein lite”.

In what follows, I explore how Frankenweenie transforms (rather than simplifies) the Frankenstein tale within the imagined space of a child’s world.  I use the term “transformation” with an awareness of its applicability to studies of animation as an art form, a technology, and a mode of representation.  As both Susan Napier (2000) and Paul Wells (1998) have noted, animation has metamorphic qualities that distinguish it from live-action cinema and that manifest at the levels of story, body, and space.  Wells has also argued that the process of adapting a literary text into animated film can involve “an act of literal transformation which carries with it mythic and metaphoric possibility” (2007, 201).  In this way, the idea of transformation allows us to discuss children’s animated films as adaptations without making assumptions about animation as a medium (for instance, that it is inferior to live-action cinema) or about children as audiences (for instance, that they are incapable of understanding textual and intertextual complexities).

Transformation and the child’s eye view

In her analysis of the filmic adaptation of Maurice Sendak’s beloved picture book Where the Wild Things Are, Sarah Annunziato (2014) explores the construction of a “child’s eye view”, arguing that the film – while drawing attention and public comment for its scariness and mature themes – is appropriate for young audiences because it imagines the world as seen through a child’s eyes.  Similar claims can be made of Frankenweenie, which constructs a child’s view of the world and repeatedly invites its viewers to inhabit this childlike space.  In both these films, the creation of a child’s eye view specifically involves moments of scariness rather than excluding them.  The relationship between Frankenstein and Frankenweenie differs, however, from that between the book and film versions of Where the Wild Things Are because it involves a shift from adult to child audience.  In Burton’s film, therefore, employing a child’s perspective allows for a significant rethinking of the original tale.

In its simplest sense, this child’s eye view is visible in the depiction of New Holland and its residents.  While certainly reminiscent of some of Burton’s other visions of suburbia, New Holland is best described as a small town landscape seen through a child’s eyes: a place of long shadows and neat lines, of fantasy and darkness, of strange children and menacing adults.  The frequent use of low camera angles to depict some of these adult characters (such as the mayor Mr Bergermeister and the science teacher Mr Rzykruski) aligns us with Victor and invites us to adopt a child’s perspective.  While not menacing or imposing, Victor’s parents, too, are adults as seen by children: simplistic to the point of caricature, caught up in trivial or meaningless “grown-up” concerns (Victor’s father talks endlessly about his work as a travel agent; Victor’s mother is repeatedly seen vacuuming the house and/or reading romance novels).  On the other hand, the world of Victor (the child’s world) is depicted as complex, detailed, and intricate.  This is best represented by the attic, a cluttered space of creativity, invention, and play – and a notable contrast with the rest of Victor’s house and suburb, which are neat, sparse, and boring.

This inherent difference between adults and children – and the resultant conflict, always seen from the child’s perspective – is central to the plot of Frankenweenie.  From the opening scenes we learn that Victor is misunderstood by his mostly well-meaning parents, who worry that he spends too much time alone and will “turn out weird”.  His father encourages Victor to take up baseball, which leads inadvertently to Sparky’s death: the little dog meets his doom while chasing a ball hit by Victor.  The subsequent depiction of Victor’s grief is highly moving, all the more so because his parents do not seem to understand the extent of his sadness.  His mother offers clichés and platitudes: “If we could bring him back, we would” and “when you lose someone they never really leave you – they just move into a special place in your heart”, which Victor interprets as hollow and macabre (“I don’t want him in my heart”, he objects, “I want him here with me”).  These early scenes suggest a sense of turmoil beneath the calm surface of even the most loving parent-child relationship: a version, perhaps, of the “love-hate relation” that Johnson (1982, 6) detects within Shelley’s visions of monstrosity.  They also reveal that the world seen through a child’s eyes is not a simple place, even though it may be dominated by fantasies (such as the desire to bring Sparky back to life, which Victor soon fulfills).

It is through these early depictions of conflict, death, and grief that the film captures the thematic spirit of Mary Shelley’s novel.  In Shelley’s text, Victor Frankenstein is driven to create his monster by a desire to suspend mortality and escape the horrors of death and decay: Shelley’s Victor is both haunted and inspired by the death of his mother, Caroline, which leads him to seek out scientific means of “renew[ing] life where death had apparently devoted the body to corruption” (Shelley 1993, 43).  For children, the death of a pet is often a first experience of mortality; thus in Frankenweenie it is the dog, Sparky’s, death that allows Victor to confront the notion of perishability that so horrifies his predecessor in Shelley’s novel.  This experience of death and perishability also precipitates the story events and initiates the move into the horror genre by inspiring Victor’s act of monster-making.

The scene in which Victor reanimates Sparky provides Burton and his team with much opportunity to revel in horror movie history and to pay homage to the films of James Whale, particularly Frankenstein (1931) and its sequel, The Bride of Frankenstein (1935).  Lightning flashes and thunder crashes as Victor sews Sparky’s body back together and fixes bolts to his neck; the body is then covered by a sheet and raised through the roof to receive the life-giving electric charge.  Yet here, too, the child’s eye view is at work.  Attentive viewers will notice that Sparky’s body is laid out on an ironing board, and that toys, appliances, and other household objects form part of the elaborate life-giving apparatus.  Signifiers of “childhood” and “ordinariness” are thus interwoven with the signifiers of life, creation, and monstrosity borrowed from Whale.  Instead of fingers twitching and eyes opening, Sparky’s “alive-ness” is signified by a wagging tail; and instead of proclaiming “It’s alive!” like his predecessor in the Whale films, young Victor Frankenstein says “You’re alive”.  This shift in language reveals that the monster has been created according to a child’s desires and wishes: the moment of creation is framed by Victor’s desire not only for Sparky to still be alive but for the friendship, happiness, and unconditional love that a pet often represents.  Accordingly, the child views the monster as a friend and companion (you) rather than as the product of an experiment (it).

This transformation of Frankenstein to suit a child’s perspective certainly involves a degree of softening, a removal of some aspects of violence and conflict that define the original tale.  For instance, Shelley’s novel and most of its adaptations are constructed around the conflict between monster and maker – this conflict is not present in Frankenweenie.  As we would expect given the film’s target audience, Burton and his screenwriter John August also de-sexualise the Frankenstein tale: another notable absence is Shelley’s sub-plot involving the creation of a mate for the monster, and the resultant murder of Victor’s bride Elizabeth on their wedding night.  This does not mean, however, that Frankenweenie shies away from an exploration of monstrosity and horror.  Indeed, while Sparky himself is not depicted as a true monster, the film is replete with images of monstrosity.  These come particularly in the form of the creatures that Victor’s classmates bring to life: pets and other icons of familiarity, domesticity, and innocence (sea monkeys, a fluffy white cat, a dead hamster) who become snarling, terrifying, rampaging beasts.  The image of these monsters running amok through the fairground of New Holland encapsulates the film’s transformation of its source material.  This scene is only tenuously connected to Shelley’s plot, yet it resounds with Frankensteinian questions and dilemmas, particularly as they might be understood by children: When you have created your monster, what are you going to do with him/her/it?  And what happens if your monster (your game, your story) escapes your control?  While exploring the lighter side of monster-making, then, the film also explores the darker side of play, re-interpreting the Frankensteinian themes of creativity, perishability, and the life/death boundary so that they are seen from a child’s perspective.

Paratexts, intertexts, and the complex world of Frankenweenie

From Frankenweenie: an Electrifying Book. One of the many examples of the process of stop-motion animation and the making of Sparky.

From Frankenweenie: an Electrifying Book. One of the many examples of the process of stop-motion animation and the making of Sparky.

While not a notable box office success, Frankenweenie received a generally positive critical reception.  The film is rated highly – at 87% – on the aggregate review website Rotten Tomatoes, and is frequently described by reviewers as an enjoyable product for both children and adults (see, for instance, Paatsch 2012; Chang 2012; Mazmanian 2012).  Occasionally, charges of simplification are levelled at the film: Peter Bradshaw in The Guardian describes Frankenweenie as “a sentimental kind of retro gothic lite, appearing under the Disney banner” (2012), while A.O. Scott in the New York Times writes that “the movie, a Walt Disney release, also feels tame and compromised” (2012).  Other reviewers found the film dark enough to be entertaining, with many making positive mention of Burton’s ability to balance the sweetness of a children’s story with the darkness of a horror film.  Leigh Paatsch in the Herald Sun, for instance, commends the film for “deftly balancing blatant eeriness with a chipper cheeriness that excuses many a macabre event” (2012).  Lou Lumenick in the New York Post praises the film for its “creepy but basically sweet humor” (2012), as does Matthew Bond in the Daily Mail Australia who describes it as “strange, but also touching and lovely” (2012).  In Time magazine, Richard Corliss addresses the film’s boundary-crossing quality when he notes approvingly that “Frankenweenie’s message to the young” is that “children should play with dead things” (2012).

This positive reception sets Burton’s film apart from other recent children’s films that play with horror tropes and characters, such as the aforementioned Hotel Transylvania and Igor, both of which received lukewarm reviews.  Hotel Transylvania in particular was frequently criticised for its shallow approach to the narratives it draws upon, including Shelley’s Frankenstein (see, for instance, Reynolds 2012; Collin 2012).  L. Kent Wolgamott in the Lincoln Journal Star (2012) observes that while Frankenweenie did not perform as well at the box office as Hotel Transylvania, it is “by far, the superior film” (and he contextualises this comment by urging readers not to “consider box-office returns to be the only measure of a film’s success”, adding that with Frankenweenie Burton has created a “masterpiece”).

Some reviews of Frankenweenie mention the construction of a child’s eye view.  Adam Mazmanian in The Washington Times, for instance, identifies this as the means by which the film “draw[s] in young audiences”, adding that its “knowing winks at horror-movie history will appeal to grown-ups” (2012).  It is interesting that Mazmanian feels the need to separate the film’s audience into these two distinct categories, and that he distinguishes the “adult” and “child” sections of the audience by an ability (or lack thereof) to “get” the film’s intertextual references.  Wolgamott takes this further, praising the film for its references to classic horror movies but adding “that’s not anything the preschool through middle school animation crowd is going to get, or could possibly care about” (2012).  Both critics agree that intertextuality is a means by which Frankenweenie resists simplification and becomes something more than a light and fluffy children’s film.  At the same time, both critics produce distinct readings of the film’s child and adult audiences, and locate the qualities of media literacy and cultural awareness (which might enable the decoding of the film’s intertextuality) squarely within the adult space.

It is certainly true that Frankenweenie is littered with intertextual references: to other texts in the Frankenstein mythos (particularly the films of James Whale), to films in Burton’s oeuvre (such as Edward Scissorhands), and to texts in the horror genre more broadly (such as the Japanese monster movie Gamera).  This is coupled with a playful self-reflexivity that we often see in filmic adaptations of Frankenstein.  As Esther Schor (2003) has pointed out, adaptations of Shelley’s novel – from the early stage productions to the first known Frankenstein film in 1910, and beyond – often depict the monster’s coming-to-life in a spectacular and self-referential way; most filmic versions, in particular, play upon what William Nestrick (1979, 292) has termed the “myth of animation” – a thematic link or bridge between the Frankenstein tale and cinema’s own powers to bring a still image, body, or scene to life.  Frankenweenie, of course, is an animated film, and this brings new meaning to Nestrick’s “myth”.  The technologies of movie-making and, specifically, stop-motion animation are spectacularised in the image of Sparky’s coming-to-life, adding another layer of intertextuality to a film already rich with cultural references.

It may be tempting to assume, as do Mazmanian and Wolgamott, that children are excluded from this intertextual conversation.  Indeed, it has become increasingly common for children’s films to engage in a dual mode of address, enchanting children with stories, songs, and imagery while offering jokes, intertextual references, or clever moments of self-awareness to adults.  The implication is that long-suffering parents should be rewarded for watching films with their children or otherwise lured into the watching process by the promise of adult-centric entertainment.  Burton’s film is somewhat different because the intertextual references are closely bound to the narrative – they are less an amusing aside for adults than part of the film’s very fabric.  They are also, potentially, a means of encouraging audiences to connect with the source material.  While Frankenweenie does not openly strive to generate reverence for (or even awareness of) Mary Shelley, her novel, and the act of reading Frankenstein, it arguably promotes a more complex form of literacy that speaks directly to the process of adaptation itself.  By referencing the movies of James Whale, in particular, Burton positions his film within a web of Frankenstein texts and also destabilises the primacy of Shelley’s novel as source text: adapting Frankenstein, we are told, is a complex business that involves the engagement with already apparent intertextuality rather than the “recovery” of a single source text from out of the depths of adaptation history.

It is likely, furthermore, that many of the children who constitute Frankenweenie’s primary audience are able to decode the film’s intertextual references due to their familiarity with the horror genre and its tropes, characters, and conventions.  As noted above, children have traditionally been locked out of the horror genre; in recent years, however, encounters between young audiences and horror have been initiated through a plethora of child-friendly horror texts: as well as Hotel Transylvania and Igor, these include the films ParaNorman (Sam Fell and Chris Butler, 2012) and Monsters vs Aliens (Rob Letterman and Conrad Vernon, 2009), the video game Plants vs Zombies (PopCap Games, 2009), the books and television series Grossology (Sylvia Branzei, 1992-1997; Nelvana Limited, 2006-2009), and Chris Riddell’s Goth Girl books (2013-2015), as well as older but still relevant texts such as The Simpsons (which frequently lampoons the genre through its “Treehouse of Horror” episodes).  Meanwhile, imagery and tropes from the Frankenstein tale have been so pervasively circulated in popular culture for such a long time that children are likely to have some degree of familiarity with the tale even if they do not connect it to its original source.  Indeed, it is not uncommon for children’s texts to make passing reference to the tale and its characters (for instance, an episode of the cartoon Spongebob Squarepants is entitled “Frankendoodle”, while Dav Pilkey’s Captain Underpants books contain a character named “Frankenbooger”).

It is also likely that children today are familiar with complex levels of intertextuality and are adept at negotiating intersecting currents of media; thus Cathlena Martin writes of the “overlapping intertextual nature of children’s culture” (2009, 86).  In her analysis of the transmedia adaptation of the novel Charlotte’s Web, Martin claims that an enjoyment and understanding of intertextuality may come more naturally to today’s children, who “experience transmedia stories on a regular basis” and therefore “no longer view the printed text as the only way to experience [a literary classic such as] Charlotte’s Web”, whereas adults are more likely to “resist multi-media adaptation, relying on the supremacy of print text as ‘high art’” (2009, 88).  This returns us to the concept of “fidelity” to an original text, and suggests that in discussions of adaptation for children fidelity is likely to be a concept imposed by adult readers and critics rather than something inherently understood or valued by children.  If this raises concerns over the disappearance of “the book” as a cultural object, it also demonstrates that “simplicity” is not a concept that sits well with the highly interconnected, transmedia quality of children’s culture today.

The promotional release of the free Frankenweenie: an Electrifying Book. The e-book explores the production of Frankenweenie: readers are given access to production photographs, original artwork, and interviews. This is a promotional mock poster for a film titled "Return of the Vampire Cat".

The promotional release of the free Frankenweenie: an Electrifying Book. The e-book explores the production of Frankenweenie: readers are given access to production photographs, original artwork, and interviews. This is a promotional mock poster for a film titled “Return of the Vampire Cat”.

Interestingly, the promotional material for Frankenweenie played upon this ability in young audiences to understand and enjoy intertextuality.  Elliott reminds us that “[t]ie-in merchandise produces and distributes the culture of Disney beyond the cinema” (2014, 195); yet the marketing campaign for Frankenweenie took a very different route from the usual toys, games, and Happy Meals associated with Disney and with the process of Disneyfication.  Instead, the film was promoted through such unusual means as the release of six mock B-movie posters each featuring one of the child characters together with the monster he/she creates (including Night of the Were-Rat: a Tale of Terror featuring “Edgar E. Gore” and Return of the Vampire Cat featuring “Weird Girl”).  These promotional texts not only foreground the film’s child protagonists (as opposed to its adult characters) but serve to locate “childhood” within the parameters of the horror genre and within monster-movie history.  If entryway paratexts guide and instruct our viewing of a media text, as Gray (2010) suggests, these posters invite us to connect childhood with monstrosity in a way that “preps” us for the viewing of Frankenweenie itself (whether we are adults or children).  They also underscore the overall playfulness of the film and relatedly its resistance to the processes of Disneyfication and simplification.  Due to the foregrounding of the child characters, furthermore, the posters specifically address child audiences and clearly include them in the film’s intertextual conversation.

Another key aspect of the film’s promotion was the release of a free e-book entitled Frankenweenie: an Electrifying Book.  Designed for audiences of all ages, the e-book explores the production of Frankenweenie: readers are given access to production photographs, original artwork, and interviews, with particular emphasis on the process of stop-motion animation and the making of Sparky (who we can view as a sketch, a 3-D model, and a finished “product”).  In this way, the e-book allows children access to Nestrick’s “myth of animation” and to the idea of animation as a “bridge” between the narrative and the technology of Frankenweenie.  The e-book also makes the film’s intertextuality more evident.  It begins, for instance, with a foreward by actor Martin Landau accompanied an image of the character he voices (Mr Rzykruski); Landau discusses his previous collaboration with Tim Burton, the film Ed Wood, and his role in this film as Bela Lugosi, star of the horror classic Dracula (Tod Browning, 1931).  A pop-up button informs us that Landau’s character is also “a nod to Vincent Price, the late actor known for his iconic roles in various horror films” (Disney Book Group, 2012).  The e-book thus enables or enhances the ability of any audience member (including children) to decode the film’s intertextual references – and even, arguably, leads young audiences back to the various source texts that inspired Frankenweenie.

In this way, the film (together with its promotional material) both assumes and encourages a level of cultural literacy regarding the Frankenstein tale and, more broadly, the horror genre itself.  As this analysis has demonstrated, the film’s intertextuality works together with its paratexts to cultivate an awareness of what lies beyond its own textual boundaries.  Frankenweenie thus imagines and constructs its audience to be a media-literate and curious child.

Conclusion

Prior to the release of Tim Burton’s Frankenweenie, the thought of an animated film based on Mary Shelley’s Frankenstein and released under the Disney banner might have horrified literary purists and fans of horror cinema alike.  An animated Frankenstein, in which darkness and moral conflict are replaced by cute animal side-kicks and catchy songs, may well have been taken as a sign of Disney’s cultural domination and its ability not just to appropriate literary material but to colonise sites of literary and cultural meaning.  Burton’s film, however, demonstrates that “Disneyfication” is not the only route to adapting a literary classic for children, and that the transformation of such a tale within the space of a child’s worldview need not involve a simplification process.  As noted above, we can contextualise Frankenweenie within a recent trend in media and popular culture that has seen the horror genre re-imagined for young audiences; yet Burton’s film can be read not just as an example of “horror for kids” but as a startlingly successful transformation of a previously inaccessible tale in line with the concerns that define a child’s world.  Importantly, Frankenweenie’s most powerful images are not cartoonish renditions of monsters and mad scientists – they are the images of Victor grieving for Sparky, and of the neighbourhood kids struggling to control the monsters they have unleashed.  These themes of loss, and of losing control, are central to the film’s re-imagining of a classic horror tale according to a child’s eye view.  In this way, Frankenweenie makes Frankenstein accessible to children and also gives adult viewers a sense of what horror, otherness, and monstrosity could mean to a child.

 

Works cited

Annunziato, Sarah.  2014.  “A Child’s Eye View of Where the Wild Things Are: Lessons from Spike Jonze’s Film Adaptation of Maurice Sendak’s Picture Book.”  Journal of Children and Media 8 (3): 253-266.

Bell, Elizabeth, Lynda Haas, and Laura Sells.  1995.  “Introduction: Walt’s in the Movies”.  In From Mouse to Mermaid: The Politics of Film, Gender, and Culture, edited by Elizabeth Bell, Lynda Haas, and Laura Sells, 1-17.  Bloomington and Indianapolis: Indiana University Press.

Bond, Matthew.  2012.  “Dr Burton’s gothic horror has a heart: Frankenweenie is touching and enjoyable”.  Daily Mail Australia, October 23.  Accessed August 6, 2014.  http://www.dailymail.co.uk/tvshowbiz/article-2221392/Frankenweenie-movie-review-Tim-Burtons-gothic-horror-heart.html

Bradshaw, Peter.  2012.  “Frankenweenie: first look”.  The Guardian, October 11.  Accessed August 6, 2014.  http://www.theguardian.com/film/2012/oct/10/frankenweenie-review-london-film-festival-tim-burton

Brooker, Will.  2012.  Hunting the Dark Knight: Twenty-first Century Batman.  London and New York: I.B. Tauris.

Buckingham, David.  1996.  Moving Images: Understanding Children’s Emotional Responses to Television.  Manchester and New York: Manchester University Press.

Chang, Justin.  2012.  “Review: Frankenweenie”.  Variety, September 20.  Accessed August 6, 2014.  http://variety.com/2012/film/reviews/frankenweenie-1117948379/

Collin, Robbie.  2012.  “Hotel Transylvania review”.  The Telegraph, October 11.  Accessed August 6, 2014.  http://www.telegraph.co.uk/culture/film/filmreviews/9602345/Hotel-Transylvania-review.html

Corliss, Richard.  2012.  “Tim Burton’s Frankenweenie Movie Review: A Re-animated Delight”.  Time, October 4.  Accessed August 6, 2014.  http://entertainment.time.com/2012/10/04/tim-burtons-frankenweenie-a-re-animated-delight/

Elliott, Kamilla.  2014.  “Tie-Intertextuality, or, Intertextuality as Incorporation in the Tie-in Merchandise to Disney’s Alice in Wonderland (2010)”.  Adaptation 7 (2): 191-211.

Frankenweenie: an Electrifying Book.  2012.  Disney Book Group.  Accessed August 20, 2014.  Available at https://itunes.apple.com/au/book/frankenweenie-electrifying/id557041056?mt=11

Gaiman, Neil.  2006.  “The Bride of Frankenstein”.  Neil Gaiman (official website).  Accessed 14 June 2013. http://www.neilgaiman.com/p/Cool_Stuff/Essays/Essays_By_Neil/The_Bride_of_Frankenstein.  From Cinema Macabre, edited by Mark Morris.  Hornsea: PS.

Gray, Jonathon.  2010.  Show Sold Separately: Promos, Spoilers, and Other Media Paratexts.  New York: New York University Press.

Hastings, A. Waller.  1993.  “Moral Simplification in Disney’s The Little Mermaid”.  The Lion and the Unicorn 17 (1): 83-92.

Johnson, Barbara.  1982.  “My Monster/My Self”.  Diacritics 12: 2-10.

Lavalley, Albert J.  1979.  “The Stage and Film Children of Frankenstein: A Survey”.  In The Endurance of Frankenstein: Essays on Mary Shelley’s Novel, edited by George Levine and U.C. Knoeplfmacher, 243-289.  Berkeley: University of California Press.

Leitch, Thomas.  2003.  “Twelve Fallacies in Contemporary Adaptation Theory”.  Criticism 45 (2): 149-171.

Leitch, Thomas.  2007.  Film Adaptation and its Discontents: From Gone with the Wind to the Passion of the Christ.  Baltimore: John Hopkins University Press.

Lemire, Christy.  2012.  “Frankenweenie Review: Tim Burton Reminds Us Why We Love Him”.  The Huffington Post, October 2.  Accessed August 6, 2014.  http://www.huffingtonpost.com/2012/10/03/frankenweenie-review-tim-burton_n_1935142.html

Lumenick, Lou.  2012.  “‘Frankenweenie is a Monster Piece!”.  The New York Post, October 5.  Accessed August 6, 2014.  http://nypost.com/2012/10/05/frankenweenie-is-a-monster-piece/

Martin, Cathlena.  2009.  “Charlotte’s Website: Media Transformation and the Intertextual Web of Children’s Culture”.  In Adaptation in Contemporary Culture: Textual Infidelities, edited by Rachel Carroll, 85-95.  London: Continuum.

Mazmanian, Adam.  2012.  “Movie Review: Frankenweenie”.  The Washington Times, October 4.  Accessed August 6, 2014. http://www.washingtontimes.com/news/2012/oct/4/movie-review-frankenweenie/#ixzz37OlxdQap.

Mellor, Anne K.  2003.  “‘Making a Monster’: an introduction to Frankenstein”.  In The Cambridge Companion to Mary Shelley, edited by Esther Schor, 9-25.  Cambridge: Cambridge University Press.

Napier, Susan J.  2000.  Anime: From Akira to Princess Mononoke – Experiencing Contemporary Japanese Animation.  New York: Palgrave.

Napolitano, Marc.  2009.  “Disneyfying Dickens: Oliver & Company and The Muppet Christmas Carol as Dickensian Musicals”.  Studies in Popular Culture 32 (1): 79-102.

 Nestrick, William.  1979.  “Coming to Life: Frankenstein and the Nature of Film Narrative”.  In The Endurance of Frankenstein: Essays on Mary Shelley’s Novel, edited by George Levine and U.C. Knoeplfmacher, 290-315.  Berkeley: University of California Press.

Paatsch, Leigh.  2012.  “Film review: Frankenweenie Enchants Adults Too”.  Herald Sun, October 25.  Accessed August 6, 2014.  http://www.heraldsun.com.au/entertainment/movies/frankenweenie-enchants-adults-too/story-e6frf8r6-1226503055322

Reynolds, Simon.  2012.  “Hotel Transylvania Review”.  Digital Spy, October 9.  Accessed August 6, 2014.  http://www.digitalspy.com.au/movies/review/a422323/hotel-transylvania-review.html#~oM8bcdFfwKA0yf

Sayers, Frances Clarke.  1965.  “Walt Disney Accused”.  Horn Book 41: 602-611.

Schor, Esther.  2003.  “Frankenstein and Film”.  In The Cambridge Companion to Mary Shelley, edited by Esther Schor, 63-83.  Cambridge: Cambridge University Press.

Scott, A.O.  2012.  “It’s Aliiiive! And Wagging Its Tail – ‘Frankenweenie,’ Tim Burton’s Homage to Horror Classics”.  The New York Times, October 4.  Accessed August 6, 2014.  http://www.nytimes.com/2012/10/05/movies/frankenweenie-tim-burto…age-to-horror-classics.html?smid=tw-nytimesmovies&seid=auto&_r=0

Semenza, Gregory M. Colón.  2008.  “Teens, Shakespeare, and the Dumbing Down Cliché: The Case of The Animated Tales”.  Shakespeare Bulletin 26 (2): 37-68.

Shelley, Mary.  1993.  Frankenstein, or, The Modern Prometheus.  Hertfordshire: Wordsworth Editions.

Smith, Sarah J.  2005.  Children, Cinema and Censorship: from Dracula to the Dead End Kids.  London and New York: I.B. Tauris.

Stam, Robert.  2005.  “Introduction: the Theory and Practice of Adaptation”.  In Literature and Film: A Guide to the Theory and Practice of Film Adaptation, edited by Robert Stam and Alessandra Raengo, 1-52.  Malden, MA: Blackwell.

Townshend, Dale.  2008.  “The Haunted Nursery: 1764-1830”.  In The Gothic in Children’s Literature: Haunting the Borders, edited by Anna Jackson, Karen Coats, and Roderick McGillis, 15-38.  London and New York: Routledge.

Waldby, Catherine.  2002.  “The Instruments of Life: Frankenstein and Cyberculture”.  In Prefiguring Cyberculture: An Intellectual History, edited by Darren Tofts, Annemarie Jonson and Alessio Cavallaro, 28-37.  Cambridge and London: MIT Press.

Wells, Paul.  1998.  Understanding Animation.  New York: Routledge.

Wells, Paul.  2007.  “Classic Literature and Animation: All Adaptations are Equal, but Some are More Equal Than Others”.  In The Cambridge Companion to Literature on Screen, edited by Deborah Cartmell and Imelda Whelehan, 199-211.  Cambridge and New York: Cambridge University Press.

Whelehan, Imelda.  1999.  “Adaptations: the Contemporary Dilemmas”.  In Adaptations: From Text to Screen, Screen to Text, edited by Deborah Cartmell and Imelda Whelehan, 3-19.  London and New York: Routledge.

Wolgamott, L. Kent.  2012.  “Frankenweenie a box-office bomb, but superior film”.  Lincoln Journal Star, October 10.  Accessed August 6, 2014.  http://journalstar.com/entertainment/movies/l-kent-wolgamott-fran…b-but-superior/article_42409e82-89b9-5794-8082-7b5de3d469e2.html.=

Zipes, Jack.  1993.  “The Trials and Tribulations of Little Red Riding Hood”.  In The Trials and Tribulations of Little Red Riding Hood, edited by Jack Zipes, 17-88.  London and New York: Routledge.

Zipes, Jack.  1995.  “Breaking the Disney Spell”.  In From Mouse to Mermaid: The Politics of Film, Gender, and Culture, edited by Elizabeth Bell, Lynda Haas, and Laura Sells, 21-42.  Bloomington and Indianapolis: Indiana University Press.

 

Films cited

Burton, Tim.  2012.  Frankenweenie.  USA: Walt Disney Pictures.

Leondis, Anthony.  2008.  Igor.  USA: Roadshow Entertainment.

Tartakovsky, Genndy.  2012.  Hotel Transylvania.  USA: Columbia Pictures.

Whale, James.  1931.  Frankenstein.  USA: Universal Pictures.

Whale, James.  1935.  The Bride of Frankenstein.  USA: Universal Pictures.

 

Bio:

Erin Hawley teaches in the Journalism, Media, and Communications program at the University of Tasmania.  Her current research interests include children’s media culture, adaptation, and media education.

Volume 26

$
0
0

“You gave me no choice”: A queer reading of Mordred’s journey to villainy and struggle for identity in BBC’s Merlin – Joseph Brennan

$
0
0

Abstract: This essay performs a queer reading of the Mordred character—that great archetype of the treacherous villain—from BBC’s Merlin (2008–2012) so as to examine his role in a series that garnered a devoted following among ‘slash fans,’ who homoeroticise male pairings. By charting the various catalysts that set this villain on his path, we are privy to insights into the representations and (queer) metaphors of this popular British series and what these elements have to tell us about this reimagined legendary villain. This reading is supported by analysis of slash fanart (known as ‘slash manips’), which support my reading and delve into typologies that help examine the construction and journey of Mordred as the archetypal villain, as well as his multiple identities of knight and magician, and queer associations of his struggle for self. This reading offers insight into the reimagining of an iconic villain, as well as the various types and queer metaphors the character’s journey in this popular series illuminates.

Introduction

The Arthurian legend’s Mordred, like Bram Stoker’s (1897) Count Dracula or Arthur Conan Doyle’s (1894) Professor Moriarty, is one of literature’s most iconic villains; his portrayal in the legend’s best-known rendition, Thomas Malory’s (1485) Le Morte d’Arthur, for example, is as a Judas figure. (For those unfamiliar with the legends of King Arthur, Aronstein 2012 is an accessible introduction.) The Mordred character’s morphological qualities as the archetypal villain (see Propp 1968), combined with his weight in Arthurian literature, meant his appearance and relationship with Arthur—that great hero of Western literature and folklore, fated to die at Mordred’s hand (see Sutton 2003)—was highly anticipated from the start of the BBC’s recent television adaptation of the legend, Merlin (2008–2012). Mordred was also a major source of tension for the titular character in the series, ‘Merlin the Magician,’ who in this adaptation keeps his magical identity as the most powerful wizard in all of Albion (Britain) secret from ‘Arthur the King’ until Arthur’s death at an also-magical (and also-knight) Mordred’s hand in the climactic Battle of Camlann, which ended the program’s five-year run. This essay performs a queer reading of the Mordred character so as to examine his role in a series that has garnered a devoted following among slash fans, who create artistic works that actualise latent homoeroticism in popular texts. This reading is bolstered by analysis of select ‘slash manips’ featuring the character. A form of visual slash, these images help to anchor this author’s reading by connecting it with fans’ own queer interpretations of Mordred and his interactions with other men, Merlin and Arthur specifically. By charting the various catalysts that set this villain on his path, we are privy to insights into the representations and (queer) metaphors of this popular British series, and what these elements have to tell us about this reimagined legendary villain. Further, such a reading allows us to hypothesise about how Mordred’s villainy could all have been avoided if only his dual identities of Magician and Warrior had been accepted by his mentor, Merlin, and his master, Arthur.

Merlin (2008–2012)

Spanning five years and 65 episodes, Merlin chronicles the namesake’s acceptance and fulfilment of his destiny to assist Arthur in becoming the king of legend. Advising him along the way is his guardian Gaius (Richard Wilson) and a dragon Kilgharrah (voiced by John Hurt); while King Uther (Anthony Head), and later Morgana (Katie McGrath) and Mordred (Alexander Vlahos), are his main hindrances. It differs from most interpretations of the King Arthur legend by making Merlin and Arthur (portrayed by Colin Morgan and Bradley James, respectively) contemporaries (Sherman 2015, 93) in a world where magic is outlawed. The resultant need for secrecy from Merlin became a central narrative drive throughout the series, with the character only revealing his true self to Arthur in the final episode—an eventuality anticipated from the pilot. For many fans, Merlin’s ‘magic reveal’ in the final episode invites comparison with coming out as homosexual, for it is only after revealing his true self to Arthur that the pair’s love for each other may be acknowledged. Queer viewers can easily identify with characters such as Mordred and Merlin, who keep their identities secret in fear of an unaccepting society, forming a “wishful identification” (see Hoffner and Buchanan 2005) with such characters’ struggle for acceptance and identity in a universe hostile to ‘their kind.’

The finale saw the death of King Arthur in the arms of his manservant, Merlin, an event that was foreshadowed from the first episode of the final season.[1] Arthur is slain by his former knight and surrogate son, Mordred, who feels betrayed by both Arthur and Merlin, two men that represent two sides of himself—Warrior and Magician—that he failed to reconcile. This essay’s queer reading of the Mordred character is from the position of an aca–fan (an academic and fan, see Brennan 2014b). It is written with the belief—put forth by Henry Jenkins in his seminal text on television fan cultures, Textual Poachers—that “speaking as a fan is a defensible position within the debates surrounding mass culture.” (1992, 23) To this end, I use fan readings of the series and analyse select photo montaged fan works (known as ‘slash manips’), including some from my own practice, to support my reading and delve into typologies that help examine the construction and journey of Mordred as the archetypal villain, as well as his multiple identities of knight and magician, and queer associations of his struggle for self.

Medieval (Homo)Eroticism, Queer Readings, and Slash Manips

Scholarship on the series, in the form of chapters in edited collections (see Elmes 2015; Meredith 2015) and journal articles (see Foster and Sherman 2015 for a special issue on the subject in Arthuriana), have begun to explore its significance. In particular, scholars have examined its representations and the value of its unique version of a legend that is broadly familiar to most viewers (Britons particularly). Such familiarity, as Jon Sherman points out, makes up much of Merlin’s appeal (2015, 97). Among this scholarship is my own article (see Brennan 2015), which performs a queer reading of the Lancelot character (the great Romantic archetype) as he appears in this BBC series and the works of Thomas Malory, T.H. White, and Marion Zimmer Bradley. In this recent article, I situate the popular series in the long heritage of Arthurian adaptation. The article also includes an examination of a tradition of using queer theory to analyse Arthurian texts (see Brennan 2015, 21–22). In particular, I explore the proposition by certain medievalists (see Burger and Kruger 2001; Zeikowitz 2003) that a ‘queer approach’ (see Halperin 1995) to texts of or set in the Middle Ages can be useful in making “intelligible expressions of same-sex desire.” (Brennan 2015, 21) The applicability of queer readings to this series is perhaps illustrated best by the fan followings it has inspired, which contribute to its status as a ‘cult text.’ (See Hills 2004 and his definition of cult television as a complex interaction among television texts, discourses about them, and the fan practices these texts inspire; also see Machat 2012, who examines Merlin fanfic trailers to explore how fans of the series remix the canon relationship of its male protagonists.)

Of particular relevance to a queer approach to television series such as Merlin are the products of ‘slash’ fans and their exploration of homoeroticism in popular texts, often of which lack representations of homosexuals (see Russ 1985; Bacon-Smith 1992; Jenkins 1992). Slash derives its name from the convention of using a forward slash (/) to designate sexual male pairings, such as ‘Arthur/Mordred’ (see Jones 2002, 80). Slash fans produce texts in the form of fiction, video, and art to depict their (often subversive) homoerotic readings. The attraction of Merlin to many slash fans can be read as a result of Merlin and Mordred’s secret identities as sorcerers in a world where the practice of sorcery is punishable by death. For many fans, magic here is a metaphor.[2] And when magic is read as a metaphor for homosexuality, as David M. Halperin reminds us, the term ‘queer’ becomes available: to “anyone who is or feels marginalized because of her or his sexual practices.” (1995, 62) I have examined the Merlin/Arthur pairing previously (see Brennan 2013) in an article that also introduces a form of slash that had at that time yet to receive scholarly attention, namely ‘slash manips.’ (See Brennan 2014a for more on the significance of slash manips with respect to how slash practice has been defined.)

Slash manips remix images from the source material (such as high resolution screen shots or promotional images from Merlin) with images from scenes selected from gay pornography. Most commonly, these works come in the form of two characters’ heads (often with expressions of exertion) digitally superimposed onto gay porn bodies (that generally match the physicality of the characters in question). It is a process I describe as the ‘semiotic significance of selection’ (see Brennan 2013). This present article includes analysis of select slash manips involving the Mordred character, all of which are reproduced here with the permission of the respective artists. The inclusion of these works is useful in the context of a queer reading of Mordred because the visual impact of these digital manipulations, in addition to complementing discussion of symbolism of certain scenes, also themselves are distinctly ‘queer.’ Such imagery is in of itself an embodiment of the “project of contestation” this is queering, in addition to helping disrupt “our assumptions about medieval culture and textual practices.” (Lochrie 1997, 180)

Reading Character: Mordred-as-Villain

In his seminal syntagmatic structural analysis of folklore, Morphology of the Folktale, Vladimir Propp (1958) develops a typology that identifies seven character types in folktales, each with a role to play in forwarding the narrative, namely: Villain, Donor, Helper, Princess, False Hero, Dispatcher, and Hero. By focalising the story through Merlin, two central heroes emerge in this retelling: Merlin and Arthur. (Ordinarily Merlin would be the ‘helper’ character type, the hero’s guide who prepares Arthur and provides him with magical assistance.) As my close reading will demonstrate, with Merlin-as-hero Mordred is consigned to the villain type, as he is never viewed by this character with anything other than suspicion of villainy; from the perspective of Arthur-as-hero, conversely, Mordred is a false hero, a character once viewed as good who becomes evil, much like the series’ other false hero, Morgana (known to legend as Morgan le Fay), who in this version of the legend, Mordred turns to after being betrayed by the heroes of the story. This essay explores how the heroes’ own categorising of Mordred’s character ensures his path as villain, as confirmed by Mordred’s final words to Arthur: “You gave me no choice.” (V.13 [abbreviated season and episode number]) This reading is similar to Mary Stewart’s 1983 novel, The Wicked Day, which retells the legend from Mordred’s perspective, portraying him sympathetically as a victim of circumstance and confirming that we are all the heroes of our own story.

Mordred as he appears in Merlin is fascinating not only because he is a villain of the series—and villains are often fascinating in queer readings—but further because he bridges the central characters of Merlin and Arthur, or ‘Merlin/Arthur,’ who are described in the series as “two sides of the same coin” (Kilgharrah, V.3). In a queer reading, Merlin (manservant)/Arthur (master) as two sides of the same coin create a binary chain of tails/heads, bottom/top, passive/active, sorcery/non-sorcery, intuition/rationality, magic/strength, feminine/masculine, homosexual/homosocial. Mordred as both sorcerer and knight, straddles these positions in Merlin, moving freely between them, which is in part why the titular character—with his intention to “Keep the magic secret” (a series tagline)—can only ever see Mordred as a threat. Conversely, to Mordred, Merlin represents someone with magic like himself. Someone who can help him negotiate his dual identity of knight/sorcerer. As this essay’s close reading of select episodes will reveal, by not trusting him, what Merlin ultimately denies Mordred (freedom to be himself), is also what he ultimately denies himself.

Reading Character: Mordred and the Magician/Warrior Archetype

William P. McFarland and Timothy R. McMahon (1999) employ the four masculine archetypes of King, Lover, Magician, and Warrior (see Moore 1991; Moore and Gillette 1990, 1992) to outline the respective benefits of each to homosexual identity development. The King archetype displays “qualities of order, of reasonable and rational patterning, of integration and integrity” (Moore and Gillette 1990, 62); the Lover is “deeply sensual, sensually aware, and sensitive to the physical world in all its splendor” (ibid., 121); the Magician bears the characteristics of “thoughtfulness, reflection, and introversion,” exhibiting “the ability to connect with inner truths” (McFarland and McMahon 1999, 51); and the Warrior incites others to “take the offensive and to move out of a defensive or holding position about life’s tasks and problems” (Moore and Gillette 1990, 79).

These archetypes are useful in introducing the characters of Mordred, Merlin, and Arthur, each of whom, in addition to being literal personifications of these archetypes, display a combination of the corresponding traits in their representation: Mordred (as Lover, as Magician, as Warrior), Merlin (as Lover, as Magician), and Arthur (as King, as Warrior). These archetypes are useful in plotting the binary of Arthur/Merlin, primarily King/Magician, and the manner in which Mordred belongs to both men, while ultimately struggling and eventually failing to exist in the grey area between the well-defined and policed binaries the men embody. For while being Magician and Lover affords Merlin (as Helper) attributes that Arthur both needs and does not possess himself (as King and Warrior; hence the earlier ‘coin’ metaphor), these are identities that Merlin conceals, that bring shame within the context of the series, for they also bear feminine (Lover) and queer (Magician) connotations; and thus Merlin is treated as such in the series, excluded from Arthur’s homosocial circle of knights, and ridiculed for his sensitivity, his lack of masculine worth—“Pathetic. You’re pretending to be a battle-hardened warrior, not a daffodil.” (Arthur to Merlin, I.2). By being King, Arthur “stabilizes chaotic emotion and out-of-control behaviors” (Moore and Gillette 1990, 62), he controls the unruly feminine, which is how sorcery is defined (and portrayed by Morgana), and thus needed to be outlawed, by the ultimate Father and King, Uther.

In this essay I examine the otherness of Mordred and how his pole personas of Warrior/Magician, knight/sorcerer, hero/villain, toy with Merlin and his efforts to maintain separation between such identities. In particular, I consider the Druid boy’s appearances over the final season of Merlin and his transition to Arthur’s favourite knight, as well as the fluidity and openness with which he occupies positions of otherness, as is supported by slash manips featuring the character. The essay also explores how Mordred subverts the homosocial order of Camelot in a way Merlin never could, eroticising the sacred bonds between Arthur and his men.

Arthur/Mordred: The Erotic Bonds of Heroes and Villains

Figure 1. The Arthur/Merlin/Mordred homosocial triangle (V.1).

Figure 1. The Arthur/Merlin/Mordred homosocial triangle (V.1).

Male heroes and villains of legend and myth share obsessive bonds and a covert homoeroticism (Battis 2006). The villain becomes obsessed with the hero’s body, “with finding his weakness, with penetrating or shattering or inflicting violence upon him” (ibid.). In his obsession, the villain becomes a “failed version” of the hero, needing to eradicate the hero to validate his own perverse ethical agenda, not just interested in ruling the world, but in “ruling the hero’s body” as well (ibid.). Writing here on the comic book tradition and the queer potential of the central antagonism of Clark Kent/Lex Luthor as they appear in the television series Smallville (2001–2011), Jes Battis’s description is also suited to the rivalry of Arthur/Morgana.[3] As villain and woman, Morgana seeks to disrupt and possess all that Arthur is—chivalric order, his reign, and his legacy—so as to impose her own worldview on the realm. “I want his annihilation, Mordred,” she tells him in V.2. “I want to put his head on a spike and I want to watch as the crows feast on his eyes.” While not homoerotic, there is a taboo eroticism inherit in Arthur/Morgana due to their blood relation, and the romantic references to the pairing in season one—such as in I.5, when Guinevere confides in Merlin that she hopes one day Arthur and Morgana will marry. Mordred, who responds to Morgana’s blood thirst by urging her to “calm yourself,” (V.2) is different. He is, in the end, fate’s and Morgana’s pawn—particularly when compared with other adaptations in which the character appears, T.H. White’s The Once and Future King particularly (see Thomas 1982). That Mordred’s villainy is an extension of Morgana’s perverse agenda is an idea put forward by Erin Chandler, who argues that at times (such as in season three):

the series focuses on Morgana playing what is essentially the legendary Mordred role, turning against her father, Uther, and everyone dear to him for his past actions and his refusal to acknowledge his errors. (2015, 109–10)

After all, while in Merlin Mordred may wield the sword that delivers the fatal blow, Morgana is the one who makes it unbeatable by forging it in dragon’s breath (Edwards 2015, 81).

Mordred’s portrayal as pawn explains why interest in the character from the perspective of slash fans seems to be less about his antagonism with Arthur—though there is certainly homoeroticism in that regard—and more about the love and devotion that turns sour and leads to respective betrayals of each other. Mordred defies Morgana at the start of season five, in fact wounds her in favour of Arthur’s vision of a nobler way, making the transition from Druid nomad to Arthur’s favourite knight in the space of a few episodes. As a man of magic, who also wishes to prove to Merlin his devotion to Arthur, the character self-sacrifices for the greater good until Arthur asks of him a sacrifice that is too much: to allow the woman he loves to be executed. To have done so, to have let the girl die, which would be to betray himself (the Lover). In the end Mordred is as betrayed by Arthur and Merlin (his mentor, his ‘helper,’ if you like) as he himself betrays. Until their mutual destruction he still desires Arthur, smiling when Arthur returns a mortal wound, welcoming the opportunity to join Arthur in death.

Mordred enacts a kind of homosocial, or ‘erotic’ (to appropriate Eve Kosofsky Sedgwick’s use of the term, see 1985) triangle with Arthur and Merlin, challenging Merlin and his decision to maintain secrecy. He also is endeared to Arthur, trusting him completely, a trust that is in his eyes betrayed; although there is more to it than that, Mordred has a part to play in Arthur’s fate. The triangle enacted by these men is visible from their first meeting as adults (V.1). In this scene (to be explored further in the next section), as Merlin recognises Mordred for the threat he is, an instant bond is formed between Arthur and his future knight (see Figure 1). Concerning the bond of Arthur and Mordred, there are traces of erotic connection between the men in the literature also. In Wilfred Campbell’s 1895 play Mordred: A Tragedy in Five Acts—in which the character is cast in the role of tragic anti-hero rather than villain—Mordred makes the point that Arthur’s affection for Launcelot “outweighs his affection for the queen, suggesting a possible homosexual subtext and therefore implicitly threatening Arthur with sexual blackmail.” (Yee 2014, 15) Benjamin Franklin Fisher IV takes this observation further when he suggests that Mordred’s suspicions in this play are not entirely unfounded; for, as Launcelot says, “I love thee, King, as doth no other man.” (1990, 171) The significance of such a suggestion of eroticism—whether valid or not—is that, as Pamela M. Yee argues: “the fact that Mordred introduces the possibility of inappropriate conduct between king and knight indicates that both he and Campbell are preoccupied with definitions of proper masculine behavior”. (2014, 16) In the second half of this essay, I will consider via close readings of episodes and analysis of slash manips, the ease with which Mordred negotiates and simultaneously inhabits dual positions—knight/sorcerer, hero/villain, lover/destroyer. A quality that renders him an intriguing and highly ‘slashable’ figure throughout the final season of the series, and a character that has something important to say about the villain’s journey.

V.1: Arthur’s Bane is Mordred’s Destiny

Figure 2. Mordred (V).

Figure 2. Mordred (V).

Mordred (portrayed by Asa Butterfield, I–II; Alexander Vlahos, V) is first introduced as a young Druid boy in three episodes over seasons one and two (I.8, II.3, and II.11). He is the first to call Merlin by his Druid name, ‘Emrys,’ and plays a crucial role in introducing Morgana to sorcery early in the series. He is saved initially when Arthur allows him to escape execution by Uther, an act of mercy that endears Arthur to the character and explains the bond they later share: Arthur does, in a way, give Mordred life. Kilgharrah the dragon prophetesses that the young Druid will bring about Arthur’s demise and therefore that Merlin “must let the boy die.” However it is only at the end of this episode (I.8) that viewers learn this character is in fact the Mordred of legend. As Sherman points out, in Merlin the plot device of “introducing a figure or object from Arthurian legend while withholding his, her, or its name” (as with Mordred, Geoffrey of Monmouth, and Excalibur, for example) is a pattern that is repeated throughout the series (see Sherman 2015, 91 and 94). Resultantly, when the character returns in season two, Merlin attempts unsuccessfully to have him captured, knowing he will be killed if he is. These are actions Mordred vows never to forgive and never to forget. He does not return again until the final season (V.1). Recast as an adult (the 24-year-old Vlahos, see Figure 2), he becomes a central character until the series’ end twelve episodes later (V.13). There is significance to be found in this recasting. For the Mordred of season five, while an adult, remains still somehow younger, more innocent, more easily corrupted than the other men who sit among Arthur’s ‘circle.’ He is also now at a suitable age to be ‘paired’ by slash fans with other adult males.

Mordred’s reintroduction comes while Merlin and Arthur are separated from the Knights of Camelot and being held as captives of slave traders. Mordred’s entrance is by way of intervention, preventing one of the men from killing Arthur: “Shouldn’t we leave it to the Lady Morgana to decide their fate?” Assisting Arthur up from the ground, their hands still clasped, Mordred says, “You don’t remember me do you? You saved my life once, many years ago.” The scene (see Figure 3) in which Arthur and Mordred first meet as adults is rich in visual symbolism. Mordred, with his black fur, clean appearance, and well-tailored-yet-exotic attire stands apart from the filthy brutes of the party he travels with. His pallid complexion, blue eyes, blood red lips, and black, curly hair makes him an alluring presence, set against a woodlands backdrop of lush greenery. All this contrasts with Arthur’s golden hair and reflective armour: he sits stark in the shot. Mordred’s appearance in furs and associations with the Druids make him almost wolf-like in appearance, a lone wolf boy with bushy fur and piercing eyes. Combined with the appearance of the character in Merlin’s dreams throughout the final season, such imagery is phallic and homoerotic, as Sigmund Freud’s psychoanalytic reading of the ‘Wolf Man’ myth reveals (see 1955). The ‘Wolf Man’—as Freud’s patient has come to be known—is a case that appeared in From the History of an Infantile Neurosis. It details “the primal scene,” the witnessing by a child of a sexual act. In this case from the 1910s the patient, a Russian aristocrat, has an anal fixation: a predilection for heterosexual relations in which he penetrates his partner from behind, and where he is unable to move his bowels without an enema administered by a male attendant. The patient has a recurring dream of a tree full of white wolves, which Freud relates to a time when, just age one-and-a-half, the patient was exposed to his parents having coitus a tergo (“from behind”), and thus a “repressed homosexual attitude” developed (Freud 1955, 64). As Lee Edelman writes, “the Wolf Man observed at first hand what being used from behind entailed.” (1991, 96) Edelman, in connecting the case with passages in texts that depict sodomy between men, argues that the Wolf Man case “carries more specifically the psychic inscription of the anal-erotic organization.” (98)[4] The erotic potential of Arthur and Mordred’s first adult meeting is explored in my 2013 slash manip, The Coming of Mordred (see Figure 4). The work employs binary symbolism of colour and physiology (gold/black, muscular/slight, hairless/hairy, light/dark) to represent the contrast in the Arthur/Mordred dynamic; while the connection of their bodies, their hands exploring each other’s naked flesh, foreshadows the (erotic) intimacy to follow. Like the base image onto which the characters have been placed, it is a work of foreplay.

Figure 3. Mordred and Arthur’s first meeting as adults (V.1).

Figure 3. Mordred and Arthur’s first meeting as adults (V.1).

There is an unkempt wildness to Mordred that resembles Morgana, a character who has undergone a transition from colourful and regel gowns (I–III) to black furs and unkempt sensuality (IV–V), from the warmth of the ward of Camelot to the icy climate of exile; a transformation from young and beautiful into the series’ main antagonist (Mediavilla 2015, 52), a transformation that coincides with her embracing sorcery. Cindy Mediavilla argues that the televisual format “presents many opportunities for characters to evolve from one season to the next.” (2015, 52) And that of all characters, “Morgana’s transformation is, by far, the most profound.” (ibid.) Making Morgana “one of the most complex and fascinating Arthurian characters depicted on television.” (ibid.) Further, summing up the connection between the journeys of Mordred and Morgana in the series, Elysse T. Meredith argues that in Merlin, “Mordred’s path is a rough reversal of Morgana’s.” (2015, 165) In many regards a resemblance in the evolution of these characters is fitting, especially given that in many retellings of the legend, Mordred is the unwanted son of Arthur and Morgana (Edwards 2015, 50). There is a quality of heightened sexuality signified by the appearances of the adult Mordred and season five’s Morgana, which ties the sorcerer with the sexual, and the taboo of magic with the taboo of unbridled sexuality, at odds with the chaste chivalric order of Arthurian knights.

In the first episode of season five, despite travelling with their captors, Mordred continues to protect Merlin and Arthur, even smuggling them food. And when the pair escape and Arthur is presented with the opportunity to kill Mordred, he restrains, “He showed us kindness.” When Mordred is reunited with Morgana, she is both delighted and surprised to see him alive. “Sorcery frightens people,” Mordred says, “even those who claim to support it.” He is of course speaking of Merlin, whose decision to keep his identity secret, Mordred never fully reconciles. “You see a lot,” Morgana replies. “I’ve learned to,” Mordred says. “I’ve had to. If I was not to be burned at the stake or exploited for another man’s gain.” We realise at this point that Mordred too has changed, he no longer associates with the Druids. He is an outcast, like Merlin, having to hide in plain sight to survive. We never learn why this is, the mystery of his background adding to the suspense of the character and his intentions. Morgana becomes hostile when Mordred informs her that they had Arthur in their grasp and that he escaped. She accuses Mordred of letting him go. Mordred is clearly taken aback by Morgana’s outburst and detailing of how she wishes for Arthur’s head on a spike. Their reunion is cut short when the alarm is sounded: Arthur has come to free his men.

Figure 4. The Coming of Mordred, Merlin/Mordred slash manip. By chewableprose.

Figure 4. The Coming of Mordred, Merlin/Mordred slash manip. By chewableprose.

While Morgana is successful in capturing Arthur, she is stopped from killing him by Mordred, who decides in a moment of intensity to change sides. It would seem that Arthur’s willingness to risk his life—“Had to free my men.”—inspires Mordred to literally stab his own kind in the back with a dagger. In the following scene, a confused Merlin asks the Diamair—the key to all knowledge—“If Mordred is not Arthur’s bane than who is?”, to which the Diamair replies, “Himself.” This is Arthur’s betrayal of Mordred to which I earlier referred. Mordred does, by all appearances, change sides; however it is Arthur’s later decisions that ultimately lead Mordred to double cross him, decisions ‘helped’ by Merlin. Mordred returns to Camelot and is knighted. In the scene following, Merlin offers to remove his cape, and queries Mordred’s defection:

MERLIN          You saved Arthur’s life, why?

MORDRED Because Arthur is right, the love that binds us is more important than the power we wield. Morgana had forgotten that.

Merlin disrobing Mordred is a titillating sight for slash fans. It connotes a changed dynamic for the former rivals. While Mordred was previously an outsider and Merlin had Arthur’s ear, now Mordred is granted access to Arthur’s inner circle. Merlin is now subservient to Sir Mordred, and must interact with him accordingly. Such is the symbolism attached to the removal of the ceremonial cape. Yet there is also subterfuge in the scene. Merlin veils a threat of exposure through the line, “if Arthur knew.” A threat that is of course empty, as Mordred holds the same damning knowledge over Merlin. Theirs is a stalemate. Merlin resists the shift of power, the subtext of this scene being his jealousy.

Mordred and Merlin are “not so different,” as Mordred identifies earlier in the episode. His rationale for turning on Morgana bears uncanny semblance to a scene from the previous season (IV.6), when a captured Merlin accuses Morgana of knowing nothing of loyalty, caring only for power. Also, they both keep their magical identities hidden from Arthur. This essay suggests that Merlin’s suspicion of Mordred is misplaced, and in fact helps ensure his eventual betrayal (as is argued below in regard to the events of V.5). As the focal character of the series, Merlin’s suspicion—however unwarranted—manifests itself in slash art that exploits the potential power, symbolic and supernatural, Mordred has to control Merlin. My 2013 slash manip Like A Beast is a case in point. In the work, I exploit the derogatory connotations of the ‘doggystyle’ position (of being fucked “from behind,” to refer back to the Wolf Man myth) and signifieds of dispassionate, focused, in control (Mordred) versus shocked, overwhelmed, distant (Merlin) in my selection of facial expressions. Merlin’s expression in particular evokes all the passivity, phallus-accommodating, and penetrative potential of the toothless, gaping mouths of side show carnival clowns ready for ball play. Such imagery is also supported by Merlin’s performance in the series of a medieval fool.[5] The Camelot banner and digitally-engorged scrotums combined with the ‘movement’ of the sexual position—Mordred employing elements of the ‘leap frog’ doggystyle variant, ‘balls deep’ inside Merlin—helps convey my intended subversion of Merlin, the power afforded to Sir Mordred, and the fallacy of his knighthood, which is built on a lie and a constant ‘threat-of-outing’ game with Merlin.

Other artists have also explored the new power differential between Merlin and Mordred, and further, the new affordances with Arthur that come as a result of Mordred’s knighthood. In an untitled 2014 work by wishfulcelebfak, who posts his works to LiveJournal, Mordred sits on Arthur’s cock (perhaps symbolic of a throne). In text accompanying the work, the artist situates the image:

Arthur (bradley james) helps druid Mordred (alexander vlahos) come out of his shell, by introducing him to “knights of the round table” aka sex buddy club.

Morgana can only offer Mordred some cheap magic tricks and a wooden dildo, but Arthur can offer him unlimited gay sex with all the hunks of the kingdom. Which side will Mordred choose? (wishfulcelebfak 2014)

Expressed in the above are the benefits that come with Mordred’s inclusion in the Knights of the Round Table, including certain ‘homosocial rituals,’ which wishfulcelebfak has (homo)sexualised. The work of Ruth Mazo Karras is useful here, her 2002 From Boys to Men, for example, examines formations of masculinity in late medieval Europe through a queer reading of the bonds that ignite among knights. The message of this manip is just how much Arthur has to offer.

Similarly, a 2012 work titled Breaking in a New Knights by endless_paths, also a LiveJournal artist, depicts Arthur entering Mordred ‘from behind.’ The accompanying text: “Who needs merlin when you have knights” (endless_paths 2012a), makes clear the role (once occupied by Merlin) that Mordred now fills; or in the context of the sexual act depicted, the willingness of Mordred to provide a ‘space’ for Arthur to fill. The artist implies that Mordred’s hole is more compatible with the cock of a king than that offered by his manservant. This implication is in much the same spirit as the erotic rituals that may have taken place between knights, such as bathing in front of each other to verify health and masculinity, as recounted in the 1300s by French knight Geoffroi de Charny in his Book of Chivalry (as noted by Zeikowitz 2003, 64–65; Zeikowitz also details intimate interactions between knights in Geoffrey Chaucer’s Troilus and Criseyde and Sir Gawain and the Green Knight, author unknown). Concerning the erotic rituals of Arthur and his knights, Mordred specifically, my 2014 slash manip It’s Good to be Bad, describes just such a ritual:

Mordred knew it was wrong that, when the other knights were not looking and the Queen was away, he would get down on his knees in the grass in that private spot behind the castle and take Arthur’s manhood in his mouth, and keep it there until the King moaned, withdrew and showered him with his seed. Mordred knew it was bad to be so suggestive in front of the others in gesturing for his King to repeat the ritual more and more, but such dangerous displays were also what made it feel so good (chewableprose 2014)

In Arthur’s eyes Merlin and Mordred are entirely different (a theme explored in endless_paths’s manip): one is brave and noble and knightly, the other a friend and manservant yes, but not possessing qualities necessary to be a knight. Mordred is given recognition and place at Arthur’s right side, which is everything prophesised, but not realised, about Merlin and Arthur’s relationship. In Kilgharrah’s words to Merlin: “The Druid boy, his fate, and Arthur’s are bound together like ivy around a tree.” (V.3) While the legend is clear about the significance of such a statement, in Merlin there is the implication that it is the character Merlin’s unwillingness to trust Mordred’s sincerity that in part ensures Arthur’s grim fate. That Merlin may have had a role to play in the death of Arthur is supported by Chandler, who argues that in Merlin, and indeed much of the literature on which it is based, there is no single contributing factor in Arthur’s downfall (2015, 110). As Gaius, Merlin’s most trusted friend, tells him: “People change, perhaps you should give [Mordred] the benefit of the doubt.” (V.2) Merlin never does.

Seeking a Father, Seeking a Son: Arthur and Mordred’s Search for Each Other

Etymologically Mordred is Latin and means “painful,” an apt descriptor for a character difficult to watch. From a slash perspective, he is painful because he had so much promise. The promise was despite the character’s “weight of history,” a phrase used by series co-producer Julian Murphy to explain certain inevitable conclusions to the series (see Brennan 2015, 37; also see Sherman 2015, 83 who discusses audience expectations around Arthurian retellings). Being introduced as an adolescent to the ‘of age’ Merlin and Arthur early in the series, understanding Mordred’s portrayal relies on remembering that he is much younger than contemporaries Merlin and Arthur—easy to forget given that Alexander Vlahos, the actor recast in the role, is aged within two years of Merlin actor Colin Morgan. In the legend the character is often Arthur’s illegitimate son (to Morgause in Malory and White, and to Morgana in Bradley’s 1982 The Mists of Avalon), which perhaps explains Arthur’s father-like devotion, and Morgana’s protectiveness in this version of the story. Mordred wishes to please Arthur, and when that fails, repurposes this wish for Morgana. He gives up Merlin’s secret identity late in the final season (V.11) as demonstration of his devotion to Morgana’s cause, committing himself to the destruction of his father-figure, and the Law-of-the-Father (see Lacan 1977, 67).

The Oedipal potential of the Arthur/Mordred/Morgana relationship is plain to see, and has been noted by scholars (see Worthington 2002) in their readings of other iterations of the Arthurian legend. In renouncing Arthur and turning to the ‘dark side’ (see Figure 5) Mordred also foregoes all knightly, chivalric artifice. He embraces the sorcerer, traitor, feminine side of the binaries he once moved between. Keeping in mind Mordred’s age and his search for guardianship, before shifting sides, Arthur and Merlin emerge as two potential surrogate fathers, the erotic potential of which is as pronounced in Merlin as it is in the incestuous unions that spawned Mordred in many other adaptations (most notably in Malory). Mordred’s search for a father is met with Arthur’s search for a son and heir and is most evident in V.5. It is a search at odds with Merlin’s own quest to prove himself to Arthur, the tragedy of which rings true when we consider that Arthur dies before producing an heir.

Figure 11. Left–right: Mordred in service to Arthur; Mordred in service to Morgana (V).

Figure 5. Left–right: Mordred in service to Arthur; Mordred in service to Morgana (V).

In a scene from V.5 that follows a training session, Arthur makes clear to Merlin his intention to mentor Mordred, and speaks with an admiration and pride he does not of any of his other knights. Mordred’s prowess with a sword confirms how little we know of his life in the intermediary years since we last saw him. Where did he learn to fight in a manner that would impress the king? Furthering the surrogate father metaphor, Mordred is half Merlin, half Arthur, he has both of their skills and the potential to become the best of both men.[6] Mordred reaches out to both men, and while Arthur reciprocates Mordred’s love, Merlin shuns it. This is despite Gaius’s—Merlin’s own father-figure—efforts to convince Merlin that Mordred will not necessarily betray Arthur:

The future has many paths, that is only one. […] Seeing’s not the same as knowing, and we must know before we act.

In this episode Merlin acts before he knows, seizing an opportunity to ensure Mordred dies, actions that in fact ensure Mordred’s survival and the continuation of the prophecy of ‘Arthur’s bane.’

V.5: “I Cannot Save the Life of a Man Destined to Kill Arthur”

Arthur displays his faith in Mordred by inviting him on a routine patrol of the woods surrounding Camelot. Merlin objects in an early scene that labours his inability to afford Mordred the opportunity to prove himself, suggesting yet again that there could have been a very different outcome for all concerned if he had. The purpose of the patrol is to confront a rogue sorcerer, Osgar, who when confronted presents Arthur with a relic of the ‘Old Religion.’ Such relics and reference to magic as an ‘Old Religion’ adds to the mysticism of magic as it is represented in the series (via glowing eyes, potions, collection of herbs for poultices, etc.). Naturally, given his unsuperstitious nature and traits of King and Warrior (Moore and Gillette 1990, 62, 79), Arthur is not too concerned. The sorcerer dies from wounds sustained in his confrontation with the patrol and is buried in secret by Merlin. Mordred notices:

MORDRED What would the king say? Sorcerers are not permitted marked graves. It’s all right, Merlin, I’d have done the same. He was one of us, after all.

MERLIN          It won’t always be like this. One day we’ll live in freedom again.

MORDRED You really believe that?

MERLIN          I do.

MORDRED Until then, we go unmarked in death as in life.

It is their first scene alone since Merlin disrobed Mordred following his knighting. And Mordred begins as Merlin had before, with a veiled threat of exposure. Before the sorcerer Osgar had died he had told Arthur there was still time to find his “true path.” This warning mirrors Gaius’s “many paths” comment to Merlin. Kilgharrah confirms this later in the episode when he tells Merlin: “The future is never clear, there are many paths, they do not all lead to Camelot’s ruin.” It follows, therefore, that not all paths lead to Mordred’s villainy. Within Merlin, Mordred is seeking someone with whom he can confide, someone with magic like himself who can help him negotiate his dual identity. This is what Merlin ultimately denies him, and himself. Merlin is so used to keeping his identities separate, he is unable to understand Mordred, a man who refuses to give up on others knowing that side of himself. That becomes clear in this scene as Mordred seeks surety that he will not always have to hide who he is. In the end, it is Morgana who gives him this certainty of self. In the episode, Gaius convinces Arthur to investigate the relic, a journey that takes them to the White Mountains and the dwelling of the ‘Disir,’ representatives of the Old Religion (all women). When conflict inevitably follows, Mordred is gravely wounded while protecting Arthur. Mordred’s only hope for survival is Merlin’s magic, which Merlin will not use because of fear of who Mordred will become. Gaius rightly notes that letting someone die based on a prophecy of what they may one day do is out of character for Merlin. Interestingly, this scene is similar to the scene between Arthur and Morgana in V.1 that convinced Mordred to change sides:

ARTHUR          What happened to you, Morgana? As a child, you were so kind, so compassionate.

MORGANA      I grew up.

Merlin remains committed to his decision to let Mordred die for the greater good, as the experience of ‘growing up’ has taught him. This is perhaps where Mordred’s youth, as a man yet to ‘grow up’ and thus in need of guidance and understanding, becomes significant. Believing it his only recourse, Arthur returns with Merlin to the Disir, prepared to lay down his life for Mordred’s. The Disir tell Arthur he must embrace magic, and is given the night to decide. “My heart says do anything I can to save Mordred,” Arthur says to Merlin that night by campfire, a recurrent setting of intimacy and phallic symbolism (“tongues of flame” [Freud 1930, 37]) for the men. “But I have seen what misery unfettered sorcery brings. Before my father outlawed magic, Camelot was almost destroyed by sorcery. In my own time, Morgana has used it for nothing but evil. What would you do? In my place?” Arthur seriously considers the prospect that magic may not be as evil as his father thought, and even if it is, seems prepared to accept that threat in exchange for Mordred’s life. He asks Merlin for his advice on what he thinks they should do: “So what should we do? Accept magic? Or let Mordred die?” Merlin chooses the latter, and seals the fate of both men: “There can be no place for magic in Camelot.”

Arthur tells the Disir of his decision, returning with a heavy heart to Camelot. When he arrives he is delighted to discover that Mordred is alive and well, Mordred running and embracing Arthur. Merlin then realises in a scene with Gaius that by influencing Arthur not to allow magic to return to the realm, he had ensured Mordred’s path to bring about Arthur’s death:

MERLIN          How could I have been so stupid?

GAIUS             You did what you thought was best.

MERLIN          I assumed the best way to protect Arthur was to kill Mordred.

GAIUS             A perfectly natural assumption.

MERLIN          But all I did was make sure he lived. That was the Disir’s judgment. Mordred’s life is Arthur’s punishment for rejecting magic.

GAIUS             You mustn’t blame yourself.

MERLIN          But it is my fault. Mordred is alive and well. He’s free to play his part in Arthur’s death and there’s nothing I can do to prevent it. Nothing.

I am inclined to disagree with Merlin’s logic, as expressed in the above dialogue. Given reference in this episode to the many paths of fate, and the Disir’s promise to spare Mordred’s life should Arthur accept magic, it seems more plausible that it is not Mordred’s life that is punishment, but rather forthcoming catalysts—namely the character Kara—that will lead Mordred to stray onto a different path. Merlin is right in so far as this cannot now be prevented; the sentence has been passed: Arthur will die at Mordred’s hand, and Merlin ensured it. This reasoning makes sense when considered in relation to a key fan criticism (see Caspers 2013) of Merlin ending when it does, which is that the prophecy of Merlin and Arthur side-by-side, uniting the lands of Albion and returning magic to the realm is never realised. It would seem this is the hero’s critical mistake. As Gaius words it, Merlin did what he thought was ‘best,’ but not what was ‘right.’ As Arthur prophetically told Merlin in V.1: “No matter what adversity we face, we stand for what is right. To betray our beliefs, Merlin, that is what would destroy everything we strive for.”

This is the tragedy of this particular retelling. By betraying the beliefs that Arthur and Merlin had lived by, and that had seen them escape certain death many times previous, Merlin had ensured Arthur’s destruction. This point also explains another fan criticism of the plotting of the final episode (see Caspers 2013), which is that Arthur and Merlin had survived worse in the past. This time was different, this time Arthur’s fate was decided in advance. The earlier scene where Mordred doubts whether magic will ever not be outlawed lends further credence to the argument that had Arthur chosen Mordred’s life over his decree, Mordred would not need to go on “unmarked in death as in life.” The episode ends with Arthur with his arms around Mordred, hoisting him into the air (see Figure 6), it serves as grim reminder—for Arthur/Mordred shippers[7] particularly—of what might have been.

Figure 6 Arthur hoists Mordred into the air in a playful embrace (V.5).

Figure 6. Arthur hoists Mordred into the air in a playful embrace (V.5).

V.9: “Three’s Better than Two, Isn’t That Right, Merlin?”

Mordred continues to reach out to Merlin in the lead-up to the cataclysmic event that reroutes him onto the path of Arthur’s destruction. And Arthur continues to treat Mordred like a son. The events of V.9 are a good illustration of this. In the plot for this episode, Mordred and Leon are the only knights Arthur trusts with information of a plan intended to disrupt potential leaks in the ranks. The episode is the final in the ‘evil!Guinevere trilogy,’ in which Guinevere is enchanted to serve Morgana, and in it Merlin and Arthur set out with an unconscious Guinevere to meet ‘The Dolma,’ a mysterious elderly female sorcerer, in hopes of a cure. Mordred, having noticed Merlin acting strangely, follows them. It is just as well he does too, coming to the rescue when a cliff fall leaves Merlin unconscious and Arthur pinned beneath a boulder. Mordred is praised that evening around a campfire: that site of homoerotic significance. There, sitting around erect flames, Arthur makes reference to the triangle Mordred effects in the Arthur/Merlin dynamic: “Good to have you with us. Three’s better than two, isn’t that right, Merlin?” That evening, Mordred once again confronts Merlin, expressing a desire for amicable relations between them:

MORDRED You don’t trust me do you, Merlin?

MERLIN          I believe you to be a fine knight.

MORDRED But not one to be trusted. It’s all right, I know you have the king’s best interests at heart. I only wish you would believe that I do too. One day I shall prove my loyalty to you and the king. Then I hope we may be friends.

MERLIN          I would wish for nothing more.

When an attack from Morgana renders Mordred unconscious, Merlin convinces Arthur to leave him for dead. Yet another refusal by Merlin to believe in Mordred, which in turn facilitates Morgana and Mordred’s first meeting since his defection:

MORDRED Why don’t you kill me?

MORGANA      My argument’s not with you, Mordred. How could it be? We’re of a kind.

MORDRED Never.

MORGANA      You wear the uniform well but we both know what lies beneath. Do you think Arthur would tolerate you for one minute if he knew the truth? One of his knights, a sorcerer.

MORDRED One day he will know. One day we will be accepted.

MORGANA      Your naïveté would be charming if it wasn’t so dangerous.

Mordred defeats Morgana using magic, his eyes glowing gold: symbolising the fire Morgana has ignited within (see Figure 7); ambers of doubt—and of Camelot’s destruction, as the prophecy goes—are being fanned, which again would not have been the case had Arthur embraced magic in V.5. At the episode’s end Mordred reveals that he had known the mysterious sorceress Arthur had gone to meet was in fact Merlin, and vows to keep his secret yet again, to trust that Merlin’s intentions are just: “Have no fear. I will not divulge your secret. I admire you. It can’t be easy to do so much for so little reward.” This episode and the meeting with Morgana marks the beginning of the end.

Figure 13. Mordred defeats Morgana using magic (V.9).

Figure 7. Mordred defeats Morgana using magic (V.9).

V.11: “You’re Breaking His Heart. You’ll Lose His Trust”

Arthur’s sentence—to die at the hands of a Druid—begins with Mordred’s betrayal in V.11 and is complete only two episodes later. In V.11 Mordred (as Lover) shelters a childhood friend and implied lover, Kara, who is subsequently captured and sentenced to death after killing several of Arthur’s men and making an attempt on Arthur’s life. Mordred pleads with Arthur on Kara’s behalf for clemency, weeps and kneels before him, “I beg you, Arthur.” Arthur is moved by the display and responds in a father-like manner: “You know there’s nothing I wouldn’t do for you.” Yet refuses to yield the sentence, for she is a danger to his people. Merlin watches these events unfold with great interest, well aware of what is a stake, and pleads to Arthur on Mordred’s behalf:

MERLIN          You’re breaking his heart. You’ll lose his trust.

ARTHUR          There’s nothing I can do. In time Mordred will understand that. He’ll come to forgive me.

MERLIN          I fear you’re wrong, Arthur.

Kara exploits Mordred’s feelings for her, poisoning him against Arthur to further her own cause against Uther’s doctrine: “No matter what he preaches, he is no different from his father.” Mordred resolves to free Kara and smuggle her out of Camelot. However before he does, he returns to Arthur to apologise for what he is about to do, and to say goodbye: “You took me in. I will always remember that, and everything you’ve done for me.” Recognising Mordred’s speech for what it is, Merlin confronts Mordred and his intention to free Kara. Mordred warns Merlin not to betray his trust. “Tell me you wouldn’t do the same for the woman you love,” Mordred says. “You see, you can’t.” When Merlin discusses the situation with Gaius, he is reminded that what Mordred is planning: “It’s nothing you haven’t done yourself a hundred times before.” And yet, as Merlin has always done, he applies a double standard where Mordred is concerned, betraying his trust and telling Arthur of Mordred’s intentions. It is one final failure on Merlin’s behalf to choose another path for Mordred, the man who so admires him.

Mordred and Kara are captured in the woods beyond the castle, Kara having killed a guard during the escape. They are imprisoned, Kara’s sentence standing and Mordred’s pending. Merlin makes another attempt to persuade Arthur to free Kara. And it works. The next morning, in the throne room before all of the court Arthur offers Kara a chance: “If you repent your crimes, I will spare your life.” Arthur’s love for Mordred is such that he would betray his own beliefs—allowing a sorcerer and killer to go free—if it will mean winning back Mordred’s favour. Slash manip artist endless-paths speculates on Arthur’s devotion and the seductiveness of the Mordred character in a 2012 Arthur/Mordred manip titled A Knight Doing His Duty. In a brief statement accompanying the work and setting up the action depicted, endless_paths writes: “Sometimes the power of a sorcerer is to [sic] much to resist.” (2012b) The manip configures the two in the missionary position and is set in Arthur’s chambers, two qualities that connote intimacy and familiarity between the pair: they have done this before. In line with the ‘semiotic significance of selection’ (Brennan 2013) in the work, Mordred, as you would expect, is slighter in stature, while Arthur is particularly limber. In a plank position, Mordred folds Arthur’s knees back and by his sides, elevating his arse for deeper penetration. Arthur’s arms reclined behind his head; his toes pointed and clenched; and his chin pressed to his chest allowing for full view of Mordred’s cock entering him: Arthur is entirely committed to the act and maximising the full range of his penetrator’s motion. Both men have relaxed expressions and line of sight to each other.

Despite Arthur’s best efforts to
alleviate tensions with Mordred via an offer of clemency, Kara
remains resolute: “You deserve everything that’s coming to you, Arthur Pendragon.” Mordred never learns of Arthur’s offer to pardon Kara. In a state of acute grief, Mordred uses magic to free himself following her execution (see Figure 8) and travels to Morgana directly, to whom he reveals that the identity of the man who had been stalking her dreams, Emrys, is none other than Arthur’s manservant, Merlin. Once again, connection can be made here between Mordred and Morgana’s journeys to villainy, in particular this critical episode and its sequence of events, which can be compared with a storyline from season one. As Jennifer C. Edwards explains, after witnessing Uther’s resolve to execute a man of magic (Alvarr in I.12) who had provided her with comfort, “Morgana changes from a loving ward to a treacherous rebel and even goes so far as to plot Uther’s death.” (2015, 51) A similar fate befalls Mordred here, whose “betrayal of Arthur results not from inherent malevolence but from the death of his childhood sweetheart.” (Meredith 2015, 165)

Figure 14. In a state of grief, Mordred uses magic to set himself free from his cell and from Arthur (V.11).

Figure 8. In a state of grief, Mordred uses magic to set himself free from his cell and from Arthur (V.11).

Conclusion

Reflecting on her experience of the aftermath of a public execution of a criminal during a residence in Scandinavia, Mary Wollstonecraft (1802) writes:

[…] executions, far from being useful examples to the survivors, have, I am persuaded, a quite contrary effect, by hardening the heart they ought to terrify. Besides, the fear of an ignominious death, I believe, never deterred any one from the commission of a crime; because, in committing it, the mind is roused to activity about present circumstances. It is a game of hazard, at which all expect the turn of the die in their own favour; never reflecting on the chance of ruin, till it comes. In fact, from what I saw, in the fortresses of Norway, I am more and more convinced that the same energy of character, which renders a man a daring villain, would have rendered him useful to society, had that society been well organized. (208)

Wollstonecraft’s reflection is resonant with the execution of Kara, which is the catalyst for spurring Mordred the Lover to betray and destroy his King. In her critique of the spectacle of the public execution, Wollstonecraft makes the case that villainy is not innate, but rather due to some external, societal failure. Such an observation is comparable with my argument in this essay about the Mordred character, that great archetype of the treacherous villain. That the societal failure of a pre-unified Albion, in which magic is banned and Merlin the Magician feels the need to hide himself, is what leads Mordred onto his villainous path. This reading offers insight into the popular reimagining of an iconic villain, as well as the various types and queer metaphors the character’s journey in this popular series illuminates and rouses within the minds of fans. The inclusion in this essay of works by slash manip artists both demonstrate the appeal of a queer reading of the Mordred character, while also supporting broader queer readings of Merlin as a program full of homoerotic potential.

T.H. White’s adaptation of the Arthurian legend has been read by some scholars as an allegory to the horrors of the Second World War. In it Mordred is a Hitlerian character. He turns to new technology to bring about a ‘New Order’ (1958, 620–21). If Hitler sought to destroy civilisation; in White, by valorising power above honour, Mordred destroys chivalry (Thomas 1982, 50). In Merlin, Mordred is more a pawn of fate than an agent of destruction; he carries out Arthur’s sentence from the Triple Goddess (V.5) under Morgana’s—High Priestess of the Triple Goddess—instruction. He stands as example of the dire consequences of secrecy. Merlin’s unwillingness to trust him, and resolve to remain closeted about his secret identity, seals Mordred and Arthur’s fate of mutual destruction. When Mordred strikes the fatal blow in V.13, he says to Arthur: “You gave me no choice.” When Arthur returns with a fatal strike of his own, Mordred smiles, he will not go into death unmarked or alone.

 

References

Aronstein, Susan. An Introduction to British Arthurian Narrative. Gainesville: University Press of Florida, 2012.

Bacon-Smith, Camille. Enterprising Women: Television Fandom and the Creation of Popular Myth. Philadelphia: University of Pennsylvania Press, 1992.

Battis, Jes. “The Kryptonite Closet: Silence and Queer Secrecy in Smallville.” Jump Cut: A Review of Contemporary Media, no. 48 (2006).

Bradley, Marion Zimmer. The Mists of Avalon. New York: Random House, 1982.

Brennan, Joseph. “Slash Manips: Remixing Popular Media with Gay Pornography.” M/C Journal: A Journal of Media and Culture 16.4 (2013).

Brennan, Joseph. “Not ‘From My Hot Little Ovaries’: How Slash Manips Pierce Reductive Assumptions.” Continuum: Journal of Media & Cultural Studies 28.2 (2014a): 247–64.

Brennan, Joseph. “The Fannish Parergon: Aca–Fandom and the Decentred Canon.” Australasian Journal of Popular Culture 3.2 (2014b): 217–32.

Brennan, Joseph. “‘You Could Shame the Great Arthur Himself’: A Queer Reading of Lancelot from BBC’s Merlin with Respect to the Character in Malory, White, and Bradley.” Arthuriana 25.2 (2015): 20–43.

Burger, Glenn, and Steven F. Kruger. “Introduction.” In Queering the Middle Ages, edited by Glenn Burger and Steven F. Kruger, i–xxiii. Minneapolis: University of Minnesota Press, 2001.

Caspers, Kirsten. “Why Merlin Should Return to the Screen,” Den of Geek!, February 13, 2013, accessed September 3, 2015.

Chandler, Erin. “Pendragons at the Chopping Block: Elements of Sir Gawain and the Green Knight in the BBC’s Merlin.” Arthuriana 25.1 (2015): 101–112.

chewableprose. “Chewyboys It’s Good to be Bad,” LiveJournal, July 8, 2014, accessed September 8, 2015.

Doyle, Arthur Conan. “The Final Problem.” In his The Memoirs of Sherlock Holmes, 256–79. London: George Newnes, 1894.

Edelman, Lee. “Seeing Things: Representation, the Scene of Surveillance, and the Spectacle of Gay Male Sex.” In Inside/Out: Lesbian Theories, Gay Theories, edited by Diana Fuss, 93–118. New York: Routledge, 1991.

Edwards, Jennifer C. “Casting, Plotting, and Enchanting: Arthurian Women in Starz’s Camelot and the BBC’s Merlin.” Arthuriana 25.1 (2015): 57–81.

Elmes, Melissa Ridley. “Episodic Arthur: Merlin, Camelot, and the visual Modernization of the Medieval Romance Tradition.” In The Middle Ages on Television: Critical Essays, edited by Meriem Pages and Karolyn Kinane, 99–121. Jefferson: McFarland, 2015.

endless_paths. “17th December: Breaking in a New Knights,” LiveJournal, December 17, 2012a, accessed September 7, 2015.

endless_paths. “A Knight Doing His Duty,” LiveJournal, December 31, 2012b, accessed September 7, 2015.

Fisher IV, Benjamin Franklin. “King Arthur Plays from the 1890s.” Victorian Poetry 28.3–4 (1990): 153–76.

Foster, Tara, and Jon Sherman, eds., “King Arthur in the Twenty-First Century: Kaamelott, BBC’s Merlin, and Starz’s Camelot.” [Special Issue] Arthuriana 25.1 (2015).

Freud, Sigmund. Civilization and its Discontents. Translated by James Strachey. New York: W.W. Norton and Company, 1930.

Freud, Sigmund. “From the History of an Infantile Neurosis.” In The Standard Edition of the Complete Psychological Works of Sigmund Freud 17, edited by James Strachey, 1–124. London: The Hogarth Press, 1955.

Halperin, David M. Saint Foucault: Towards a Gay Hagiography. Oxford: Oxford University Press, 1995.

Hills, Matt. “Defining Cult TV: Texts, Inter-texts, and Fan Audiences.” In The Television Studies Reader, edited by Robert Clyde Allen and Annette Hill, 509–23. London: Routledge, 2004.

Hoffner, Cynthia, and Martha Buchanan. “Young Adults’ Wishful Identification with Television Characters: The Role of Perceived Similarity and Character Attributes.” Media Psychology 7.4 (2005): 325–51.

Jenkins, Henry. Textual Poachers: Television Fans and Participatory Culture. London: Routledge, 1992.

Jones, Sara Gwenllian. “The Sex Lives of Cult Television Characters.” Screen 43.1 (2002): 79–90.

Karras, Ruth Mazo. From Boys to Men: Formations of Masculinity in Late Medieval Europe. Philadelphia: University of Pennsylvania Press, 2002.

Lacan, Jacques. Ecrits: A Selection. London: W.W. Norton & Company, 1977.

Lochrie, Karma. “Mystical Acts, Queer Tendencies.” In Constructing Medieval Sexuality, edited by Karma Lochrie, James Schultz, and Peggy McCracken, 180–200. Minneapolis: University of Minnesota Press, 1997.

Machat, Sibylle. “‘Prince Arthur Spotted Exiting Buckingham Palace!’: The Re-Imagined Worlds of Fanfic Trailers.” In Film Remakes, Adaptations and Fan Productions: Remake/Remodel, edited by Kathleen Loock and Constantine Verevis, 197–214. Basingstoke: Palgrave Macmillan, 2012.

Malory, Thomas. Le Morte d’Arthur. London: William Caxton, 1485.

McFarland, William P., and Timothy R. McMahon. “Male Archetypes as Resources for Homosexual Identity Development in Gay Men.” The Journal of Humanistic Counseling, Education and Development 38.1 (1999): 47–60.

Mediavilla, Cindy. “From ‘Unthinking Stereotype’ to Fearless Antagonist: The Evolution of Morgan le Fay on Television.” Arthuriana 25.1 (2015): 44–56.

Meredith, Elysse T. “Gendering Morals, Magic and Medievalism in the BBC’s Merlin.” In The Middle Ages on Television: Critical Essays, edited by Meriem Pages and Karolyn Kinane, 158–73. Jefferson: McFarland, 2015.

Moore, Robert L. The Magician and the Analyst: Ritual, Sacred Space, and Psychotherapy. Chicago: Council of Societies for the Study of Religion, 1991.

Moore, Robert L., and Douglas Gillette. King, Warrior, Magician, Lover: Rediscovering the Archetypes of the Mature Masculine. San Francisco: HarperSanFrancisco, 1990.

Moore, Robert L., and Douglas Gillette. The King Within: Accessing the King in the Male Psyche. New York: William Morrow & Co., 1992.

Padva, Gilad. “Dreamboys, Meatmen and Werewolves: Visualizing Erotic Identities in All-Male Comic Strips.” Sexualities 8.5 (2005): 587–99.

Propp, Vladimir. Morphology of the Folktale. Volume 9. London: American Folklore Society, 1958.

Russ, Joanna. “Pornography by Women for Women, with Love.” In her Magic Mommas, Trembling Sisters, Puritans and Perverts: Feminist Essays, 79–99. Trumansburg: Crossing Press, 1985.

Scodari, Christine, and Jenna L. Felder. “Creating a Pocket Universe: ‘Shippers,’ Fan Fiction, and The XFiles Online.” Communication Studies 51.3 (2000): 238–57.

Sedgwick, Eve Kosofsky. “Gender Asymmetry and Erotic Triangles.” In her Between Men: English Literature and Male Homosocial Desire, 21–27. New York: Columbia University Press, 1985.

Sherman, Jon. “Source, Authority, and Audience in the BBC’s Merlin.” Arthuriana 25.1 (2015): 82–100.

Stewart, Mary. The Wicked Day. London: Hodder & Stoughton, 1983.

Stoker, Bram. Dracula. London: Archibald Constable and Company, 1897.

Sutton, John William. “Mordred’s End: A Reevaluation of Mordred’s Death Scene in the Alliterative Morte Arthure.” The Chaucer Review, 37.3 (2003): 280–5.

Thomas, Jimmie Elaine. “The Once and Present King: A Study of the World View Revealed in Contemporary Arthurian Adaptations.” Ph.D. Thesis, University of Arkansas, 1982.

Tollerton, David C. “Multiculturalism, Diversity, and Religious Tolerance in Modern Britain and the BBC’s Merlin.” Arthuriana 25.1 (2015): 113–27.

White, T.H. The Once and Future King. New York: GP Putnam’s Sons, 1958.

wishfulcelebfak. “Hugedicklover’s Top 10 Gift – Random Merlin Hunks Fucking,” LiveJournal, November 9, 2014, accessed September 7, 2015.

Wollstonecraft, Mary. “Letter XIX.” In her Letters Written During A Short Residence in Sweden, Norway, and Denmark, 207–18. Second Edition. London: G. Woodfall, 1802.

Worthington, Heather. “From Children’s Story to Adult Fiction: T.H. White’s The Once and Future King.” Arthuriana 12.2 (2002): 97–119.

Yee, Pamela M. “Re-presenting Mordred: Three Plays of 1895.” Arthuriana 24.4 (2014): 3–32.

Zeikowitz, Richard E. Homoeroticism and Chivalry: Discourses of Male Same-Sex Desire in the 14th Century. Basingstoke: Palgrave Macmillan, 2003.

 

Notes

[1] During Arthur’s quest to save his knights from Morgana in V.1, Merlin encounters a Druid seer who tells him of ‘Arthur’s bane,’ the prophecy of Arthur’s death at the hands of a Druid (Mordred). Merlin is told: “Now more than ever it is you and you alone that can keep Arthur safe.” It sets a sinister tone for the final season. Coupled with the season’s tagline “The die is cast,” it suggests that Arthur’s death is an inescapable destiny, which ushers back to season one’s tagline, “You can’t escape destiny.”

[2] See Tollerton 2015, who discusses the “freer hand” Merlin has “to gesture toward modern concerns and make ethical judgements on issues of diversity and society.” (123)

[3] Not surprising, given that the format of Smallville (depicting Clark Kent before he became Superman) served as principal inspiration for Merlin (Brennan 2015, 39).

[4] Also see Padva 2005, who uses Freud’s reading of the homoerotic symbolism in the wolf dream to read a gay male comic, Jon Macy’s ‘Tail.’

[5] In a scene from V.1, Arthur delights in the opportunity to humiliate Merlin, forcing him to juggle for the entertainment of Queen Annis and her guests.

[6] Producing offspring based on a digital composite of two male faces is a popular practice among digital slash artists.

[7] A ‘shipper’ is a fan who wishes for a particular pairing to share a romantic relationship (see Scodari and Felder 2000).

 

Bio:
Joseph Brennan
is a sessional lecturer in the Department of Media and Communications at the University of Sydney, where he was recently awarded his Ph.D. His doctoral work involved textual analysis of photo-montaged fan works inspired by BBC’s Merlin. Known as ‘slash manips,’ in these photo remixes fans layer images of male characters from popular media with gay, and often pornographic, material. He argues that these works are of scholarly interest because they have something to tell us about sex and bodies, about the divides we erect within male sexuality, between popular and pornographic, homosocial and homosexual, the implied and the explicit. He was Teaching Fellow at the University of Sydney, 2012–2013, and a critic with Australian Art Review, 2008–2013.


Days of YouTube-ing Days of Heaven: Participatory Culture and the Fan Trailer – Kyle R. McDaniel

$
0
0

Abstract: This study analyzes the aesthetic content and user-generated feedback of fan-appropriated film trailers exhibited in on the Internet. The aim of this research is to gauge participatory culture’s involvement in the transformation of promoting archival motion pictures on the Internet. This research study looks to fan trailers as unique media entities that exist as visually empowered narratives created through specific acts of fandom. Specifically, this study investigates the audiovisual and discursive elements of competing trailers for Terrence Malick’s Days of Heaven (1978). The findings suggest that fan trailers are capable of generating myth and nostalgia for aging motion picture properties through user-generated acts. The broader goal of this project is to understand the relationship between participatory film cultures and studio-controlled motion picture content available on video streaming and sharing media channels.


Fig. 1: The memorable and Biblically referential swarm of locusts in the film Days of Heaven.

Fig. 1: The memorable and Biblically referential swarm of locusts in the film Days of Heaven.

Introduction: Trailers at a Turning Point

A YouTube video by an unknown director can suddenly blow up on the marketplace, and there will be three studios bidding for it. (Without having yet met the director!)…Maybe execs are busy watching YouTube instead of hearing pitches. Our work is virtual.   

-Lynda Obst, Sleepless in Hollywood (2013, 27).

In April 2014, an online user released a high-definition film trailer on YouTube for David Fincher’s forthcoming thriller, Gone Girl (YouTube 2014a). Several hours after the trailer’s debut, an impressive 186,000 fans had accessed the content with 276 of that number contributing written feedback to the message board on the webpage. While film fans were sharing interest and excitement for the trailer on YouTube, News Corp., the media entity that financed Gone Girl through 20th Century Fox, perceived a threat of digital piracy. The following day, the conglomerate removed the trailer and the fan commentary. In the absence of this content, News Corp. left a statement reading, “FOX has blocked [the trailer] on copyright grounds” (YouTube 2014b). This incident is representative of the contemporary state of affairs between media conglomerates with a controlling interest in motion pictures and film fans in online spaces. The presence of film trailers on the Internet presents a specific set of issues for both parties as well, especially in relation to film marketing and promotion, in addition to content ownership and control over copyright.

This study engages with how film fans interact with once-profitable motion picture properties through fan trailers on the Internet. Here, the fan trailer is defined as the act of re-editing and re-exhibiting abridged film content through online channels. Fan trailers are realized through specific and largely collective acts of user-participation, and have the potential to revitalize interest in aging film properties. This article explores the audiovisual and content-related aspects of fan trailers in comparison to a distributor-owned trailer for Days of Heaven (Terrence Malick 1978). Furthermore, the feedback or commentary on message boards is also investigated as part of this research project to locate how such discourse speaks to the collective memory of Hollywood archives. In order to understand the issues surrounding the emergence and popularity of the broad spectrum of Internet trailers, this study looks to literature on the relationship between the evolution and of fan involvement with digital cinema and new media, as well as scholarship on the history of film trailers and film promotion and advertising. The findings from this article suggest that fan trailers play a crucial role in continuing the lifespan of aging Hollywood properties or archival films. The proliferation of fan trailers through video streaming and sharing websites as well as the message board commentary suggests that fan participation is instrumental to building relationships between film and viewer. In turn, participatory cultures that interact with older film titles in online channels incorporate aspects of their public and private selves as part of this creative process. The following research questions are designed to further explore this relationship between film fans inhabiting online spaces and the evolving state of fan trailers in digital cinema: What are the content-related (i.e., audiovisual) similarities and differences between the distributor-controlled, official trailer and the fan trailers under study? And what role(s) does user-generated commentary or feedback play for these trailers?

Fig. 2: A black-and-white still of Terrence Malick on the set of the film.

Fig. 2: A black-and-white still of Terrence Malick on the set of the film.

Film Promotion in the Digital Age: New Strategies, New Rules

For much of the 2000s, Hollywood was reluctant to promote film content through online channels for fear of losing theatrical and home video revenue (Sickels 2011a). The film industry seemed confused by the ever-growing presence of the Internet and related online technologies for film exhibition. But to effectively reach a global audience, the studios and their parent media conglomerates were eventually forced to adapt to the changing media landscape. As Sickels (2011) stated: “Deals with Netflix and the like are only going to delay the inevitable…Audiences don’t want to wait, and they certainly won’t when their only reason for having to do so is an artificial time structure concocted by the studios…”(145). By the second decade of the century, the industry’s fears had become a reality, with on-demand film and television viewing radically altering the industry.

Scholars have pointed to the different complexities of film marketing in the digital age and the associated challenges for the U.S. film industry (e.g., Cunningham and Silver 2013). In Perren’s (2010) words, “A wide range of economic, cultural, political, and formal factors are at play; different entities have distinctive stakes in online distribution” (77).  In other words, films with a greater potential to appeal to a global audience receive preferential treatment from media conglomerates, as well as promoters, marketers, and distributors. With video-on-demand (VOD) revenue climbing steadily since 2010, the studios are looking to different methods for advertising motion pictures beyond the more traditional formats, which includes one-sheets of film posters and theatrical and television spots (Roxborough 2013). Film trailers on the Internet are a viable option in this evolving landscape. The Internet Movie Database and YouTube are the most frequently visited websites supporting online film trailers, with both entities supporting numerous trailers for new releases and older Hollywood titles. In effect, the spectrum of film trailers on the Internet presents a number of potential issues for the film industry. Trailers, historically controlled by studios for advertising and publicity purposes, are increasingly pirated by outside entities. One scholar argues that film industry insiders are the ones largely responsible for leaking studio-controlled content online, with the availability of illegal anti-encryption and watermarking software to bypass copyright restrictions playing a role as well (Bettig 2008, 200-201). Since the release of the DVD De-Content Scramble System (DeCSS) in 2002, film content has been descrambled and decoded for public access and use, despite the studios efforts to control motion picture content (Litman 2002).

Film fans, however, have argued that such laws overwhelmingly favor those with a financial stake in motion picture properties, thereby inhibiting individual and collective acts of creative expression (Boyle 2008). As such, studio-backed restrictions have resulted in more frequently cited instances of pirated motion pictures as well as an upsurge in websites devoted to streaming and downloading studio-owned film content (Sterbenz 2014). Scholars and journalists reporting on the film industry have addressed some of these issues in relation to film trailers. For instance, Rothman (2014) discussed how theatrical trailer standardization discourages user interactivity. Tolson (2010) reported that fan participation with film content suggests an increase in technological “play” that disrupts the traditional model of media production to consumption. Others have looked at how trailer “mobility” is encouraged in a cross-platform media environment, and the effects of contemporary trailer length and message on the viewer (see Franich 2013; Johnston 2008). While many of the issues surrounding film promotion in online spaces remain unanswered, trailers continue to serve as a primary marketing tool for motion picture studios and their parent conglomerates. Fan involvement with film trailers is a burgeoning area of contemporary film marketing and new media, but scholarship on this subject is lacking. Therefor, how participatory cultures connect to older film titles in online spaces through the fan trailer remains an unexplored avenue of study for cinema and media scholars.

Fig. 3: The film’s main titles are appropriately positioned in the concluding seconds of the Paramount Movie’s YouTube-exhibited trailer.

Fig. 3: The film’s main titles are appropriately positioned in the concluding seconds of the Paramount Movie’s YouTube-exhibited trailer.

Trailers in Transition: A Brief History and Contemporary Definitions

The most time-honored marketing strategy for film promotion is the movie trailer, commonly referred to as the “preview.” Kernan (2009) traced the genealogy of film trailers to 1919, citing the National Screen Service (NSS) as the first unified company responsible for creating these advertising spots. The author asserts that the evolution of the film industry during the 20th century affected changes in the types of motion pictures produced, thereby altering the aesthetics and meta-messages of trailers in the ensuing decades. A transition in film marketing occurred during the 1970s, and then again in the 1980s, with a rise in independent filmmaking, an upsurge of art-house theaters, and eventually, the summer blockbuster. During these decades, films trailers debuted on network television in thirty-second spots, visually supported by moments lifted from the film, and complete with the now-familiar and once-prominent voice-of-God narration. By the contemporary era, trailers had become “unique form[s] of narrative film exhibition, wherein promotional discourse and narrative pleasure are conjoined (whether happily or not)” (Kernan 2009b, 1). In essence, this period saw the rise of distinct promotional film advertisements alongside the audience’s familiarity and ability to detect such media forms.

Scholars regard the modern film trailer as both complex and historically shifting media type. A leading scholar on the history and transition of motion picture trailers suggests that these forms are specifically targeted, easily recognizable visual media that are created to capture, direct, and guide viewer attention (Wyatt 1994). Today, both media entities and online film fans aid in determining trailer standards and audiovisual elements. Trailers are guided by audiovisual messages through structured narratives to connect with the largest number of viewers through multi-platform distribution. Some have argued that film trailers in the digital era are defined by their dynamic if fleeting presence, asserting that contemporary trailers are forced to compete with other media forms to encourage audience-driven participation or feedback (see Rombes 2009a). Smartphones and digital tablets indicate an increase in trailer mobility and interactivity on behalf of audiences, who are receiving different media in shorter, eye-catching bursts (Grainge 2011).  Scholars have also argued that the efforts of fans on the Internet extend film capital beyond traditional home video or cable and network replay through film mashups or distributing abridged content (e.g., Sickels 2011c; Hoyt 2010a). Tyron (2009) traced the inception of the digital movie trailer to a fan preview for The Shining (Stanley Kubrick 1980) that gained Internet traction the same year as the inception of YouTube. According to the author, the fan trailer was an outgrowth of DVD culture “that allowed viewers to recognize that texts were ready to be ripped apart and reassembled in playful new ways” (151). Lazzarato (2006) described these types of fan creations as influential because they are “activities involved in defining and fixing cultural and artistic standards, fashions, tastes, consumer norms, and, more strategically, public opinion” (132). In sum, film fans use popular film properties to engage with and further promote such content to a wider range of consumers.

Re-appropriating and exhibiting film content is oftentimes understood as a group effort. Rose (2012a) argues that the cyclical discourse that occurs in online social networks encourages is what engages users to interact with film properties. Citing Avatar (James Cameron 2009) and The Lord of the Rings (Peter Jackson 2001-2003) trilogy as examples, the author maintains that a strong and relatable narrative or story is of the key to fan involvement. According to Rose, online visual narratives must be able to entertain as well as challenge participant-viewers, thereby encouraging individuals to take part in the creative act (233). Through user-participation and online media channels, the modern film trailer appears in transition. In an environment increasingly dominated by new media platforms and social networking, video-sharing websites are stimulating the development of relationships among social actors.

Defining Participatory Cultures and Digital Cinema

Participation raises the question of whose story is it? And, the answer I think is, it’s all of ours. In order to really identify with the story, in some way we have to make it our own.

-Frank Rose, The Art of Immersion (2012b).

­

Online users are now affecting many aspects of the motion picture industry and most recently, have turned to collaborative involvement with film trailers. Through an increasing number of video streaming and sharing websites, fans are producing and exhibiting short and hybrid motion picture forms from existing film content. Jenkins (1992) defined networked individuals who engage with and repurpose existing media materials as members of participatory cultures. These persons “speak from a position of collective identity, forge an alliance with a community of others in defense of tastes which…cannot be read as totally aberrant or idiosyncratic” (23). The author attributed the roots of this phenomenon to fan communities that built up around popular television programs, such as Star Trek, and who communicated and bonded through sharing information at conventions and fan clubs. More recently, Jenkins (2006a) has adapted his definition to include new media and social networking. Although optimistic about the endeavors of participatory cultures, Jenkins has noted the drawbacks of these communities as well, including the shifting power dynamics of group members and the involvement of corporate entities. In addition, the author has described the illegal activities of some members of participatory cultures, specifically those parties who undermine media conglomerates through acts of digital piracy and copyright infringement. Jenkins (2006b) has also commented on the burgeoning relationship between participatory cultures and digital cinema:

[I see] media fans as active participants…seeing their cultural products as an important aspect of the digital cinema movement. If many advocates of digital cinema have sought to democratize the means of cultural production and distribution to a broader segment of the general public then the rapid proliferation of fan-produced Star Wars films may represent a significant early success story for that movement (551-552).

In other words, the upsurge in digital cinema is dependent on fans in much the same way that fans are dependent upon interacting with cinematic creations. Digital cinema, as such, is oftentimes described as an outgrowth of online fan participation. Rombes (2009b) claims that collective acts of nostalgia, personal expression, and the adaptation of new technologies play a role in shaping digital cinema. Beginning with the rise of digital video and cinematography in the mid-1990s, the author discusses an additional factor in the relationship between digital cinema and the actions of participatory cultures: “There is a tendency in digital media – and cinema especially – to reassert imperfection, flaws, an aura of human mistakes to counterbalance the logic of perfection that pervades the digital” (Rombes 2009c, 2). In consideration with Rose’s (2012) insistence on powerful storytelling, Rombes argues that digital cinematic forms are generated and desirable because of factors such as pixilation and noise, which appear to mirror human imperfections. While fan intervention in existing film content raises questions for the future of digital cinema and a general understanding of what constitutes motion picture archives, participatory cultures have contributed to film marketing and promotion since the late 1990s. According to Erickson (2009a), who studied Internet film campaigns for The Blair Witch Project (Daniel Myrick and Eduardo Sanchez 1999) and others, studios appropriate fan-based advertising strategies if fan efforts prove financially successful.  This article is concerned with how participatory cultures repurpose and interact with the content of older motion picture titles. The entrance of fan trailers through online video streaming platforms suggests new territory for digital cinema, as well as the possible extension of the lifespan for archived film properties.

Fig. 4: A still image from the opening titles of a student-generated video essay for Days.

Fig. 4: A still image from the opening titles of a student-generated video essay for Days.

Case Study Film: Days of Heaven

Since it was first released, “Days of Heaven” has gathered legends to itself…[it] is above all one of the most beautiful films ever made. Malick’s purpose is not to tell a story of melodrama, but one of loss. His tone is elegiac. He evokes the loneliness and beauty of the limitless Texas prairie.

-Roger Ebert (1997a).

In the contemporary media marketplace, conglomerates and studios overseeing film distribution and exhibition pay close attention to the role of technologies in film promotion and branding. This is also true when considering how older film titles are released, with potential revenue gained from cable and network television broadcasts, DVD rentals and sales and most recently, VOD. Those with a financial stake in film archives oftentimes publicize and rerelease only a select number of dated film titles per year, with those properties having the most commercial potential regarded as particularly valuable on the marketplace. While some noteworthy and popular motion picture titles are available for little-to-no pay through video-sharing online services, media conglomerates use Netflix, Hulu, Amazon, and iTunes, for instance, to promote their most commercially viable films. It is here that the role of participatory culture and the evolution of the fan trailer in the archival value of film properties must be taken into consideration. Days of Heaven is significant because of its longstanding popularity amongst fans, its continual re-emergence in the public arena, and its location in cinematic history. Malick’s film arrived at a turning point in the New Hollywood of the 1970s. The competition between fledgling studio productions and a burgeoning independent film movement marked much of the decade’s releases (see Thompson and Bordwell 2010; Biskind 1998, et. al.). “But by the late 1970s,” Thomson (2012) writes, “there began to be fewer grown-up pictures meant to disturb and provoke” (459).

Before and after its release, Days of Heaven was considered an oddity for Paramount Pictures, a none-too-profitable feature that rested on the short reputation of its filmmaker.[1] Malick spent his early years in Hollywood penning several projects for other directors until his first feature-length film, Badlands (Terrence Malick 1973), gained traction from both audiences and critics, garnering a reputation as the second Bonnie and Clyde (Arthur Penn 1967). Patterson (2007) said Malick’s film offered the director the chance to “work outside more conventional parameters” (28). The filmmaker’s follow-up, however, was grander in scope and presented to audiences as a thematic American period piece. Set in the Great Plains of the 1910s, the narrative focused on a romantic amongst two migrant workers and a land baron. Morrison and Schur (2003) described Days as “wed[ding] Whitman’s poetic ideal of the democratic vista to the interior landscapes of Henry James, with a plot that evokes The Wings of the Dove and ends with a quasi-biblical plague of locusts” (23) [Fig. 1]. Indeed, the locusts were memorable, as was a lengthy scene in which wildfire spreads rapidly across the grasslands, scorching a vast swath of farmland. But much of the film’s storyline involved the happenings of Malick’s starring quartet – Richard Gere, Brooke Adams, Linda Manz, and Sam Shepard – with the characters’ muted emotions drawn out in close-ups paired with character voiceover.

Fig. 5: Gere and Adams’s characters traveling atop a railcar with other migrants in the film.

Fig. 5: Gere and Adams’s characters traveling atop a railcar with other migrants in the film.

Much of the film’s legend was only realizable years after its release. For one, Malick departed from filmmaking for two decades after Days, leaving a questionable legacy for a motion picture whose long-term stability rested on the director’s reputation and the film’s much-discussed cinematography. Over time, those perfectly composed images of man and nature, or what Kehr (2011) glowingly referred to as, “aesthetic shock effects [that] create vast, harmonious wholes,” were responsible for keeping the film in the minds of journalists and cinephiles (23-24). The film’s cinematography eventually became something of Hollywood lore [Fig. 2]. Ebert (1997b) detailed the infighting between credited director of photography, Nestor Almendros, and his predecessor, the notoriously cantankerous Haskell Wexler, in his “Great Movie” review of the film. Over the years, rumblings over credit for the look and feel of the film have led to a reconsideration of the man responsible for capturing such well-regarded images. In the years since its release, Malick returned to filmmaking and has garnered generally favorable reviews and some commercial success.[2] No fewer than ten book-length volumes are dedicated to the filmmaker’s resurgence, including The Terrence Malick Handbook (Smith 2012), and a number of academic and trade journal entries have surfaced on the canonical worthiness of Days (e.g., Crofts 2001; Woessner 2011; Koehler 2013, et. al.). Not surprisingly, praise and frustration for the film reigns on the Internet as well. The number and popularity of video clips available on video streaming and sharing websites suggests additional enforcement of the scholarly and journalistic discourse devoted to the film as well. While Days remains a much-debated and discussed film more than 35 years after its theatrical release, the role of trailers for the film on the Internet deserves attention in the era of cross-platform film promotion.

Selection of Trailer Case Studies: The Presence of Days of Heaven Online

The “Paramount Movies” channel on YouTube, overseen by Viacom, offers an original trailers for Days of Heaven [Fig. 3]. The Criterion Collection, responsible for marketing and distributing the Blu-ray and HD-DVD versions of the film, also displays an official trailer on its homepage for Days.[3] Mysteriously, Paramount’s trailer has received few visitors on YouTube while Criterion’s showcases an impressive 153 user-generated comments. The seeming randomness of attracting viewers to trailer content in online spaces is represented in this brief comparison, which appears to crossover to fan trailers as well (YouTube 2014c; The Criterion Collection 2014). The volume and popularity of fan trailers and video clips of Days showcased on YouTube overshadows this corporately controlled material in several ways as well. For one, the power of the video sharing website’s status as a social networking outlet is immediately evident. The “WorleyClarence” YouTube channel, for instance, has reposted an official version of Paramount’s trailer with an astonishing 360,000 views and 97 message board posts.[4] “JokerTreePictures,” described as an umbrella channel for three student filmmakers, has created a seven-minute video essay for Days that has gathered significant attention [Fig. 4]. Another YouTube user offers a promotional video compiled from scenes from Days matched with the music of Rod Stewart’s pop single, “Broken Arrow.” The sum of this content, which includes fan-exhibited interviews with the cast and crew as well as scenes lifted from the film, is evidence of the film’s presence on the Internet (YouTube 2014d).

For this study, three trailers were chosen as individual case studies based on the following criteria: 1) the recognizable differences in their audiovisual content, 2) the number of online views (i.e., “hits”), and 3) the number of message board posts or available online feedback. Two fan-appropriated trailers exhibited on YouTube were selected based on these requirements, as was the aforementioned trailer available through The Criterion Collection. The necessity of the trailer selection process was to compare and contrast elements of fan trailers with an official trailer approved by a media outlet in an effort to answer the research questions for this study. Many trailers that did not meet the research criteria were not selected because of factors such as conflicting content with the selected trailers, a lack of available user-generated discourse on message boards, and/or the number of recorded views or hits online. After completing the selection process, trailers were coded A (“WorleyClarence” YouTube Channel), B (“cnharrison” YouTube Channel), and C (The Criterion Collection), respectively. The researcher conducted individual and comparative audiovisual analyses on trailers A, B, and C and made notes on narrative structure and trailer content. This was followed by a qualitative content analysis of the online commentary or feedback on the message boards for each trailer’s webpage. In effect, the trailer selection process and resulting analyses were guided by the research questions for this study: What are the content-related (i.e., audiovisual) similarities and differences between the distributor-controlled, official trailer and the fan trailers under study? And what role(s) does user-generated commentary or feedback play for these trailers?

YouTube. 2008. “Days Of Heaven – Trailer (1978).” Last modified April 17, 2008.

YouTube. 2013. “Days of Heaven – Trailer.” Last modified on April 13, 2013.

YouTube. 2013. “Days of Heaven–A Video Essay.” Last modified on October 16, 2013.

Fig. 6: Adams and Shepard photographed in silhouette, with the symbolic farmhouse looming in the background.

Fig. 6: Adams and Shepard photographed in silhouette, with the symbolic farmhouse looming in the background.

Days of (Online) Fan Trailer Heaven

Trailer A opens with an image of Paramount Pictures’ trademark logo. The studio’s signature emblem fades into an image of brooding clouds looming over a wind-worn prairie. Thunder bellows on the soundtrack, and a shot of a bird of prey morphs into a backlit figure of a man standing in the grasslands at sunset. “In 1916, America was changing,” the narrator says in the trailer’s opening seconds. An image of a railcar passing over a bridge fades into a scene of factory workers digging through heaps of coal, followed by another wide frame of an empty sunbaked wheat field. The viewer is then swept into close-ups of the rough-hewn faces of the film’s stars – Gere, Shepard, and Adams – amidst passing railcars and horse-drawn carriages en route to the barren frontier [Fig. 5]. One minute and fifteen seconds into Trailer A, the serene mood and tone of the narrative changes abruptly. The narrator’s voice states that the film is “the story of a man who had nothing…the woman who loved him…and the man who would give her everything for a share of that love” (YouTube 2014e). With these words, the imagery moves away from the thematic scope of the land and its inhabitants and into the romantic dilemma at the heart of the film. A scene in which Gere’s field hand runs from law enforcement on horseback is juxtaposed with a quieter moment of his character embracing Adams in a quiet meadow. The next shot is an extreme close-up of Shepard’s watchful gaze, as if overseeing these scenes from afar.

As the narrative for Trailer A moves towards its conclusion, Adams and Shepard are photographed in silhouette inside the latter’s large estate, while the bedraggled face of Gere’s character peers up at the duo through a windowpane from below. This moment is framed from Gere’s perspective, with the actor and the encompassing field bathed in the deep blues of a Midwestern dusk, suggesting the loneliness his character will face with the coming of night. The film’s title appears over this closing shot, foreshadowing a troubled outcome for the trio. Trailer A presents much of the entire film’s narrative in under two minutes; what begins as a broad glimpse of turn-of-the-century westward expansion in the U.S. evolves into a minor tale of lost love [Fig. 5]. Thematically, the trailer’s primary audiovisual message suggests a heightening of nostalgia for both the American West and the Hollywood of the late 1970s, with the mythic qualities of innocence and utopia highlighted in the cinematography and production design [Fig. 6]. The professionalism of the editing in Trailer A, including the pairing of shots and sequence evolution provides a seamless story arc. Thus, the inclusion of Paramount’s introductory logo, the ‘70s-era voice-of-God narration, and the production elements suggests that this user-exhibited fan trailer was re-appropriated without revising the original trailer’s content. Therefore, Trailer A is most likely an original trailer for the film repurposed by one or more online fans. Trailer B also provides a visually compelling narrative to signal nostalgia and romanticism for the American West. But here, the viewer is immediately transplanted into to the lives of the film’s primary characters without the broader introduction of the land and its inhabitants as witnessed in Trailer A [Fig. 7].

Fig. 7: The film’s use of natural light to emphasize dramatic elements is also highlighted within the trailers.

Fig. 7: The film’s use of natural light to emphasize dramatic elements is also highlighted within the trailers.

The opening shot in Trailer B, a striking low-angle image of Gere, Adams, and the younger Manz running to catch a moving train, introduces the film’s predominant family dynamic.[5] Next is a shot of moving railcars topped with migrant travelers that segue into multiple close-ups of these characters’ hardened faces. Already, the viewer is guided toward the themes of travel and migration. The following image shows the Gere, Adams, and Manz trio atop one of the railcars, amidst the masses, fleeing the East for better opportunities. The rest of Trailer B’s running time focuses on the romantic triangle that ensues. Several important elements in Trailer B suggest a greater degree of user- repurposing. Manz’s tinny backwoods drawl, taken from the film’s narration, guides the trailer’s audio track for much of the running time, and is backed by a second musical track of delicately plucked guitar strings. In addition, the caption for Trailer B, located just below the video player on YouTube, states, “Bill, Abby, and sis arrive on the panhandle,” a sentiment only marginally correlated with the majority of the trailer’s visual narrative (YouTube 2014f). Another item that speaks to user re-appropriation is the individual shot duration, which moves at a more leisurely pace here, and seems to have been edited mostly to match Manz’s voiceover.

Further suggestive of fan involvement with Trailer B’s content is the abrupt segue from Manz’s voice and the guitar string audio tracks to the ambient sounds of trotting horses and rolling wagon wheels. Visually, the nonprofessional editing is emphasized at this point as well, with a sequence in which Gere’s character is propositioned for work by a land baron, a moment that is abruptly interrupted by a long shot of migrants moving en mass across the prairie. Throughout the two and a half-minute running time for Trailer B, the mood and tone shift in favor of different scenes from the film that drive the trailer towards a questionable conclusion. Marketing and film promotion is immediately evident on the webpage for Trailer C [Fig. 8]. The Criterion Collection offers viewers the option of purchasing several DVD versions of the film, reading a written essay on the film’s historical significance, a list of DVD special features, and links to related films from the company in addition to the trailer.

The trailer itself, however, is constructed from film content not included in Trailers A and B. In this much-abridged version, the guitar audio track preceding Manz’s narration is audibly fragmented and disassociated from any cohesive visible narrative. As such, the film’s primary visual content is made up of close-ups of the nondescript faces of migrants overlooking a land of grazing crows and antelope on the abandoned prairie. Here, Manz’s brief narration serves to introduce the film’s quiet mood and leisurely pacing. The aforementioned scene of Gere interacting with the land baron is cut prematurely in Trailer C, presumably for purposes of keeping the trailer’s length under the running time of one minute. In this version, the scene that introduces the bullhorn-gripping farm owner is interrupted by an establishing crane shot that places the viewer in the midst of migrants scampering towards the opportunity of work. Each of these moments take up several seconds worth of running time, and Criterion’s trailer closes abruptly with a surprising fade-to-black.

Fig. 11: Criterion’s webpage for Days of Heaven offers visitors a number of options to interact with the film.

Fig. 11: Criterion’s webpage for Days of Heaven offers visitors a number of options to interact with the film.

Whereas the finales of both fan-appropriated trailers on YouTube are classically structured to mirror the resolutions found in many trailers of the 1970s, the transition to a black frame in Trailer C suggests a different kind of closure. The trailer concludes by returning to a still frame of six farmhands standing in awe of an insect downpour, a somewhat iconic image from the famous “locust scene” in the film. This visual placeholder is representative of Criterion’s idyllic version of the film’s significance. As such, this striking still image speaks directly to curating the memory of Days, arguably more so than the totality of the narrative for Trailer C. Although the design of the distributor’s webpage is simultaneously content-heavy and visually arresting, this emblematic still frame stands apart, begging the visitor to click, watch or re-watch and possibly, purchase the film from the distributor.

Feedback on Heaven: The Online Discourse of Cinematic Aesthetics & Nostalgia

The contents of three hundred user-generated message board posts for Trailers A, B, and C were analyzed for this study. Most of this feedback was found to be praiseworthy of Days, with many of the user-posts lauding the film’s cinematography. The discourse on Criterion’s webpage for the film was overwhelmingly positive and found to reflect the distributor’s marketing intentions. “A beautiful spectral and view of the early 1900s mid-western America,” Mike Santoro wrote on the message board. “I love Malick’s brilliant direction in this [film]” (The Criterion Collection 2014b). Others commentators on this webpage used specific discourse that intertwined aspects of their real-world lives with the film’s history and nostalgia. “My first Malick movie, discovered when I was watching every movie on rogerebert.com’s ‘101 Movies To See Before You Die,’” Taylor P. stated. Bennett Duckworth wrote, “…thanks Dad for introducing this movie to me.” And mimicking Manz’s drawl in the character’s narration, Arthur Mhoyan said, “There were people sufferin’ in pain and hunger. Some people their tongues were hangin’ out of their mouths” (The Criterion Collection 2014c).

While single-word and somewhat elusive statements, such as “Breathtaking” and “Beautiful,” were found on the Criterion message board as well, much of the feedback was more detailed and descriptive. The lack of negative comments on the message board is further indicative of Criterion’s approach to online publicity and distribution for the film. In turn, the majority of user-feedback for Trailers A and B on YouTube was specifically targeted at the film’s cinematography. Equal parts excitement and praise for the film’s imagery was evident on both message boards, suggesting that the film’s visual approach is endorsed through fan-recall on these video-streaming webpages. For example, the “GregF” channel wrote, “…all 5 [of] Malick’s movies are beautiful but there are no words to describe Days Of Heaven…pure magic.” The “44eelz” channel posted, “i haven’t seen this movie yet but the cinematography looks amazing.” The “ErikHutt” channel added that “[Days] was shot in Alberta,” and the “MrKeepitunderyourhat” channel said, “To be honest, I’d say that the most famous aspect of the entire film is its magic hour cinematography” (YouTube 2014g).

The similarities in the content and tone of the statements analyzed across all three webpages suggest that fans are fond of the film’s historical significance and imagery. The cause-effect nature of this discourse also acts as an effort to keep the film in memory while promoting it to others. The content of this rhetoric also signifies the film’s ability to evoke an era in Hollywood history in which aesthetic power swayed and captivated audience members. In sum, much of this online discourse speaks to how film fans in online spaces curate the myth and nostalgia of aging mainstream film properties. Much of these statements reflect a sincere familiarity with Malick’s production design and the aesthetic properties of the cinematography. The statements under analysis, therefore, speak to the role of message boards in film advertising as well as the intricacies of fan-generated promotional feedback.

Promoting Hollywood Through the Fan Trailers: The Archive in Transit

YouTube. 2015. “Honest Trailers.” Accessed February 11, 2015.

This article investigated how participatory cultures use fan trailers to engage with aging Hollywood titles in online spaces. The findings suggest that online film fans utilize fan trailers to interact with others while drawing attention to archival film properties. In effect, the findings from this study demonstrate several ways in which trailer repurposing and exhibition on the Internet aids in developing fan support around older motion pictures. An upsurge in fan trailers on the Internet is a burgeoning avenue of marketing for Hollywood studios and film distributors. Through new media platforms, fan trailers have the potential to reach global audiences and encourage social networking and commentary. In this study, the number of fan trailer views and user-generated message board posts was found to play a role in supporting interest in online film content. The audiovisual elements of both fan trailers for this study were generated from existing film content and repurposed to varying degrees. Specifically, the fan-edited trailer content was found to draw attention to the emotive properties of the film text. Collectively, the trailer narratives for this study presented an overwhelmingly favorable image of the case study film, as well as its historical significance and nostalgic qualities. The textual or written discourse analyzed in message boards on the webpages under investigation was found to shape the collective memory of the case study film as well. The content from this portion of the analysis also helped in preserving a positive view of the film itself, with much of the user-generated feedback positioned to promote the film’s cinematography and production design.

The composite findings indicate that fan trailers play a detrimental role in reviving older studio properties. The unintended consequences of these actions suggest a new avenue for media conglomerates and/or film distributors in marketing older motion pictures in the digital era. With Hollywood making fewer “midrange films [with] distinctly American subject matter,” such as Days of Heaven, smaller production companies and independent channels are overtaking this once-profitable market (Goldstein 2012). The role(s) taken on by members of participatory cultures, as well as the long-term effects of their interventions in online spaces, remains to be seen. For aging Hollywood film, fan trailers appear to offer one example of a promotional tool for film distribution and archiving. In June 2015, more than 88 million viewers had accessed 107 mock fan trailers through Honest Trailers, the YouTube-hosted channel by Screen Junkies (YouTube 2015). As Erickson (2009b) suggested, “with rapidly evolving technological features and equipment, tomorrow may yield an entirely new approach to using the Internet in a film promotion campaign” (51). As technological advancements in cinema and digital media continue to unfold, new online platforms and Web channels are creating an increasing number of spaces for participatory cultures and motion pictures. While many of these changes are on the horizon, scholars have predicted a continuous stream of content-related interruptions from tech-savvy film fans, as well as an evolution in the blending of virtual selves with cinematic information in cyberspace (e.g., Hansen 2006; Hardt and Negri 2004). Although the art of re-appropriating film content on the Internet has ballooned into a truly mass phenomenon, the future and direction of the fan trailer will depend on the negotiated balance between online cinephiles and digital control of motion picture properties.

REFERENCES 

Bettig, Ronald V. 2008. “Hollywood and intellectual property.” In The Contemporary Hollywood Film Industry, edited by Paul McDonald and Janet Wasko, 195-205. Malden, MA: Blackwell, 2008.

Biskind, Peter. 1998. Easy Riders, Raging Bulls: How the Sex-Drugs-and-Rock-‘n’-Roll Generation Saved Hollywood. New York: Simon & Schuster.

Boyle, James. 2008. The Pubic Domain: Enclosing the Commons of the Mind. New Haven, CT: Yale University Press.

Cunningham, Stuart and Jon Silver. 2013. Screen Distribution and the New King Kongs of the Online World. New York, NY: Palgrave Macmillan.

Crofts, Charlotte. 2001. “From the ‘hegemony of the eye’ to the ‘hierarchy of Perception’: The reconfiguration of sound and image in Terrence Malick’s Days of Heaven.” The Journal of Media Practice 2, no. 1: 19-29.

Ebert, Roger. 1997. “Great Movie: Days of Heaven.” RogerEbert.com., December 7. Accessed January 20, 2015.

Erickson, Mary P. 2009a. “Co-opting ‘independence’: Hollywood’s marketing label.” In The Business of Entertainment: Movies, Vol. 1, edited by Robert Sickels, 129-152. Westport, CT: Praeger Publishers.

Erickson, Mary P. 2009b. “KingKong.com verses LOLTheMovie.com: Toward a framework of corporate and independent online film promotion.” In The Business of Entertainment: Movies, Vol. 1, edited by Robert Sickels, 37-54. Westport, CT: Praeger Publishers.

Franich, Dennis. 2013. “Do audiences actually WANT shorter movie trailers?” Entertainment Weekly, June 7. Accessed November 13, 2014.

Goldstein, Patrick. 2012. “Hollywood’s global strategy: Made in America, but not for Americans.” 24 Frames: Movies: Past, Present and Future (Los Angeles Times blog), January 10. Accessed December 17, 2014.

Grainge, Paul. 2011. Ephemeral Media: Transitory Screen Culture from Television to YouTube. New York: Palgrave Macmillan.

Hansen, Mark B. N. 2006. Bodies in Code: Interfaces with Digital Media. New York: Routledge.

Hardt, Michael and Antonio Negri. 2005. Multitude: War and Democracy in the Age of Empire. New York: Penguin Books.

Hoyt, Eric. 2010. “The future of selling the past: Studio libraries in the 21st century.” Jump Cut: A Review of Contemporary Media 52. Accessed January 18, 2015.

Internet Movie Database. 2015. “Days of Heaven.” Accessed March 12.

Jenkins, Henry. 1992. Textual Poachers: Television Fans & Participatory Culture. New York: Routledge, Chapman and Hall, Inc.

Jenkins, Henry. 2006a. Convergence Culture: Where Old and New Media Collide. New York: New York University Press.

Jenkins, Henry. 2006a. “Quentin Tarantino’s Star Wars?: Digital cinema, media convergence, and participatory culture.” In Media and Cultural Studies: Keyworks, edited by Gigi Durham Meenakshi and Douglas Kellner. Malden, MA: Blackwell Publishing Ltd.

Johnston, K.M. 2008. “’The coolest way to watch movie trailers in the world’: Trailers in the digital age.” Convergence 14, no. 2: 145-160.

Kehr, Dave. 2011. “Days of Heaven.” In When Movies Mattered: Reviews from a Transformative Decade, 23-27. Chicago, IL: University of Chicago Press.

Kernan, Lisa. 2009. Coming Attractions: Reading American Movie Trailers. Austin, TX: University of Texas Press.

Koehler, Robert. 2013. “What the hell happened with Terrence Malick?” Cineaste 38, no. 4: 4-9.

Lazzarato, Maurizio. 2006. “Immaterial labor.” In Radical Thought in Italy: A Potential Politics, edited by Paolo Virno and Michael Hardt, 132-146. Minneapolis, MN: University of Minnesota Press.

Litman, Jessica. 2002. “War stories.” Cardozo Arts & Entertainment Law Journal 20, no. 2: 337-365.

Morrison, James and Thomas Schur. 2003. The Films of Terrence Malick. Westport, CT: Praeger Publishers.

Obst, Lynda. 2013. Sleepless in Hollywood: Tales from the New Abnormal in the Movie Business. New York: Simon & Schuster.

Patterson, Hannah. 2007. “Two characters in search of a direction: Motivation and the construction of identity in Badlands.” In The Cinema of Terrence Malick: Poetic Visions of America, edited by Hannah Patterson, 27-39. London: Wallflower Press.

Perren, Alisa. 2010. “Business as unusual: Conglomerate-sized challenges for film and television in the digital arena.” Journal of Popular Film and Television 38, no. 2: 72

Rombes, Nicholas. 2009. Cinema in the Digital Age. New York: Wallflower Press.

Rose, Frank. 2012. The Art of Immersion: How the Digital Generation Is Remaking Hollywood, Madison Avenue, and the Way We Tell Stories. New York: W.W. Norton & Company.

Rothman, Lily. 2014. “Movie trailers will get shorter, but won’t become interactive anytime soon.” TIME, January 27. Accessed October 3, 2014.

Roxborough, Scott. 2013. “Global on-demand revenues to top $6 billion by 2018.” The Hollywood Reporter, August 14. Accessed September 8, 2014.

Sickels, Robert. 2011. American Film in the Digital Age. Santa Barbara, CA: Praeger Publishers.

Smith, Emily. 2012. The Terrence Malick Handbook – Everything You Need to Know about Terrence Malick. Unknown Location: Tebbo.

Sterbenz, Christina. 2014. “How sketchy streaming sites really work – and why some are illegal.” Business Insider, April 24. Accessed January 10, 2015.

The Criterion Collection. 2015. “Days of Heaven: Terrence Malick.” Accessed on October 22, 2014.

Thompson, Kristin and David Bordwell. 2010. Film History: An Introduction (3rd ed). New York: McGraw-Hill Higher Education.

Thomson, David. 2012. The Big Screen: The Story of the Movies. New York, NY: Farrar, Straus and Giroux.

Tolson, Andrew. 2010. “A new authenticity? Communicative practices on YouTube.” Critical Discourse Studies 7, no. 4: 277-289.

Tyron, Chuck. 2009. Reinventing Cinema: Movies in the Age of Media Convergence. New Brunswick, NJ: Rutgers University Press.

Woessner, Martin. 2011. “What is Heideggerian cinema? Film, philosophy, and cultural mobility.” New German Critique 38,  no. 2: 129-157.

Wyatt, Justin. 1994. High Concept: Movies and Marketing in Hollywood. Austin, TX: University of Texas Press.

 

Filmography

Avatar. 2009. Directed by James Cameron. USA: 20th Century Fox.

Bonnie and Clyde. 1967. Directed by Arthur Penn. USA: Warner Brothers.

Days of Heaven. 1978. Directed by Terrence Malick. USA: Paramount Pictures. 

Gone Girl. 2014. Directed by David Fincher. USA: 20th Century Fox.

The Blair Witch Project
. 1999. Directed by Daniel Myrick and Eduardo Sanchez. USA: Haxan Films.

The Lord of the Rings: The Fellowship of the Ring. 2001. Directed by Peter Jackson. USA: New Line Cinema.

The Lord of the Rings: The Two Towers. 2002. Directed by Peter Jackson. USA: New Line Cinema.

The Lord of the Rings: The Return of the King. 2003. Directed by Peter Jackson. USA: New Line Cinema.

YouTube. 2008. “Days Of Heaven – Trailer (1978).” Last modified April 17, 2008.

YouTube. 2009. “Days of Heaven – Terrence Malick (1978).” Last modified on November 9, 2009.

YouTube. 2013. “Days of Heaven – Trailer.” Last modified on April 13, 2013.

YouTube. 2013. “Days of Heaven–A Video Essay.” Last modified on October 16, 2013.

YouTube. 2015. “Honest Trailers.” Accessed February 11, 2015.


Notes
[1]
Days of Heaven’s 1978 box-office gross was $3.5 million nationwide. Compare this figure to other mainstream studio releases of 1978 that received Oscar attention and critical acclaim, such as Heaven Can Wait ($81.6 million) (Beatty 1978), The Deer Hunter (roughly $49 million) (Cimino 1978), and Midnight Express ($35 million) (Parker 1978) (BoxOfficeMojo 2014).

[2] At the time of this writing, three Malick-directed films are in various stages of development, with his next feature, Knight of Cups, scheduled for wide release in 2015.

[3] The one-hour, thirty-three minute feature film is also available for rent or purchase on YouTube.

[4] Paramount Pictures’ YouTube channel displays fewer than 4,000 posts.

[5] This image is also used near the end of Trailer A, primarily to symbolize the passage of time for migrants moving from urban to rural areas.

Bio:

Kyle R. McDaniel is a doctoral candidate in the School of Journalism and Communication at the University of Oregon. His research interests include the intersections between American cinema and digital culture in the 21st century. His forthcoming dissertation focuses on the usage and repetition of visual effects in contemporary documentary film.

 

When a Good Girl Goes to War: Claire Adams Mackinnon and Her Service During World War I – Heather L. Robinson

$
0
0

Abstract:

Claire Adams Mackinnon and her contributions to the war effort 100 years ago are largely forgotten. The product of two Canadian military families, she put aside her burgeoning film career when war broke out to train and work as a nurse before returning to the silent screen. This article examines available evidence to reconstruct this period of her career (1914-1919), encompassing her nursing experience, her role in a fundraising drive for the US Red Cross, and her starring role in a government sexual hygiene campaign, which ignited one of the first censorship storms of the early film industry. I will argue that the choices Adams made reflect not only a determination to aid the war effort, but also place her at the vanguard of the women’s movement during the volatile years of the mid-1910s. This successful actress, who spent the second half of her life in Melbourne and regional Victoria, has largely been forgotten by film history. However, by placing this early period of her work within a firm historical context, one dominated by the fight for women’s suffrage, sex education and the First World War, we gain an appreciation of her significance within the early film industry and the origins of her ongoing community service.

 

Introduction:

For anyone with a passing interest in silent film history, Claire Adams Mackinnon is an intriguing transnational figure. A British subject for much of her life, she was born in Canada to an English father in 1896. A singer and an actress, on both stage and screen, she was a child performer using her real name “Beryl” Adams before gaining success as a motion picture actress in New York. As a teenager in early Thomas Edison productions she was known first as “Clara” then “Peggy” Adams before publically assuming her family pet name “Claire” in 1918. In 1920 Adams moved to Los Angeles and enjoyed an eight-year Hollywood career. In 1931 she became an American citizen prior to spending the last forty years of her life as an Australian.[1] Today, however, she remains largely unknown, despite appearing in some of the silent era’s most influential, popular and profitable films alongside some of the best-known artists and producers of the era.

If Adams is remembered at all it is within select circles in Victoria, Australia. Following the death of her first husband, Adams enjoyed a whirlwind courtship and marriage to Donald Scobie Mackinnon, the scion of a Melbourne legal and horseracing family. Together they reinvented the Western District homestead Mooramong, creating a jazz age delight redolent of the Hollywood Hills lifestyle she had left behind. Now in the hands of the National Trust of Australia (Victoria), Mooramong remains defiantly anachronistic in a district better known for its bluestone piles and weatherboard restraint, a monument to a much-loved couple reflecting their sophisticated tastes and cosmopolitan life experiences. Mrs Mackinnon’s career is reduced to a prelude to her second marriage, like a folly of youth indulged before her adult life began.

Figure 1: Mooramong homestead, Skipton, Victoria. Image used courtesy of National Trust of Australia (Victoria).

Figure 1: Mooramong homestead, Skipton, Victoria. Image used courtesy of National Trust of Australia (Victoria).

An intensely private person throughout her life, Adams was enigmatic from her earliest interviews and has left little evidence to enable a simple or personal interpretation of her career.[2] Unlike many of her more famous contemporaries, she was not a movie star created by a studio publicity department: she was an experienced working actress, appearing across genres for a number of companies and eschewing the celebrity lifestyle. This independent approach resulted in a comparatively low public profile.[3] Nor was Adams in Hollywood when the studios began to construct their historical legacies. As such, she appears infrequently as a footnote in popular and scholarly accounts of the silent era, including the one written by her first husband, Benjamin B Hampton.[4] Eminent film scholars welcomed Hampton’s text as an authoritative account of the history of the early film industry.[5] If even her husband did not think she warranted more than an image caption, one could easily conclude that Adams was not terribly significant. However, as Lesley Speed suggests, Adams’s career is significant not only for the continuity of her workload, but for the time in which she was active, “spanning a period of major changes in the American film industry, including the transition from short films to feature-length productions, the industry’s relocation from the East coast to Los Angeles, the establishment of the star system and the formation of vertically-integrated major studios” (Speed 2015, 4).

Like the majority of women who dominated the ranks of the early motion picture industry, Adams has fallen through the gaps of film history’s indeterminacy.[6] However, as more scholars analyse the period and primary sources become more accessible, we are able to reconstruct a more accurate representation of the industry, the individuals and the era during which Adams was active, and the picture that emerges is one dominated by the women’s movement. Women made up at least 83% of the cinema audience at a time “when women’s voices were particularly valued” (Stamp 2012, 6). Women’s history scholars note that from the late 1910s “young women could not help but be influenced by the currents of the age” (Banner 1974, 151). As a young woman in New York City during the First World War, Adams was surrounded by the opportunities, debates and challenges confronting the first generation of young women free to pursue a career as a choice rather than a necessity. She was amongst the target audience for orators and activists such as Margaret Sanger promoting the birth control movement and key figures campaigning for Women’s Suffrage, which would not pass into legislation until 1920. We have no evidence that Claire Adams identified herself as an early feminist. However, through her career choices, she reflects the values and ideologies synonymous with the movement, a cultural catalyst which grew in parallel with the development of the motion picture industry.

By mainly focussing on two of her early feature films, this article will demonstrate that Claire Adams deserves recognition not only as a well-respected, adventurous and greatly admired actress of her day, but also as a young woman navigating the perils and opportunities opening up for women in the early twentieth century. She is also a representative of the forgotten contributions of countless women working in early cinema. I will examine her work of one hundred years ago, when she stepped away from Edison Studios and the early motion picture industry to train as a nurse, only to return to motion pictures in support of the troops heading off to the battlefields of Europe. By placing these early works within a firm historical context we will gain a more complete understanding of the scale and intersections of her careers, as both nurse and actress, her role at the vanguard of contentious social issues of the mid 1910s and her active participation in landmark moments of the early American film industry.

Real Life Drama

According to the Australian Dictionary of Biography, Claire Adams worked as a nurse for the Red Cross during World War I (Maxwell, 2000). This description conjures evocative images of mud and blood, battlefields and field hospitals. What drove an aspiring young actress away from the movie studio and into a hospital theatre? What data is available to determine how much official nursing she contributed to the war effort one hundred years ago?

Figure 2: Figure 2. A young ‘Clara’ Adams (centre) with co-stars from left: Marion Weeks, Yale Boss, Elsie McLeod and Gertrude McCoy in The Office Boy’s Birthday (Charles M. Seay, 1913). From Kinetogram, 1 January 1913. Source: Seaver Centre for Western History Research, Natural History Museum of Los Angeles County.

Figure 2:  A young ‘Clara’ Adams (centre) with co-stars from left: Marion Weeks, Yale Boss, Elsie McLeod and Gertrude McCoy in The Office Boy’s Birthday (Charles M. Seay, 1913). From Kinetogram, 1 January 1913. Source: Seaver Centre for Western History Research, Natural History Museum of Los Angeles County.

In 1912, Adams began performing for Thomas Edison’s film production company under the stage name “Clara” Adams and appeared in 17 motion picture shorts in two years (Internet Movie Database, 2015). At the outbreak of war in 1914, she was almost 18 years of age. Both her grandfathers had served in the British military in Canada and India before entering business and public service (Letourneau 1982; Manitoba Historical Society 2009). This family tradition and tales of exploits on the battlefield inspired Adams and her brother Gerald as children (Will 1978, 30). As young adults, they were determined to emulate their forebears and sought opportunities to volunteer for the war effort.[7] Together they had grown up in an environment that was conservative and under the influence Anglo-Victorian social strictures of duty and community service. These strictures included a strong sense of volunteerism, which saw upper- and middle-class young people from across the British Empire come forward to support their King and Country (Quiney 1998, 190). Adams set aside acting to make an active contribution to the war effort.

Like thousands of young women across the British Empire, Adams applied to join the Voluntary Aid Detachment or V.A.D service, a corps of nurses and ambulance drivers trained and managed for the Red Cross by St John’s Ambulance. However, owing to her youth and inexperience, she was turned away (“Claire Adams, Starring in ‘The End of the Road’ Is Real Western Canada Girl”, 1920). She could not change her age but was able to address her skillset. Adams enrolled at Detroit’s Grace Hospital Training School for Nurses, one of the most respected institutions of its kind in North America (“Claire Adams, Starring in ‘The End of the Road’ Is Real Western Canada Girl”, 1920). Grace Hospital was just over the border from Canada on the shores of Lake St Clair, close to Toronto where she was living with her father Stanley Adams (Sweet & Stephens transcript). By 12 June 1915, Adams was listed on a ferry passenger manifest as an eighteen-year-old student nurse travelling between the two cities (Ancestry.com, 2009).

Unfortunately the enrolment records for the Grace Hospital Training School for Nurses from this time period do not survive, but the institution’s prospectus does, specifying a strict and rigorous lifestyle for their live-in student nurses.[8] This training would equip the young ladies to assist senior staff with both male and female patients in the hospital’s surgery, children’s ward and the obstetrics wing. Other extant records from Grace Hospital include the lists of graduates, and Adams’s name does not appear there either.[9] However this would not have prevented her from gaining employment as a nursing assistant, especially at a time when people with any medical or first aid experience were in high demand. Many students enrolled and failed to make it through the rigors of training or were unsuited to the prohibitive lifestyle required of those living in (Kathleen E. Schmeling, pers. comm.).

The Canadian Red Cross website describes how anyone working as a volunteer at a hospital during the Great War was referred to as a Red Cross nurse. Unfortunately records of the thousands of people who gave their time and energies to help have been either destroyed or did not exist in the first place.[10] What do still exist though are the Canadian census records taken in 1916 (Ancestry.com, 2009). They record that Adams was staying with her mother in Winnipeg and working as a “doctor’s assistant”. As happened across the Empire, she was most likely one of the thousands of women who volunteered to work in their local hospital. Deer Lodge was one of many country houses and hotels across Canada converted to military hospitals dedicated to convalescent soldiers returning from the front (Deer Lodge Centre, 20014). The English Government was short of both funds and infrastructure to deal with the number of wounded generated by the new means of mechanised warfare. Deer Lodge was established as a repatriation hospital in Winnipeg in October 1915 following a request made by the English Government to Commonwealth nations contributing troops to the war. Each country or dominion of the British Empire became responsible for the convalescent care of their own when casualty numbers exceeded British expectations and resources.[11]

Moving back to Winnipeg to reside with her mother Lillian, a piano teacher, also offered Adams the chance to participate in another historical event. In January of 1916, the women of Manitoba Province were the first in Canada to be granted the right to vote in municipal elections and hold public office. This occurred 2 years before most of their countrywomen and 4 years ahead of women in the United States (Jackal & Millette, 2015). Even during times of war, this would have been something worth celebrating and hard to ignore.

Following the Battle of the Somme in July 1916, the number of wounded Canadian troops requiring convalescent care rose from 2,620 in July 1916 to 11,981 by the year’s end (Veterans Affairs Canada, 2015). Adams may not have experienced the action on the front, but she would have witnessed the consequences it had on soldiers. In whatever capacity she was serving, she would have been expected to meet the demands of a near five-fold increase in the number of shattered patients, some approximately her own age, in the makeshift over-crowded wards ill-equipped to deal with them.

In 1916, Adams was still only 19-years-old. The physical toll of nursing accompanied by emotional exhaustion were the reasons Adams later gave to explain her return to motion pictures, admitting that she “collapsed finally under the strain of her work” (Goldbeck 1920, 27). Although equipped with some training and experience, she may not have been capable of maintaining an emotional distance or remain objective from the suffering she confronted. Well before the Armistice was declared in November 1918, she was back with Edison Company performing for the cameras. In doing so, she continued contributing to the war effort, but it was on a scale that exceeded anything she could have achieved had she remained a doctor’s assistant in Winnipeg.

Returning to the screen in 1917 with the stage name “Peggy”, Adams appeared in 8 Edison Productions. These included Barnaby Lee (Edward H. Griffith, 1917), her first feature film running over four reels (48 minutes), and the first of several in which she was directed by Edward H Griffith.[12] She starred in Your Obedient Servant (Edward H. Griffith, 1917), the first adaptation of Anne Sewell’s classic novel, Black Beauty (see Figure 3). Her next production, Scouting for Washington (Edward H. Griffith, 1917) was a melodrama set during the American Revolution. Wild Arnica and Shut Out in the Ninth (both Edward H Griffith, 1917) show a very young Adams, all dimples and dark curls, revelling in simplistic comedy.[13]

 

Figure 3: Claire Adams with Pat O’Malley in Your Obedient Servant, (Edward H. Griffith, 1917). Edison Conquest Pictures. 1917. Source: UC Irvine Special Collections and Archives.

Figure 3: Claire Adams with Pat O’Malley in Your Obedient Servant, (Edward H. Griffith, 1917). Edison Conquest Pictures. 1917. Source: UC Irvine Special Collections and Archives.

The following year, Adams would make the pivotal films of her early career. They started with a short vignette, one of a series of social satires titled Girls You Know. It would be the last time she appeared in a motion picture produced by the Edison Company. Each sketch “featured a different type of American girl personally selected” by James Montgomery Flagg, the prodigious illustrator, cartoonist and photographer (“Edison Releases Flagg Social Satire Series.” 1917). The First World War was something of a career peak for Flagg. He’d had film experience, was a widely read social satirist and artist and produced recruitment and patriotic propaganda posters for the US government. Flagg’s most recognised image was his creation of Uncle Sam, the grizzled old patriarch on the recruitment posters who “Needs You for US Army”. [14]

Flagg was commissioned to provide the “one-reel photo-sketches … depict(ing) the grace, charm, foibles and frailties of ‘Girls You Know’, the likable types” (“Recollections of ‘Girls You Know’”, 1918). These “likeable types” featured such stereotypical characters as “The Art Bug”, the “Spoiled Child”, the “Bride” and “The Artist’s Model”. Adams was cast as The Man-eater (Jack Eaton, 1918). The influence of the war in this film is obvious. Adams’s character Nina bids a tearful farewell to her uniformed fiancé and dreams of medals pinned to his broad manly chest. Nina is a sprightly little flirt travelling to a picnic with friends, harmless but annoying in her need for male attention. She is eventually “doused off”, pushed into the river, pulled ashore then scolded like a half-drowned kitten.[15]

The Girls You Know vignettes were produced as light and satirical entertainment for the troops and for those on the home front. They also reflect the restrictive caricatures of young women in the early motion picture industry, characterisations of women indulgently pursuing or experimenting with a career (“The Artist’s Model” and “The Art Bug”). Others were defined by their relationships with parents or male partners (“The Spoiled Child”, “The Bride” and “The Man-eater”). As “the Man-eater”, shunned and dunked for her sexually assertive manner, Adams’s character suffers the entrenched social consequences as punishment for her morally dubious behaviour. The cast of the series were promoted as having been “all selected from (Flagg’s) models – the most famous in New York”, though featuring leading women “with no experience but clever ideas and no actresses of note” (“Recollections of ‘Girls You Know’”, 1918). Adams, however, was by this stage a well-established performer, and three of her four “Girls” colleagues also had previous motion picture experience.[16] Flagg, the producers and indeed audiences, may have been more attracted to a fresh pretty face than a talented individual with an identity, reflecting the low opinion many still held of the acting profession.[17] There is no small irony that she was at this time living against tradition, earning a respectable wage and supporting herself independently as a professional young woman.[18]

The Spirit of The Red Cross, (Jack Eaton, 1918)

When the United States entered the European conflict in 1917, the US chapter of the International Red Cross was charged by the government to raise funds and volunteers to maintain an active presence on the battlefields of Europe. The subsequent rise of American humanitarianism and the specific formation of a nation-wide Red Cross campaign coincided with a period when “sensationalistic mass media began to dominate American culture”. These forces, dominated by new motion picture technologies, were responsible for the “reshaping American ways of seeing, feeling, and responding to suffering by treating violence and pain as pleasure-producing commodities” (Rozario 2003, 426).

The Red Cross recognised the timeliness of motion pictures as an effective and expedient medium for stimulating potential donors and volunteers across the country (Rozario 2003, 429). The National Association of the Motion Picture Industry (NAMPI), chaired by famed producer Jesse Lasky, was appointed to the American Red Cross to promote and “interpret the organization for the American People” through the production of a suitable motion picture (“Spirit of the Red Cross 1918). Flagg was engaged to write the script. Jack Eaton, also of the Girls You Know series, was brought in to direct. Adams was cast as the lead, a nurse named Ethel, the image and embodiment of the International Red Cross. Lasky’s production expertise ensured that the resulting film, The Spirit of the Red Cross (Jack Eaton, 1918) presented “some of the best work done in motion picture making” (“Spirit of the Red Cross 1918). With such an accomplished and professional team involved, this 2-reel film was certain to not only hit but exceed the patriotic, inspirational and philanthropic targets for which all parties were aiming.

Adams’s lead character Ethel was the sweetheart of “an American youth” Sammy, played by Ray McKee. When Sammy enlists with the army and sails for France, Ethel volunteers as a nurse. The film portrays the work of the Red Cross in the field, assisting refugees, nursing and caring for the wounded. The New York Times described Sammy heading for the battlefield, envisioning “Ethel in her white uniform, watching over him”, until:

After a charge, he lies on the ground with a bullet in his chest, half conscious. The vision of Ethel awakens him, just as a German comes forward slaying the wounded. Sammy grips his revolver and shoots the enemy. Later, removed to base hospital, Ethel finds him and nurses him back to health. [19]

The Red Cross were not averse to engaging exaggerated propagandist images for the benefit of their cause.[20] Their tactics owed much to the pulp media, sensationalist newspapers and war magazines, aiming to provoke the outraged and patriotic public into “acts of benevolence” (Rozario 2003, 430). What makes this campaign remarkable is that it was the first time such techniques had been mobilised so effectively on a broad national scale, utilising the emotive mass cultural medium of motion pictures.

The American Red Cross expected their national fund raising drive to receive “considerable impetus” from the release of The Spirit of the Red Cross (“Spirit of the Red Cross”, 1918). Newspaper listings from across the country noted that cinema proprietors were donating all takings from the screening to the cause.[21] The New York Times quoted that the goal of the organisation was to raise one hundred million dollars, or around $1.8 billion in US dollars today (Measuring Worth, 2014). By the time the Armistice was declared in November 1918, only 7 months after the film’s release, the Red Cross had raised approximately four times that sum (American Red Cross, 2014). The film is credited with playing a major role in the campaign’s success. By the war’s end, one-third of the US population, approximately 39 million people, were either contributing members of the Red Cross or serving as volunteers.[22] At the cessation of hostilities, war movies were the most successful types of films in the country and “spectacle was in the ascendant” (Rozario 2003, 439).

The promotional poster Flagg created for The Spirit of the Red Cross depicts the diaphanous vision of Adams’s character floating over the battlefield. Ethel valiantly directs stretcher-bearers toward the unseen wounded. The plaintive caption declares: “Not one shall be left behind” (See fig. 4). For the many, perhaps the millions, who attended the film or saw the poster, Adams represented the thousands of nurses serving in the field, saving the fallen and tending the refugees, even though she was not entirely happy with her performance (Lake 1922, 51). She may not have been directly ministering to the wounded any longer. However, through her contribution to this film, Adams achieved more for the war effort as a nurse on screen than would have been possible had she remained one in life. Well into her old age it was commonly accepted and reiterated in biographical entries that Adams had been a nurse for the Red Cross during the First World War. Having taken part in such a high profile and hugely successful campaign, it is fair to say that Claire Adams was not simply “a nurse” for the Red Cross: she was The Nurse, the face on the movie poster, the selfless guardian of the wounded and indeed the embodiment of The Spirit of the Red Cross.

Figure 5: Figure 4: The Spirit of the Red Cross (James Montgomery Flagg, 1918). Source: World War 1 Posters from the Elizabeth Ball Collection, Ball State University Archives and Special Collections. Ball State University, 2011. All rights reserved.

Figure 4: The Spirit of the Red Cross (James Montgomery Flagg, 1918). Source: World War 1 Posters from the Elizabeth Ball Collection, Ball State University Archives and Special Collections. Ball State University, 2011. All rights reserved.

Adams was well positioned to maximise the opportunities offered by both the emancipation of women in general and the freedoms and opportunities offered by the motion picture industry. At the dawn of the Twentieth Century, middle-class women in North America were still expected to abide by the traditional Victorian roles of marriage, motherhood and marital obedience. As the 1920s loomed, however, “a woman’s alternatives to marriage were not only possible but exciting” (Banner 1974, 48). This social shift was influenced by women’s experiences backfilling male positions in the workforce and volunteering for the war effort (Quiney 1998, 193). By the time Armistice was declared, Adams was twenty-two years old with an independent motion picture career, an exciting new option embraced by increasing numbers of young women in the West. As such she exemplifies the symbiotic relationship between two new cultural forces at play, demonstrating that “Cinema history and history cannot be separated from one another”: “In the field of tension between the history of cinema in the 1910s and gender history, a movement of emancipation takes place” (Schlüpmann 2013, 24).

Women were not only welcomed but were highly valued in the early motion picture industry, occupying positions from theatre staff to major writers and directors. Like Adams, despite their prominent positions, most did not publically identify themselves with the early feminist movement (Slide 1996, 1-3). Crucially Adams had also proven she had the talent, experience and determination to excel. Having starred in the commendable The Spirit of the Red Cross, she had also gained her family’s acceptance and support (Lake 1922, 51). From 1919 she was known publically not as “Clara” or “Peggy” but as Claire Adams, the “pet name” favoured by her family. Following The Spirit of the Red Cross, there were several projects to which she would dedicate her talents. One of these was a leading role in Romance and Brass Tacks (Martin Justice, 1918), “the new Flagg comedy starring pretty Peggy Adams, the famous Broadway beauty” (“An Unusual Comedy at Vining Theatre” 1918). The next motion picture Adams found herself starring in was another high-profile government sponsored film, also made for the war effort and proving remarkably popular. However, it also drew the kind of attention that few respectable young ladies from Winnipeg may have welcomed. Adams starred in one of the first major censorship scandals of the motion picture industry.

The End of the Road (Edward H. Griffith, 1919)

Young girls, thrilled with patriotism, sometimes fail to realize that the uniform covers all the kinds of men there are in the world; men of high ideals … or, in the worst instances, men who feel that their own physical appeal must be gratified, no matter who suffers. And so, through ignorance, through emotion, take steps which will lead to bitter regret.

(Katherine Bement Davis from Colwell 1998, 73)

Following The Spirit of the Red Cross and the additional projects with Flagg, there is evidence that Adams returned to theatre. In May 1918, she is named amongst the cast members of a patriotic play called Loyalty, written by George V. Hobart and staged at the Belasco Theatre in Washington DC. Belying both Adams’s preference for patriotic pieces as well as the public’s support for them, Loyalty is described as:

Something more than a mere dramatic entertainment … the play also carries a message of hope to a world suffering from the hardships and horrors of war (“Loyalty Will Make First Appearance At Belasco Tonight” 1918, 14).

The war may have been over but the horror continued on the home front. The “The Spanish Flu” evolved in the trenches of Europe from a moderately innocuous virus into something quite lethal. The appearance of the disease in New York City, borne by returning soldiers, was first reported in August 1918. The vius attacked the young and healthy, becoming more infectious and lethal with each mutation. Many industries, including motion picture companies and theatres, were unable to operate normally or in some instances were forced to close. Adams may have been forced to take time off or found work options cancelled or postponed. By the time the city recovered months later, it had lost 20,000 to 24,000 of its six million citizens. (Dominus, 2009). However, influenza was not the only infection rife amongst the troops.

In her essay, “The End of the Road: Gender, the Dissemination of Knowledge, and the American Campaign against Venereal Disease during World War 1”, Stacie A. Colwell provides a detailed account of the social and political context which gave rise to Adams’s next project (Colwell 1998, 44-82). As the US entered the war, the number of new recruits infected by venereal diseases appalled doctors conducting medical examinations.[23] As had been customary for generations, many young men of the time were engaging in unprotected pre-marital and extra-marital sex, ignorant or uncaring of the consequences. This practice was intensified during the heightened emotional climate of the war. Young women at the same time were becoming more personally independent, challenging the social order and demanding equal rights and freedoms to men. These included sexual freedoms.[24] They were unfortunately doing so within a vacuum of ignorance, a symptom of the perennial code of dual morality encouraging young men to explore their sexuality while young women were kept sexually innocent, unaware of their body’s functions and frailties. Many, deeply smitten, unknowing or unable to say no, found themselves bearing the physical and social consequences of sex with strangers or sweethearts they feared they’d never see again. The result was a perfect storm of ignorance, class-based bigotry and misbegotten morality, and Adams stepped right in the middle of it.

Eugenics, racism and class tensions exacerbated both the fear and apportioning of blame for the spread of venereal diseases (Schaefer 1999, 21). At a time when eugenics was still considered to be a viable scientific theory, some believed that the spread of sexually transmitted diseases (STDs) such as syphilis and gonorrhoea served a deliberate political or religious agenda.[25] Others saw the epidemic as an insidious weapon activated by the working masses, crowding into industrialised urban centres to breed out the richer, more refined classes (Colwell 1998, 72). The front line troops in this intimate means of class warfare were believed to be not simply prostitutes (who had a surprisingly low level of infection due to regulatory frameworks and inspections of brothels), but liberated “diseased and promiscuous” women seducing innocent men of means who then passed the infection onto their wives (Colwell 1998, 72). In turn, these ‘pure’ middle- and upper-class women would be driven mad, infertile, or both, by the ravages of the disease.[26] With many changes being made to the social order and a popular culture rife with inflammatory propaganda regarding invaders from abroad, there’s little wonder that this kind of imbalanced fear and paranoia infected the home front, like another disease.

The U.S. War Department, ever the pragmatists, saw the issue quite clearly – sick soldiers were bad for morale, an additional expense and made for a weakened army (Colwell 1998, 47). The challenge of addressing this issue fell to the War Department’s Committee on Training Camp Activities. This committee was confronting disciplinary issues relating to groups of young, independent and assertive New Women moving to towns and areas surrounding the training facilities. Schaefer implies that some were probably entrepreneurial prostitutes. Others were most likely “the khaki mad girls”, opportunists who simply loved a man in uniform and weren’t afraid to show it (Colwell 1998, 73). The troops themselves, predictably, were considered relatively blameless.

The momentum generated by the war to stem the spread of STDs provided an opportunity for diverse community organizations with similar concerns to affiliate with the War Department. A consortium of public health and volunteer organizations was incorporated under the banner of the American Social Hygiene Association (ASHA). Combined, they represented “a hybrid of social purity and sex education movements” (Colwell 1998, 46). These organizations pressed for a means of educating the population about the facts of life, the evils of STDs and pre-marital (or extra-marital) sex. Yes, it was bad for the War Effort. However for ASHA, the spread of venereal diseases was due to ignorance within a paradoxical culture. On the one hand young people had been loosened from the corset-like confines of the Victorian moral order, yet on the other were still bound by obscenity laws that made it illegal to disseminate medical or preventative information regarding STDs.[27] A “conspiracy of silence” reigned: people did not speak openly of such things and had few available sources of information (Schaefer 1999, 21). The volatility of the subject was amplified by the controversies surrounding the birth control movement. As Margaret Sanger and her supporters discovered, it was still considered obscene and illegal in the United States to distribute materials addressing the use of prophylactics and other forms of contraception.[28] Some considered it more socially repellent to talk of venereal disease in public than to actually contract a case and not mention it (Schaefer 1999, 21). ASHA concluded that a range of sensitive and educational, yet frank and fearless, vehicles were required, aimed at both men and women to address the facts of life and some of their indelicate consequences. As the Red Cross had realised before them, the military and their new allies embraced the power of motion pictures to educate and inform the public of America. A motion picture offered an additional bonus: In the darkness of a movie theatre, no one can see you blush.

Their strategy involved dividing the audience along gender lines. Fit to Win (Edward H. Griffith, 1919) was crafted to appeal to the needs and experiences of young male recruits.[29] The End of the Road (Edward H. Griffith, 1919) was aimed at young women who, out of curiosity, ignorance or early “girl power” gone awry, were finding themselves in physically and morally perilous positions. The film was to be screened with a pre-show lecture by a medical practioner in “Ladies Only” sessions at public cinemas. In order to make the subject matter more palatable for a “delicate” female audience, the educational aspects were wrapped up in a charming love story.[30] Conveniently, director Edward H. Griffith had been drafted into the War Department’s Committee on Training Camp Activities. Previously engaged by Thomas Edison Company, and with acting experience of his own, he played a major role in the production of several films supporting the war effort. During his time at Edison, Lt Griffith had made four films with Claire Adams. She was cast as Mary, the heroine of this sensitive story, opposite renowned actor Richard Bennett, who had enjoyed an expansive and celebrated theatre career before taking to the screen.[31]

A key figure within ASHA, Katharine Bement Davis was brought in to write The End of the Road. Davis described her contribution to the project as having “been most carefully worked out in consultation with physicians on the side of fidelity to medical fact, and with teachers as to the psychological effect” (Colwell 1998, 48). Davis had an established career in education, penal reform and public health, involving intensive research into the social causes of delinquency, the efficacy of reform programs for female prisoners, as well as surveys into the nature of women’s sexuality.[32] Her work had inspired John D. Rockefeller Jr. to establish the Bureau of Social Hygiene as part of his newly established foundation (Encyclopædia Britannica Online, 2014).

The film was produced under the supervision of the Surgeon General of the US Army, a protective political arm supporting the enterprise. Additional collaborators included the National War Work Council, the YWCA, Rockefeller’s Bureau of Social Hygiene and the Famous Players-Lasky Corporation (American Film Institute Catalogue, 2015). This was a high-powered group of influential stakeholders with an interest in the film’s success. There was more than the usual amount of pressure and expectation placed on the actress playing Mary, the leading lady in this high-stakes drama. The following synopsis is from the American Film Institute Catalogue:

Mary, whose mother has instructed her about love, marriage and sex, leaves her boyfriend Paul to become a nurse in a New York hospital. Vera, encouraged by her mother to marry a rich man, takes an apartment from a young millionaire who promises marriage but only gives her syphilis. Mary and her doctor treat Vera and show her examples of the ravages of the disease. Mary meets other suffering women: an Irish servant girl betrayed by a chauffeur who dies after her baby is born; a garment worker who contracted syphilis from a soldier’s forced kiss; the invalid wife of a wealthy man whose philandering caused her condition, the blindness of her child and the suicide of another of his conquests. Paul, about to enlist, suggests that he and Mary have sex before her goes. Disappointed in him, Mary also rejects the proposal of her doctor, but later in Europe after seeing the kind of man he is, she accepts him.

On all fronts, this was a brave role for Adams. Some of the scenes featured actual patients scarred and suffering from venereal diseases, filmed on location at the women’s wards of Blackwell’s Island, New York City.[33] In her role as the nurse Mary, Adams is seen conversing with these obviously frail patients, gently supporting them as they present for the camera.[34] Most of the film was shot at the Rockefeller estate at Pocantico Hills in Mount Pleasant, New York State (Colwell 1998, 63). As a resident of New York City at this time, it would have been difficult for Adams not to be aware of the campaigns for and against women’s suffrage and access to birth control, including the incarceration and exile of Margaret Sanger, all of which received widespread publicity between 1916 and 1918 (Banner 1974, 103). There is also the possibility that she may have had first-hand experience with the physical, social and psychological impacts of these diseases during her training and duties in the obstetrics ward at Grace Hospital in Detroit. That possibility becomes more probable when considering her contact with repatriated soldiers in Winnipeg. Perhaps this experience galvanised her willingness to participate, in spite of the risks. Her participation in The End of the Road reflects the influence of her family’s military and public service background as well as her own compulsion to help others. She would require the strength of the first and her faith in the latter to confront the controversy this film was about to generate.

Figure 5: Claire Adams as the nurse Mary Lee with a patient in The End of the Road (Edward W. Griffith, 1918). Source: UC Irvine Special Collections and Archives.

Figure 5: Claire Adams as the nurse Mary Lee with a patient in The End of the Road (Edward W. Griffith, 1918). Source: UC Irvine Special Collections and Archives.

The spark that ignited a furore was the sudden end of the war. This timing meant everything. Audiences and the industry had had enough of both war and sex.[35] The military units and programs that sponsored the film’s production were disbanded. Several key elements and references of the storyline were no longer relevant or threatened to offend peace-time sensibilities. Davis, however, successfully argued that the need to educate young adults, particularly young women, about their bodies remained, and that this justified public screenings. The project was permitted to “continue until a clear opinion has developed concerning the desirability or undesirability of its being shown through commercial channels” (Colwell 1998, 66). The ASHA gained control of the copyright of the film. The distribution and exhibition remained (for the time being) under the auspices of the Surgeon General, with safeguards in place to protect the film “from sensational exploitation”. Women’s groups and other officials were invited to preview screenings to judge its value and suitability for local audiences (Colwell 1998, 64).

The film premiered on 16 February 1919 in Syracuse, New York. There were fifteen hundred people at opening night and it went on to capacity crowds in a number of other cities (Schaefer 1999, 28). Most of the film’s key personnel were not there to see it. By opening night, Griffith had gone west. Davis sailed to post-war Europe pursuing her new role as General Secretary of Rockefeller’s Bureau of Social Hygiene. There was only one participant left with any skin in the game who made herself available to face down the critics and attend the film’s premiere. Colwell notes: “For the record, actress Claire Adams, who played Mary Lee, did travel to both Syracuse and Pittsburgh to defend the film she had starred in” (66). The New York Times reviewed the film in March 1919 and enthused that The End of the Road was “the most valuable motion picture of its kind yet produced.” The picture is not pleasant or euphemistic. Neither is the subject with which it deals. It is unpleasant, however, only to the degree necessary for force, and plain spoken only to the extent necessary for clearness. It is never morbid. One feels clean after seeing it (“Opening the Road” 1919). Acknowledging that audience responses would depend upon their own moral convictions, The New York Times also reassured readers: “No film would be effective without competent acting and directing and it is an important virtue of ‘The End of the Road’ therefore, that it had both in its making … Claire Adams and Joyce Fair, in the two leading female roles, were attractive in appearance and intelligent in their interpretation of their characters” (“Opening the Road” 1919).

Despite positive support in the mainstream media, critics of the film began protesting as soon as it was released. As The New York Times forewarned, the acceptance of the film was divided along the lines of the individual’s moral view and tolerance of the story’s unique premise: it was not just “patriotic prostitutes”, “army flappers” or “camouflage dames” responsible for the spread of STDs. The film dared suggest that decent middle class men played an active part.[36] The End of the Road was the first film to present the case that STDs did not always originate in women of the lower classes, demonstrating that “syphilis and gonorrhoea are equal opportunity diseases” (Schaefer 1999, 33). This perspective contributed to establishing a new set of social and political battlelines, just as the Great War ended.[37]

The pendulum swung back to a more conservative side of society, basking in victory and longing to normalise culture and behaviour (Schaefer 1999, 34). As a government sponsored project, The End of the Road drew intense criticism, more so than other sensational films that had preceded it (Schaefer 1999, 29). ­­­­­­­­­­According to critics in Moving Picture World, copies of The End of the Road and Fit to Win had also “fallen into the hands of individuals who are allegedly exhibiting it to mixed audiences composed of men, women, boys and girls” (“Association Goes After ‘Fit to Win’”, 1919, 1141). Prominent church figures denounced the film, led predictably by the Catholic Church (Schaefer 1999, 32). By July 16, 1919, it was banned in Philadelphia. National Association of the Motion Picture Industry (NAMPI), so successful in their production of The Spirit of the Red Cross, succeeded by the end of 1919 in having the film banned across the United States. This reflects what Löhrer describes as “a specific American moral panic at play” (Löhrer 2011).

Although the film could no longer be shown in the US, The End of the Road travelled abroad.[38] In May 1920 it was exhibited in Canada under the auspices of the Canadian National Council for Combating Venereal Diseases and the principals of local colleges. Demonstrating her ongoing commitment to the work, Adams reached out to her fellow Canadians with a personal message printed in The Vancouver Sun, subsequently reprinted across Canada. She is an unbowed advocate, certain “that every girl in Canada could see this play”:

The message it conveys is one that society must learn, and I feel that this powerful drama is the most wonderful method of telling the important story to girls everywhere. I played my part with that thought in my mind, and in my heart the feeling that at least to the best of my ability I was performing a real service to womanhood (“Message to Girls Through Film By Claire Adams” 1920).

The End of the Road found welcoming and respectful audiences even further afield. There is evidence that the film was used as an educative tool in military training campaigns as far away as Vladivostok (McMaster 2014, 5). Over the next decade The End of the Road was also shown extensively across Australia, where Adams would spend the second half of her life. Promoted in The Sunday Times in October 1920 with the US plan in place to segregate audiences along gender lines, the film opened at the Sydney Town Hall on Saturday 6 November 1920 and played for 5 weeks. The Evening News described how it had been shown the previous week at a private screening for representatives from the clerical, medical and legal professions, and was granted approval to be screened by the Minister for Womanhood. A search of Australia’s digitised newspaper archives shows that the film travelled through major metropolitan and regional centres across the country, from Muswellbrook NSW to Charters Towers QLD, Clare SA to Katanning WA. There is little criticism of the film evident: rather it received endorsement from civic leaders, educators and the press across the country.[39] Adams received rich praise. In Melbourne it was shown at the Palace Theatre on Bourke Street where the Table Talk reviewer declared in March 1921, “Nothing better has been shown on the screen than the perfectly moulded features of Claire Adams.” When The End of the Road reached Hobart in June 1921, The Mercury reported that over 5,000 had already seen it in Launceston. The reviewer from The Advertiser in Adelaide, who had seen the film at the Adelaide Town Hall, enthused in July 1921 that “The picture defies description. There is a touch of genius in it. The screen has never disclosed a purer or more impressive lesson.” The following week, the same paper discussed how several members of the public showed great interest in Adams, enquiring after her background and identity. They make much of her youth, talent and beauty, but pay particular attention to her patriotic service during the war and her nursing experience at Grace Hospital in Detroit.

The film continued across Australia until July 1928, when The Goulburn Evening Post reported that the reels had been stolen in Broken Hill. By that time, however, Adams had retired from her successful Hollywood screen career. One decade later, Claire Adams migrated permanently to Australia, stepping off the ship in Melbourne on Valentine’s Day in 1938, where there may still have been more than a few moviegoers who remembered her work. There is evidence to suggest that Claire Adams Mackinnon had a copy of both The Spirit of the Red Cross and The End of the Road with her when she came in Australia. It is not known, however, if they were shown beyond the bounds of her new home, or if those who saw them during one of her movie nights had any idea of the films’ impact during and in the aftermath of the First World War.[40]

Figure 6: Claire Adams as Justyn Reed, the fiancé of Jim Apperson, played by silent screen idol John Gilbert in The Big Parade (King Vidor, 1925). Image ©Warner Bros. Source: Author’s private collection.

Figure 6: Claire Adams as Justyn Reed, the fiancé of Jim Apperson, played by silent screen idol John Gilbert in The Big Parade (King Vidor, 1925). Image ©Warner Bros. Source: Author’s private collection.

Adams’s performance in The End of the Road drew the attention of producers in Los Angeles and New York, where they were deep in debate regarding proposed censorship measures within the motion picture industry. In 1920, independent producer Benjamin B Hampton, who had also seen her work in The Spirit of the Red Cross invited her to Hollywood to star in his productions (Lake 1922, 51). They would marry in 1924. By the time she retired in 1928 she had starred in over forty feature films alongside some of the most popular and defining artists of the era. These included Tom Mix, Rin Tin Tin, Jean Hersholt, Lon Chaney and Clara Bow. Adams’s also appeared opposite John Gilbert in his break through role in The Big Parade (King Vidor, 1925), noted in the AFI catalogue as being “frequently described as the most successful silent film of all time”. Adams played Justyn Reed to Gilbert’s doughboy Jim Apperson. Justyn is a blithe ingénue, somewhat reminiscent of Adams’s flirtatious character Nina from The Man-eater, gaily encouraging her fiancé to enlist so she can see how handsome he’d look in uniform. These characters, both Nina and Justyn, could well have been constructed from characteristics of the sexually liberated yet innocent/ignorant young women who were the target audience for The End of the Road. As such, The Big Parade is a strangely coalescent bookend for her career, which practically began and ended with blockbusting war films.

Conclusion.

The value of the Claire Adams films discussed here is lost when they are removed from their historical context. They are very much a product and reflective of the era in which they were produced, which was both defining and tumultuous. The confluence of war, the women’s independence movement and the development of the motion picture industry offered women a new degree of independence and a more active role within their own lives as well as within their culture, both as consumers and producers. This relatively small period from Adams’s career encapsulates those opportunities, as well as the threats and challenges available to women in general and within the motion picture industry in particular. She is a mediator between the realities and the fictions; a nurse playing a nurse addressing current issues both on and off-screen, lending more than a modicum of verisimilitude to the productions. Her characters in The Spirit of the Red Cross and The End of the Road were strong female protagonists, created to inspire audience members and advocate for action, encouraging all women to change their lives and in doing so play an active role in changing their world.

Amidst the international commemorations of the centenary of The Great War, it is timely to assess Adams’s life of service in front of the camera and behind the scenes, not only in Australia, but also for our allies in the conflict – Canada and the United States of America. She was, after all, a proud citizen of each country at different stages of her life. That her achievements have been forgotten is also symptomatic of the times. The role of women in the motion picture industry has been undervalued and overlooked for most of the previous century, so the “loss” of Claire Adams from film history is not unusual. The scale of public awareness generated around the superstars of Silent Hollywood, fed by the scandals, stereotyping and celebrity culture that engulfed the industry, has diminished the memory of her career, especially if it is measured and evaluated in terms of current and historic memory. My biography of Adams (in progress) explores her significance, of the difference she made as an actress and as a private citizen, and how she never lost the sense of public service first demonstrated in the choice of roles discussed here. Indeed, Adams continues to support community causes, including the Australian Red Cross, via a substantial trust established after her death in 1978.[41] By avoiding the superficiality and pressures of the studio system, Adams may have missed out on Hollywood immortality. However, her personal and professional choices demonstrate that she was not overly concerned by the vagaries of popular opinion. Her priorities were always much closer to home.

 

 

DEDICATION:

Many people in Australia, the US and Canada have contributed to this article and my research into the life of Claire Adams Mackinnon. However, I would like to dedicate this article to Mr Roger Mayer, who passed away in March 2015. Some will remember him as the recipient of the Jean Hersholt Humanitarian Academy Award in 2005. Others will for his longstanding leadership of the U.S National Film Preservation Foundation. I will always remember him as a kind gentleman who upon hearing of my interest in Claire Adams insisted I pursue it with the words “Every piece of the story is worth saving”.

 

Bibliography:

American Film Institute Catalogue. “The End of the Road.” Accessed 29 August 2014. http://www.afi.com/members/catalog/DetailView.aspx?s=&Movie=15366

“An Unusual Comedy at Vining Theatre.” Ashland Tidings. Tuesday 31 December 1918. 1. Library of Congress.

Ancestry.com, “Census records for District 15, Province of Manitoba. 1916. Census of Manitoba, Saskatchewan and Alberta. 51.”, Canadian Census Records. 1916. Accessed 27 October 2009.

Ancestry.com, List or Manifest of Alien Passengers Arriving Port of Michigan, June 1915, Sheet No. 35, Border Crossings: From Canada to U.S., 1895-1956, accessed 27 October, 2009.

 Ancestry.com, Petition for Citizenship, Claire Adams Hampton, United States of America, No. 3441, 4 February 1931, U.S. Department of Labor, accessed June 2014.

“Association Goes After “Fit to Win””. 1919. The Moving Picture World. May 24. 1141. Margaret Herrick Library, Academy of Motion Picture Arts and Sciences.

Banner, Lois W. 1974. Women in Modern America: A Brief History. New York: Harcourt Brace Jovanovich, Inc.

Barnaby Lee. The American Film Institute Catalogue. Accessed 29 August 2014. http://www.afi.com/members/catalog/DetailView.aspx?s=&Movie=2108

A Brief History of the American Red Cross, The American Red Cross, accessed 19 August 2014, http://www.redcross.org/museum/history/ww1a.asp

The Canadian Red Cross. 2014. “About the Canadian Red Cross”, Accessed 15 August 2014, http://www.redcross.ca/who-we-are/about-the-canadian-red-cross/history-frequently-asked-questions

“Claire Adams, Starring in ‘The End of the Road’ Is Real Western Canada Girl”, 1920. Saskatoon Phoenix. June 19. Library of Congress.

Colwell, Dr Stacie A., “The End of the Road: Gender, the Dissemination of Knowledge, and the American Campaign against Venereal Disease during World War I.” In The Visible Woman: Imaging Technologies, Gender and Science, edited by Paula A. Treichler, Lisa Cartwright and Constance Penley, 44-82. New York: New York University Press, 1998.

Dall’Asta, Monica and Duckett, Victoria. 2013. “ Kaleidoscope: Women and Cinematic Change from the Silent Era to Now.” In Researching Women in Silent Cinema: New Findings and Perspectives. Edited by Monica Dall’Asta, Victoria Duckett, Lucia Tralli. Women and Screen Cultures. Vol 1. University of Bologna in association with the Victorian College of the Arts, University of Melbourne and Women and Film International. 8-11.

Deer Lodge Centre. 2014. “History.” Accessed 19 August 2014. http://www.deerlodge.mb.ca/aboutHistory.html

 

Dominus, Susan. 2009. “In 1918 Flu Outbreak, a Cool Head Prevailed.” The New York Times. May 1.

“Edison Releases Flagg Social Satire Series.” 1917 Motion Picture News, Vol 16. No 26. December 29. 4591. The Seaver Centre for Western History Research, Natural History Museum of Los Angeles County.

Encyclopædia Britannica Online, 2014. “Katharine Bement Davis”. Encyclopædia Britannica Inc.. Accessed 17 Aug. 2014 http://www.britannica.com/EBchecked/topic/152802/Katharine-Bement-Davis

“The End of the Road.”1921. The Advertiser. 11 July. Retrieved from TROVE, National Library of Australia. http://nla.gov.au/nla.news-article73372048

“The End of the Road.” 1921. The Advertiser. 16 June. Retrieved from TROVE, National Library of Australia. http://nla.gov.au/nla.news-article35249538

“The End of the Road.” 1921. The Mercury. 8 June. Retrieved from TROVE, National Library of Australia. http://nla.gov.au/nla.news-article23515162

“The End of the Road.” 1921.The Northern Daily Leader. 10 February. Retrieved from TROVE, National Library of Australia, http://nla.gov.au/nla.news-article104816157

“End of the Road.” 1920. Sunday Times. 31 October. Retrieved from TROVE, National Library of Australia, http://nla.gov.au/nla.news-article120516928

““End of Road” Barred.” 1919. The Moving Picture World. July. Margaret Herrick Library, Academy of Motion Picture Arts and Sciences.

“End of the Road Barred.” 1919. Variety. 18 July. Margaret Herrick Library, Academy of Motion Picture Arts and Sciences.

“The End of the Road: a Movie of Disease.” 1920. Evening News. 3 November. Retrieved from TROVE, National Library of Australia, http://nla.gov.au/nla.news-article117303505

Goldbeck, Willis, “My Lady Claire”, Motion Picture Classic, 1920, from Mooramong Collection, National Trust of Australia (Victoria).

Hampton, Benjamin B., History of the American Film Industry from its beginnings to 1931, Dover Publications, Inc., New York, 1970, ed. Richard Griffith, Curator Emeritus, The Museum of Modern Art Film Library.

Hampton, Benjamin B., A History of the Movies, Covici, Friede, New York, 1931.

Hellier, Donna, Endnotes to “Social History Report” included in O’Connor, John and Thurley, Mooramong Buildings and Structures: Conservation Analysis Report for the National Trust of Australia (Victoria), July 1989.

“Items of Interest”, Goulburn Evening Post. Tuesday 17 1928, Retrieved from TROVE, National Library of Australia, http://nla.gov.au/nla.news-article102527976

Jackal, Susan and Millette, Dominique. Revised 2015. “Women’s Suffrage”. Historica Canada. Accessed 12 June 2015. http://www.thecanadianencyclopedia.ca/en/article/womens-suffrage/

Katz, Esther. 2000. “Margaret Sanger.” American Dictionary of Biography Online. Accessed 3 February 2015. http://www.anb.org/articles/15/15-00598.html )

Lake, J. Marion. 1922. “A Young Lady in Earnest.” Motion Picture Classic. May.

Letourneau, J. A. Rodger, “Kennedy, William Nassau”, Dictionary of Canadian Biography, University of Toronto/Université Laval, 2003. Accessed August 13, 2014. http://www.biographi.ca/en/bio/kennedy_william_nassau_11E.html.

Löhrer, Gudrun. Conference Report. Communicating Good Health: Movies, Medicine and the Cultures of Risk in the Twentieth Century. 26-27 May 2011. Brocher Foundation. Geneva/Hermance, Switzerland. Accessed 18 August 2014.

Louisiana Film History by Parish, Hollywood on the Bayou. 2015. http://www.learnaboutmovieposters.com/newsite/louisiana/Parishes/Orleans.asp

“Loyalty Will Make First Appearance At Belasco Tonight.” 1918. The Washington Times. 26 May. Library of Congress.

The Manitoba Historical Society, “William Herbert Adams”. Last modified 24 December 2009. http://www.mhs.mb.ca/docs/people/adams_wh.shtml

Marshall, Edward. 1914. “New York’s First Woman Commissioner of Correction”, The New York Times, 11 January.

Maxwell, Virginia. 2000. ‘Mackinnon, Claire Adams (1896–1978)’. In Australian Dictionary of Biography, National Centre of Biography, Australian National University, Accessed 15 August 2014. http://adb.anu.edu.au/biography/mackinnon-claire-adams-10996/text19553,.

McMaster, Christopher T., “The International Military Police and the Allied Intervention in the Russian Civil War”, Student Pulse, Vol 6, No. 04, Retrieved 18 August 2014. http://www.studentpulse.com/a?id=891

“Message to Girls Through the Film By Claire Adams”, The Vancouver Sun, May 21, 1920. Proquest.

Measuring Worth. Williamson, Samuel H. “Purchasing Power of Money in the United States from 1774 to Present.” 2013. URL: www.measuringworth.com/ppowerus/

“Opening the Road.” 1919. The New York Times. March 2. Proquest.

“Park Avenue at 40th Street, 1920s”. Digital Murray Hill from the Mina Rees Library. Accessed 2 December, 2011. http://library.gc.cuny.edu/murrayhill/items/show/271

Petition for Citizenship, Claire Adams Hampton, United States of America, No. 3441, 4 February 1931, U.S. Department of Labor, from Ancestry.com, accessed June 2014.

 Prospectus of the Grace Hospital Training School for Nurses, 1888. Grace Hospital Collection Papers 1880-1978, Walter P Reuther Library, Wayne State University.

Quiney, Linda J. 1998. “Assistant Angels: Canadian Voluntary Aid Detachment Nurses in the Great War.” Canadian Bulletin of Medical History. 02/1998. Accessed 15 August 2014.

“Recollections of ‘Girls You Know.’”1918. Motion Picture Magazine, V15, No 5, June, 78.

Rozario, Kevin. 2003. “Delicious Horrors”: Mass Culture, The Red Cross, and the Appeal of Modern American Humanitarianism.” In American Quarterly, Vol. 55, No. 3 (September). American Studies Association, The Johns Hopkins University Press, 417-454.

Schaefer, Eric. 1999. Bold! Daring! Shocking! True! A History of Exploitation Films, 1919-1959. Durham: Duke University Press.

Schlüpmann, Heide. 2013. “An Alliance Between History and Theory.” In Researching Women in Silent Cinema: New Findings and Perspectives. Edited by Monica Dall’Asta, Victoria Duckett, Lucia Tralli. Women and Screen Cultures. Vol 1. University of Bologna in association with the Victorian College of the Arts, University of Melbourne and Women and Film International. 13-26.

Slide, Anthony. 1996. The Silent Feminists: America’s First Women Directors. London: The Scarecrow Press, Inc.

Spartacus Educational. “James Montgomery Flagg.” Accessed 14 August 2014. http://spartacus-educational.com/ARTflagg.htm

“Spirit of the Red Cross.” 1918. The New York Times, 21 April. Proquest.

“Splendid Work of American Red Cross Society Graphically Told in Series of Single Reel Motion Pictures Now Ready for Screen”. 1918. Exhibitors’ Trade Review. December 7. Vol 5. No. 1.

Stamp, Shelley. 2012. “Women and the Silent Screen”. In The Wiley-Blackwell History of American Film. Edited by Cynthia Lucia, Roy Grundman, and Art Simon. University of Santa Cruz. Accessed 10 January 2015.

Theatre advertisements. 1919. The Youngstown Daily Vindicator. 4 April. Library of Congress.

“V.A.D Hospitals in Northumberland and Durham 1914-1918.” Donmouth Local History Society. Accessed 18 February 2015 http://www.donmouth.co.uk/local_history/VAD/VAD_hospitals.html

“What People are Saying and Doing.” 1921. Table Talk. 10 March. Retrieved from TROVE, National Library of Australia. http://nla.gov.au/nla.news-article146718304

Will, Beverley. 1978. “Gallant and Romantic were the days of Claire Adams.” Green Place. October. Mooramong Collection, National Trust of Australia (Victoria).

The World War 2 US Medical Research Centre, http://www.med-dept.com/articles/venereal-disease-and-treatment-during-ww2/ Accessed 29 January 2015.

Unpublished sources:

  • Hone, Geoff. (Chair of the Scobie and Claire Mackinnon Trust.) Email to author. June 2014.
  • Miley, Nancy. Adams Family History, Unpublished Manuscript. Author’s private collection.
  • Schmeling, Kathleen Emery. (Archivist) Grace Hospital Archives, Wayne State University. Email to author, May 21 2014.
  • Speed, Dr Lesley. “‘In the best film star tradition’: Claire Adams and Mooramong”, Screening the Past, due for release 2015. http://www.screeningthepast.com/
  • Sweet, Jill and Stephens, Barbara Adams. Oral History Transcript, Author’s private collection.

Filmography:

  • Caneva, Lina, 2009, Mooramong – Private Hollywood, Caneva Media Productions.
  • Eaton, Jack, 1918, The Man-eater, Thomas A. Edison, Inc.
  • Griffith, Edward H., 1917, Scouting for Washington; a tale of the Revolution, Thomas A. Edison, Inc.
  • Griffith, Edward H., 1917, Shut Out in the Ninth, Thomas A. Edison, Inc.
  • Griffith, Edward H., 1919, The End of the Road, American Social Hygiene Association, Famous Players-Lasky Corp & US War Department.
  • Vidor, King, 1925, The Big Parade, Metro-Goldwyn-Mayer.

 

Notes:

[1] Adams’s petition for US citizenship document confirms that she was a subject of the British Dominion of Canada and that her last foreign address was in Toronto. From there she entered the US permanently at Detroit on 12 June, 1915. She first declared her intention to become a US citizen in September 1922, two years before she married Hampton. Her petition for citizenship was granted in Poughkeepsie on 4 February 1931, having resided in Pawling, NY since 1 September 1928.

[2] Willis Goldbeck in Motion Picture Classic (1920) described Adams as “instinctively Britishly reserved, a person whom one cannot hope to know in a day, or a month.”

[3] As of this date, I have yet to find Claire Adams on the cover of either fan or trade magazines, though several profile interviews and letters to editors indicate she had a fan base. Without a studio publicity department behind her, and as part of a company that promoted all-star casts, she was able to enjoy a more private life and manage her own profile accordingly, hence my concept of her as an independent artist.

[4] Originally published as A History of the Movies, by Covici, Friede, New York, 1931, Hampton’s work was reissued in 1971 as History of the American Film Industry from its beginnings to 1931, edited with an introduction by film historian Richard Griffith.

[5] In his introduction to the second edition of Hampton’s book, Griffith asserted “That it is the best history of the movie business to date there can be no doubt’, and positions it in terms of authenticity and objectivity above Terry Ramsey’s A Million and One Nights (1926) and Lewis Jacobs’ The Rise of the American Film (1939).

[6] As described by Monica Dall’Asta and Victoria Duckett in “Kaleidoscope: Women and Cinematic Change from the Silent Era to Now.” 2013, 8.

[7] At only fifteen years of age, Gerald ran away to join the Scottish Highlanders, but was dragged back to London by his Great Aunt Mabel Adams with whom he was staying. As soon as he was old enough he enlisted, but had only just completed training when the war ended. He never saw any action on the front. Nancy Miley,“Gerald Drayson Adams (1900-1988)” from Adams Family History Notes, Unpublished Manuscript.

[8] This prospectus was written late in the 19th century. The Grace Hospital School of Nursing (GHSN) offered two years of vocational (if somewhat rudimentary) instruction during which time students would be under constant supervision, on and off duty. Students were permitted one afternoon off per week and were encouraged for the sake of their own health to spend one hour each day in the open air. They were allowed two weeks leave per year, one evening off per week and required permission from the school’s Principal to be out later than midnight. In return, the students would receive training in elements of hospital work, namely the dressing of wounds and burns, applications of fomentations and poultices, making beds and methods for avoiding bedsores.

[9] Adams’s name on all official documents was her birth name, Beryl Vere Nassau Adams, but this does not show up on the list of graduates either.

[10] According to the Red Cross Canada website, “Red Cross nurse” was a term applied to anyone volunteering in a hospital in any capacity, including letter writing and visiting. Millions of young women contributed to this international effort and there was simply not the capacity to establish or maintain complete records of every volunteer. “About the Canadian Red Cross”, Accessed 15 August 2014, http://www.redcross.ca/who-we-are/about-the-canadian-red-cross/history-frequently-asked-questions

[11] A New Set of Needs, from the website of Veterans Affairs Canada, describes the inadequate hospital infrastructure, and venues converted to repatriation hospitals following England’s requests to the Dominions to take care of their own wounded. Accessed 11 January 2015, http://www.veterans.gc.ca/eng/steannes-hospital/about-us/history

[12] According to the American Film Institute Catalogue entry, Barnaby Lee was probably not widely released. http://www.afi.com/members/catalog/DetailView.aspx?s=&Movie=2108

[13] Scouting for Washington, Wild Arnica and Shut Out in the Ninth were viewed by the author at the Library of Congress, Washington D.C.

[14] Flagg asserted that his Uncle Sam became “the most famous poster in the world”. As quoted on the American Treasures of the Library of Congress website, accessed 13 February 2015, http://www.loc.gov/exhibits/treasures/trm015.html

[15] The Man-eater was viewed by the author at the Library of Congress, Washington D.C.

[16] The other lead roles in the “Girls You Know” series were played by Dorothy Wallace, Mary Arthur, Martha Mansfield and Peggy Hopkins. According to IMDb, Hopkins had appeared in 2 Columbia productions. Dorothy Wallace had one previous film credit. Martha Mansfield had four previous motion picture credits and seemed destined for a successful career. In 1920 she starred opposite Lionel Barrymore in Dr Jekyll and Mr Hyde (John S. Robinson, 1920). Mansfield died in horrific circumstances in 1923 after her period costume caught fire on set. See IMDb http://www.imdb.com/name/nm0543806/bio?ref_=nm_ov_bio_sm

[17] Banner suggests that it was considered “a disgrace for a woman’s name to appear in public print” though this began to change before the war (20).

[18] According to Nancy Milley, her father Claire’s Uncle, Ernest Adams and her Grandmother, did not approve of Claire’s profession, even into the 1920s when she was at the height of her career. “Ernest Dupin Adams” from Adams Family History Notes, unpublished manuscript, author’s private collection.

[19] Several scenes were shot in the Jackson Barracks in New Orleans, which was in use as a training and processing facility for the United States Army. This suggests that recruits may have been used as extras in the film, perhaps to add a layer of verisimilitude to the scenes of battle and troop movements. From website Hollywood on the Bayou, Louisiana Film History by Parish, http://www.learnaboutmovieposters.com/newsite/louisiana/Parishes/Orleans.asp

[20] The Red Cross War Council subsequently established a “Bureau of Pictures” in early 1918 “to tell upon the screen the splendid story of the Red Cross”. These single reel shorts featured such titles as “Broken Lives”, “Victorious Serbia” and “Russia: a Land Worth Saving”, reflecting their ongoing work in Europe and with veterans up to and following the Armistice. From “Splendid Work of American Red Cross Society Graphically Told in Series of Single Reel Motion Pictures Now Ready for Screen”. 1918. Exhibitors’ Trade Review. December 7. Vol 5. No. 1.

[21] In the El Paso Herald, for example, the proprietors of the Grecian Theatre placed a large advertisement, paid for by another local business, outlining a special “Red Cross War Fund Benefit”, from which they would donate all proceeds from the evening’s entertainment to the cause. Patrons were encouraged to “make your quarters jingle” at the box office as “Lives Over There” depended on their support. Wednesday 15 May 1918, 5.

[22] The release of the film was also accompanied by a nation wide door knock appeal (American Red Cross, 2014).

[23] The World War 2 US Medical Research Centre estimates that during the Great War, venereal disease “had caused the Army lost services of 18,000 servicemen per day.” http://www.med-dept.com/articles/venereal-disease-and-treatment-during-ww2/ Accessed 29 January 2015.

[24] As described by Banner, early feminists saw male sexuality as being at the heart of female oppression. Even before the war, early feminist Inez Milholland wrote that “we are learning to be frank about sex … and through all this frankness runs a definite tendency toward an assault on the dual standard of morality and an assertion of sex rights on the part of women” (116).

[25] According to Schaefer, “Discourses on venereal diseases and eugenics were so tightly intertwined as often to be inseparable” (21).

[26] The irony is that throughout the nineteenth century, prostitutes were seen as a necessary evil in order to protect the virtue of pure women. The North American sex industry operated in relatively structured or semi-licensed conditions where the workers were regularly checked for STDs (Banner 1974, 76). The change came in the early twentieth century when sexually active young women identified sex with strangers as a possible way out or momentary escape from the drudgery of their working class lives. If they accepted payment for sex, they could earn two to three times more than they could as a domestic servant or salesgirl (Banner 1974, 81).

[27] Schaefer laid out the confluence of concerns that surrounded the spread of STDs in American society, which included Eugenics, Industrialization, Abstinence and Prohibition. The constituents incorporated under the ASHA banner represented most of these concerns. Progressives were concerned about the political and social boundaries being blurred by the urban, industrial and technological developments of the early Twentieth Century (18 – 23).

[28] According the Katz, Sanger had already experienced exile from the US in 1914 when in 1916 she was arrested and imprisoned for opening the nation’s first birth control clinic. In 1917 she had also made a motion picture titled “Birth Control” which had been confiscated by New York authorities.

[29] Fit to Win was shown at training camps where a lecturer would be available to explain some of the finer points of the storyline and address any medical questions.

[30] The film’s writer, Katherine Bement Davis, had already conducted extensive research into the possible impact of sex education on young women, which contradicted prevailing Freudian based theories that too much knowledge of sexuality and its consequences could permanently damage young women (Colwell 1998, 60).

[31] In 1914 Bennett made his movie debut with Damaged Goods, a film adapted from the theatre script that “pictures the terrible consequences of vice and the physical ruin that follows the abuse of moral law.” Bennett had married his co-star from Damaged Goods, Adrienne Morrison, with whom he had three daughters, two of which would go on to major Hollywood careers: Joan and Constance Bennett.

[32] Banner describes Davis as a leading figure of the Progressive movement, instigating major studies and reforms in the fields of urban poverty, women’s prisons and sex education (Banner 1974, 98). The New York Times ran a major profile of Davis when she was named New York’s first female Commissioner of Correction in January 1914. See Edward Marshall’s “New York’s First Woman Commissioner of Correction”, The New York Times, 11 January 1914.

[33] The location where these scenes were shot is mentioned in the notes section for the catalogue entry for The End of the Road in the American Film Institute Catalogue accessed 29 August 2014, http://www.afi.com/members/catalog/DetailView.aspx?s=&Movie=15366. Schaefer describes how many of the more confronting scenes of open wounds were cut in order to please the censors and keep the film in theatres, but some scenes were later reintroduced by exploitative distributors (29-30).

[34] The End of the Road was viewed by the author at the Library of Congress, Washington D.C.

[35] The issues facing the film were compounded by the film industry’s wish to appear respectable and beyond moral reproach in the face of a growing amount of sex scandals and exploitation films (Schaefer, 30). There was also an acceptance of sex education taking place within schools, making the role of sex education films in public cinemas largely redundant (Colwell, 69).

[36] According to Colwell (71), there were around ten other films addressing the spread of STDs, however they depicted women as the source of infection and transmission. The End of the Road differs significantly by drawing attention to the complicity of men.

[37] Schaefer notes that Damaged Goods (Thomas Ricketts, 1914) also starring Richard Bennett was the first to have established this premise on film. Damaged Goods reinforced the claim that venereal diseases were the scourge of the lower classes and inflicted upon young men of means and family during moments of weakness caused by drunkenness or deliberate seduction by fallen women from the lower classes (23).

[38] The copy viewed by the author in the Library of Congress was in Dutch, indicating it was intended for audiences in the Netherlands and other Dutch-speaking colonies.

[39] This section could not have been conducted without the use of Trove, the National Library of Australia’s online database of Australia’s newspapers. The articles directly referenced are noted in the above reference section. http://trove.nla.gov.au/newspaper/result?q=The+End+of+the+Road

[40] The Mooramong Buildings and Structures: Conservation Analysis Report for the National Trust of Australia (Victoria) of July 1989 contains a list (Appendix A) of 48 motion pictures “compiled from film reels held at Mooramong”. The fate of these films is unknown.

[41] According the Geoff Hone, the Scobie and Claire Mackinnon Trust have donated approximately AU$2 million to the Royal Children’s Hospital as well as other significant amounts over the years to the Australian Red Cross.

 

Bio: Heather L. Robinson is a Research Associate & PhD Candidate in the School of Humanities and Creative Arts, Flinders University. She is also an Honorary Research Associate (History) at the Los Angeles Natural History Museum.

‘Rock‘n’roll’s evil doll’: the Female Popular Music Genre of Barbie Rock – Rock Chugg

$
0
0

Abstract: Fostering male tradition in popular music, rock’n’roll history often underrated the early Girl Group chart-topping era of 1958-63 after Elvis and before Beatlemania. By the 1990s-2000s, Riot Grrrl and Girl Power success was again devalued by that homosocial music scene. Beset by neoliberal managerialism, even academic and market research played it safe, recognising corporatist Indie and nationalist Brit-Pop, while Riot Grrrl revolt into Girl Power style components of a new female genre went unrecognised. Consecutive social exclusion (Riot Grrrl) and social capital (Girl Power) factors in the sound, dubbed Barbie Rock from stereotyped songs, like global hit ‘Barbie Girl’ (1997-8) were intensified by the shifting role of primary and secondary definers in digital media. For popular music, such shifts included 1) rock-press computerisation; 2) moral panic news; and 3) video monopoly. Illustrated with quotes from Barbie Rock fanzine Polymer, the paper culminates in an in-between Riot Grrrl and Girl Power case study of Barbie Rock front-woman, Caroline Finch. The high profile of female pop music today (‘rock’n’roll’s’), although demonised (‘evil’), confirms the ongoing influence of this 1990s genre on a now digitalised era of the plasticised body (‘doll’).

        

Barbie, Barbie, still in her teens
Bell of the parties, a tom boy in jeans

                                      ‘Barbie’ – The Beach Boys, 1962

Figure 1: Polymer News 

Figure 1: Polymer News

Introduction: ‘rock’n’roll’s evil doll’[1]

Apparently nothing new happened in 1990s music. ‘I shall not be discussing new genres’ says genre expert Negus, ‘this would require the lucky researcher to be in the right time and place to chart their emergence’ (1999: 29). If we take the risk society seriously, that ‘time and place’ has disappeared into the virtual reality of an information age. Even postmodern criticism is unsure of the extent to which contemporary ‘art’ is ‘mediocrity squared. It claims to be bad – “I am bad! I am bad!” – and it truly is bad.’ (Lotringer, 2005). The prevailing view finds entire cultural scenes, let alone popular music, in lockstep vicious recycle mode. Yet a revolution turned counter-revolution, keen to dismiss new female music as dissipated ‘victim-babes’ (Greer, 1999) or incongruous as ‘a camel on a bicycle’ (Raphael, 1995), is met with ‘Riot Grrrl’ resistance (Riley, 1994) and ‘Spice Girl’[2] dissent (Lumby, 1998). While self-exclusion glossed by token inclusion maybe the virtual failure of ‘Barbie Rock’ (see part 4 below), what men don’t know and Grrrl Power understands is that for rock‘n’roll this is also its actual success.

My illustrative data for the Barbie Rock genre was captured in opened-ended research from primary sources. Tanner endorses musical genres as ‘better addressed with more qualitative research’ (2008: 189). In this case over fifty musicians, writers and experts were interviewed in Polymer magazine between 1997 and 2002. Semi-structured and tailored questions, snowballed from celebrity participants, drew diverse reactions to the sexed genre, ranging from fanatical gusto to detached cool. Attentive to notoriety, it was considered true to their initial fanzine sample to ‘spell out’ the names of participants, with reference to Bourdieu’s precedent (1988: 278). Artefact of intense independent and mainstream media activism (fanzine to glossy magazine), Barbie Rock integrated original low fi ‘potty mouthed’ Brat Mobile Riot Grrrls with high tech ‘wanna be’ Spice Girls Power. Théberge corroborates magazines, ‘as a central element in the “framing” of popular musical forms’ (1991a: 271).

Formulated in a context of hung-parliament cartels, the notion of ‘social exclusion’ unites Blair’s New Labour to Giddens’ Third Way, linking a potential for improved poverty studies with criteria like ‘non-participation’. In practice, contradictory accounts noted the social wage benefiting low-income groups while high-income group levels fell (Bradshaw, 2004: 173). Or alternately, loss of the public realm, social individual and democracy based collective provision (Hall, 2005: 328). Predictably, events like increased economic polarisation and exclusion,[3] uncorrected by token multicultural inclusion or mutual obligation, largely confirm the latter view. Less a non-participatory than anti-intellectual check on popular culture, utopian neoliberal policy moved business cycles away from subversions dear to rock-press turned academics (Reynolds, 1988), now co-opted in education rehab. Stale formats from industry models of 1980s ‘Indie’ underculture (Hesmondhalgh, 1999: 35) consoled post-genre quietism accompanied by neutralising trivia, like the TV ‘rock quiz’. Against these general factors of resistance to new genre subcultures Hesmondhalgh concedes particulars, like an ‘increased policing of copyright, in ways that have favoured the oligopolistic corporations dominating cultural production, actually inhibits creativity rather than promotes it’ (2007: 88). Such life experience pressures on primary and secondary media definers, overriding female music innovation, displaced a normally creative function of both the rock-press and musician alike:

No, rock’n’roll has no future, absolutely not! – Craig N. Pearce, journalist (Quatro)[4]

Sorry, are like you suggesting that the Paradise Motel can be referred to as ‘Barbie Girl’? – Merida Sussex, Paradise Motel (Stone)

Figure 2: Polymer Stone

Figure 2: Polymer Stone

Addressing the theme of social exclusion directly, Bayton itemised ‘“constraints” facing the potential female musician’ as ‘material’ (money, equipment, transport) and ‘ideological’ (hegemonic masculinity and femininity). ‘What was interesting was the way in which women are able to overcome or evade…exclusion’ (1998: 189). Interestingly, she suggested ‘escapes’ or ‘resistances’ (role models, feminism and lesbianism) centred on theories of symbolic interactionism. Contextualising the argument further, here I suggest that the social ‘exclusion’ of significant genre experience was determined not only by the classic sexism targets of feminism, but the refractions of popular music by new technology. Explored below, these are described as institutional driven rock-press computerisation; moral panic reportage; and video monopoly. Based on ideas from Baudelaire to Bataille, such shifts are haunted by Baudrillardian ‘evil’ or demonisation of women (‘Lilith’). Whether already noted as industry/culture problems of ‘production’ (Negus, 1998), or ‘commodification’ by the male ‘producer/record company/music business’ (Stras, 2010: 3) this sexed nexus also facilitates the first female genre of popular music.

I bought her a full length Barbie silver fox,

But she just lies in her Barbie box

                     ‘Barbie’ – Shower Scene From Psycho, 1986

 

1 Rock-press computerisation: speak no evil

Founded on the ‘existence of valuable relationships’, Bradshaw argues that ‘social capital does not seem to be particularly related to poverty, possibly because the poor have more time to maintain them’ (2004: 184). Cultural industry data shows that most ‘genre’ pop artists live on low incomes in semi-poverty, buoyed by a successful ‘star system’,[5] comprised of marketing formats to counter unpredictable sales. However, the raw material of music is exchangeable form, not ‘industrial’ content or ‘consumer’ style. The expert or ‘primary definer’ (Hall et al., 1978) and management or ‘entrepreneur’ (Martinelli, 1994), value-add to these music forms recorded from studio or live performance. In official male-dominated pop decades from the 1950s for example, rock journalism (‘hacks’) in this role credibly claimed to operate outside vested economic or political interests. But since the professionalisation of writing or ‘routinisation of innovation’ (Martinelli, 1994: 480) effect of internet technicism (‘hackers’), biopower fragmentation and niche marketing have led to social exclusion and music sales decline (Mathieson, 2006). Factors perpetuating a homosocial trend of ‘exclusion of femininity from rock’ (Davies, 2001).

Both the ‘corporate strategy’ focused concept of Indie genres (Hesmondhalgh, 1999), and academic ‘theory’ of genres (Fabbri, 1980: 6) overlook the creative role of primary definers or cultural intermediaries (Negus, 1999: 18). For Hesmondhalgh, Bourdieu’s ‘cultural intermediary’ concept is ‘confusing and unhelpful’ (2007: 67). Harley and Botsman’s ‘No payola and the cocktail set’ examined this function in the heyday of a pre-internet, extra-institutional and under-theorised rock-press (1982). After three decades of rock’n’roll genres, the 1980s academic finally itemised this journalistic discourse of ‘hacks’ as downplaying the ideological and commercial outcomes of popular music writing in terms of airplay, sales and popularity (‘historicising, idolising and posing’). For these researchers, the Sex Pistols were approved on ‘tactical’ grounds, unlike romantic ‘Punk’ recycling charismatic ideology of race-music Rockabilly and vitalism Psychedelia. Punk hagiography only repeated this personality cult logic of the 1950s and 1960s. Ensuing 1980s Post-Punk parent versus subculture readings were also seen as trapped ‘within an overworked and useless construction of power’ (1982: 252). However, Harley and Botsman’s reservation about an entrepreneur rock-press is finally offset by the self-conscious reflexive method of Londoners Paul Morley and Ian Penman. In journalism similar to Australian Craig N. Pearce or American Lester Bangs, this rock-writer-as-intellectual as its vital media illusion preceded the internet techicism Diaspora.

According to sociology, ‘the weakening of the role of the innovative entrepreneur is seen as a basic factor, although not the only one, of the crisis of capitalism’ (Martinelli, 1994: 479). Conversely, cultural studies visualise the role of institutional ‘primary definer’ as ultimately vested by, ‘branches of the state and its fields of operation – through the formal separation of powers; in the communications field it is mediated by the protocols of balance, objectivity and impartiality’ (Hall et al., 1978: 220). With a variable supply of social capital, stringer to freelance rock-press journalism shuttled between these commercial entrepreneur to state primary definer roles. ‘Definition of rock journalism: People who can’t write, doing interviews with people who can’t think, in order to prepare articles for people who can’t read’ (Zappa and Occhiogrosso, 1990: 221). Recognised as its highpoint, the contrasting 1980s reflexive rock-press ushered in, ‘the progressive decay of the entrepreneurial function by virtue of the routinisation of innovation’ (Martinelli, 1994: 478-9). Although trivialised as ‘thin cultural studies’ (Beilharz, 1995: 133) and ‘the cultural turn’ (de la Fuente, 2007: 120), while it lasted, the professionalisation transition that succeeded this flashy journalism did actually deliver some valuable insights into popular music.[6]

Rather than ‘romantic Punks’ or ‘parent versus subculture’, with ‘increasing reliance on “expert” opinion’, today’s ‘professional’ tends to overlook the ‘popular consciousness’ (Abbot-Chapman, 2007: 242). Opposite of the ‘sociologism’ that claims a ‘genre community’ is ‘always…conscious of their precise role in musical reality’ (Fabbri, 1980: 6), this rock-press devaluation thwarts participants from self-defining their primary knowledge based on lived experience. Thus routinised female music goes ‘unreported in the [rock] press’ (Bayton, 1998: 78):

Barbie Rock! Very good – what’s that? – Caroline Kennedy, Dead Star (Hex)

Sure, if they want to call it Barbie Rock, then call it Barbie Rock! – Ian Meldrum, journalist (Hex)\

In the 1990s, the professionalisation or ‘bureaucratisation’ (Brett, 1991) of writing was driven by an internet technicist computerisation overseen by university research. Unlike New Zealand, Australian sociology is university centred (Germov and McGee, 2005). Bayton refers to radio Disc Jockeys as the significant rock‘n’roll gatekeeper. But the ‘hegemonic’ male journalism she identifies (1998: 3) is today fuelled by an ex-Cold War internet over-researched as computerisation. On it women are routinely devalued in ‘sexual’ terms. By arguing ‘we can no longer speak Evil’ (or a demonised female music genre), Baudrillard refers to this computerisation (2009: 97). In the above quotes from Caroline or Ian, the effect is validation doubt about a devaluated genre experience (cf. ‘degenerate’ modernist art militarily repressed until 1945 (Bradbury and McFarlane, 1976)). In Australian contemporary journalistic and scholarly studies of rock‘n’roll appearing after Riot Grrrl and Girl Power, the female genre is not acknowledged. The authors paid their dues in the 1990s street press (Mathieson, 2000), 1980s news media (Breen, 1999) and 1970s rock-press (Walker, 1996). While Mathieson joined the corporation, describing the ‘Indie’ music scene interestingly as a ‘Sell In’,[7] Breen and Walker were later to hitch their stars to the university. Contrasting this commercial or institutional cooptation, the new as yet unidentified genre would remain viable if seen ‘as a tacitly condoned mechanism of Subversion and foil to State control’ (Chugg, 1989: 64).

She’s very smart,

She can dance well,
Bang,
bang, bang,

Twist Barbie

’Twist Barbie’ – Shonen Knife, 1992

Figure 3: Polymer Inner City

Figure 3: Polymer Inner City

2 Moral panic news: hear no evil

Unlike genetic or animal research for drug based solutions of ‘social inclusion’ (Bonner, 2006: 4), while noting such ‘behaviourist ideological baggage’, Bradshaw argues cogently that ‘social inclusion is not necessarily the opposite of social exclusion – though the emphasis of the state as agent is welcome’ (2004: 184). But like privatisation creep, such interventions can cause social exclusion. For example, Hubert shows how technology is ‘creating a new category of socially excluded children’, also suggesting that ‘people who are medically cured in Western terms may not return’ from ‘social death’ (2000: 3-4). In Australia, this logic is demonstrated by a high-incidence of child abuse in the whole community on the one hand (Herald Sun, 2007); and the indigenous peoples nominated as scapegoat on the other. Clearly such ‘relegation of people to nature’ behaviourism appears a ‘way of legitimating exploitation and exclusion from civilised society’ (Hubert, 2000: 5). The initiating moral panic debuted on television (ABC, 2006) amplified by the press (Age, 2006) then military intervention (Age, 2007), eventually earned under-reported opposition from the UN. Similar disproportionate media reaction was already noted in the mid-90s as, ‘moral panic in response to rock and roll more generally’ (Grossberg, 1995: 368).

Developing the work of Cohen (1973), Hall et al. (1978: 222) frame ‘moral panics’ as a crisis of consent or ‘hegemony’, at times escalating into ‘general panic’ when, ‘all dissensual breaks in the society’ are perceived as threats to ‘law and order’. Media definers, ‘play a crucial but secondary role in reproducing the definitions of those who have privileged access, as of right, to the media as “accredited sources”’ (1978: 58). Two decades later, McRobbie and Thornton argue that subculture and genre ‘marketing strategy’ functions of moral panics are ‘priceless PR campaigns’ (1995: 565). However, unequal readerships of large circulation dailies compared to limited distribution fanzines contradict this thesis. The experience of fanzines, like Polymer, Thunderpussy, or Riot Grrrl is a case in point. Each gained only a small circulation despite huge music chart sales. Yet by including such demonised minority ‘folk devils’, moral panics present a smallest number utilitarian calculus tailored to the ‘biopower’ set theory of primary definers (Foucault, 1981). These xenophobic primary definitions of sex, race, class or age bring ‘life and its mechanisms into the realm of explicit calculations’, submitting ‘life integrated into the techniques that govern and administer it’ (1981: 143). McRobbie later refits this ‘biopolitical strategy’ set theory (2007: 730) for post-feminist ‘Top Girls’ to deconstruct revived sexism in Blair’s New Britain. Conversely, ‘Barbie Rock’ is a way of subverting stigmatising stereotypes, like ‘virgin, mother, and prostitute’ (Stras, 2010: 4):

That’s why heavy metal’s become so popular, because it’s tribal. Female rock stars they come and they go, but they never seem to leave a mark – Dr Pepper, journalist (Stone)

I actually agree with you. I think that there is more interesting, the most interesting bands at the moment are either female led or have got girls in them – Justine Frischman, Elastica/Suede (Hex)

The measure of women’s significant success in rock‘n’roll, a female genre was already anticipated as far back as the 1980s. Saxophonist Louise Brooks of critically acclaimed group The Laughing Clowns believed, ‘barring incredible turns to the right, that is a trend that will continue. Quite possibly, women conceptualise differently, but if so, it’s quite invisible’ (Shien, 1987: 84). Yet key events, like a ‘Women in Rock’ issue of Rolling Stone (1997) gave the nod (if seen as ‘pitifully small’ after Destiny’s Child and The Spice Girls’ triumph – Stras, 2010: 5) while dodging the genre. In the bigger picture, a ‘social inclusion’ military Intervention on demonised aboriginals (lower socio-economic class ‘social death’ by a discredited ‘race’ scientism) confirm ‘incredible turns to the right’ fears. Unjustly blamed for the larger ‘white’ society in denial. Similarly, media sexism that reduces women musicians, for example to ‘glamour shots’ (Bayton, 1998: 14) deactivates their female popular music genre. The quoted biopolitical doubts of David, Justine and Louise are vindicated. For Baudrillard, the evil demon of media images occurs as a ‘precession’ of the real by models. These ‘invert the causal and logical order of the real and its reproduction’ (1984: 13). For instance, if not for homage to aboriginal Buried Country (Walker, 2000), 1990s Barbie Rock assimilated to 1970s Punk by Australia’s ‘pre-eminent critic and historian’ with inaccuracies like ‘1991: The Year Punk Broke’ (Walker, 1996: 280) may also have telescoped his oeuvre into demonisation.

I’m a Barbie girl in the Barbie world,

Life in plastic it’s fantastic

‘Barbie Girl’ – Aqua, 1997

Figure 4: Polymer Thing

Figure 4: Polymer Thing

3 Video monopoly: see no evil

According to the social theory of Daly and Silver (2008), social capital and social exclusion can be at once opposites (rather than inclusion) and interchangeable; one the antidote (to the other) and both merged into a continuum. Ultimately, ‘social capital may actually increase social exclusion’ (2008: 556). Resisting assimilation, they also retain the distinction between cultural studies (social capital) and sociology (social exclusion). In social practice, Bourdieu’s example of a relational ‘hiatus’ between the ‘statutory expectations’ of déclassé academics with devalued social capital and lack of ‘opportunity’ leading to social exclusion, explains the student unrest of ‘May 1968’ (1988: 163). Baudrillard’s cultural example is the ‘virtual’ music ‘restored to technical perfection’ by excluding ‘noise and static’ that has to reinvest in some noise to restore musical capital (2007: 28). Bayton finds such ‘a relational hiatus’ in Riot Grrrl technophobia of live and studio ‘technical skills’ restored by technophile training (1998: 7). Reynolds and Press similarly describe Riot Grrrl creativity as limited to music ‘content’ that excludes ‘form’ (1995: 187). This polarisation of social exclusion and social capital coheres as the limit of lo fi Riot Grrrl technophobic content, opposed to hi tech Girl Power technophile form and interposed by a relational hiatus of Barbie Rock technical skills. Like ‘social’ media adapted to female ends, the genre of 1990s rock‘n’roll lies here.

The horizon of all popular music, in Jazz (for Riot Grrrl, Tracy Chapman’s roots music) sociologists argue that, ‘objects do not possess sociality, people do, and it is through the embodied nature of inter-subjective human social action that objects come to have contingent relevance’ (Gibson, 2006: 185). Gibson locates this relationship between normative frameworks of performance and limiting parameters of musical instruments, creatively minimised by musicians who reach sufficient degrees of technical facility for improvisation. Contrarily, for cultural studies, ‘the technical mastery of space and time contributes not only to the rationalisation of musical production, but also to the creation of a myth of community’ (Théberge, 1991: 110). Rather than inter-subjectivity, Théberge highlights both the simulation of community through increasing spatial rationalisation of audio material in separation recording, and the star-system driven and cost efficiency control of overdubbing (for Girl Power, Destiny’s Child’s dance music). In this mode, creative improvisation is reconstructed in the studio, refuting the classless ‘myths of technology’ technicism of ‘McLuhanesque’ leftists (1991). Today this digitally anatomised community, when ‘social control’ over women and ‘sexist jokes abound’ (Bayton, 1998: 6), has again been profoundly transformed by, ‘the predominance of music videos in the marketplace’ (Théberge, 1991: 109).

Banks traces the ‘incorporation’ of live and recorded popular music into a video monopolised ‘market place’ back to the arrival of privatised cable channel, MTV (1998: 293). This raised a small 23% percentile of top 100 Billboard acts with videos in 1981 to 97% by 1989 (295). Hesmondhalgh confirms the 1990s transition to a two-dimensional ‘stabilizing pop mainstream oriented towards video promotion, and synergies with visual mass media.’ Like Polymer participants, Indie labels were ‘determinedly against these commercial methods’ (1999: 38) for ‘trivialising them, and dealing with them solely in terms of their physical attractiveness’ (Bayton, 1998: 25):

It’s OK to play music if you’re a beautiful girl, and I felt like it was getting a bit too much of that image – Laura McFarlane, Sleater Kinny/Ninety Nine (Vee)

I think I’d be comfier with ‘Barbie Rock’ if it turned out that really you were talking about the strange new breed of boys that seem to have no hormones and no sperm count – Paul Morley, journalist (Hex)

Reviving 1960s Psychedelic versus 1970s Punk role-set conflict and simulating slick TV advertising, by the 1990s visual clips had replaced live tours as the means of self-promotion, modelled on performance (‘authentic’) and concept (‘synthetic’) formats now feted in annual video awards (Banks, 1998: 295). With privatisation creep prioritising money-making visuals over musical talent (303), the ‘hot’ 3D sound of radio was digitised into ‘cool’ 2D sight of TV, unreceptive to the Barbie Rock limit genres of Riot Grrrl revolt into Girl Power style. Grossberg reads the ‘hip attitude’ of TV as a ‘refusal to take anything…seriously’ (1995: 376), where the ‘explicit conjunction of images and songs seems to multiply the possibilities of interpretation’ (370)[8]. The neutral mood, affect or emotion standing apart from ideas. A neutrality Breen (1999) ascribed to the Girl Power pop of Kylie Minogue, supposedly eclipsed by Midnight Oil’s Brit-Pop style nationalism.[9] ‘Where rock was considered to rely on a set of established practices based on musicianship and a relationship to audience, pop was a disposable image of little lasting value’ (1999: 67). But if TV ‘screens out’ new genres or sexual inequality evils (Baudrillard, 2007: 78), it also facilitated free to air music in Australia. Albeit excluded in late-late night programs, like the ABC’s Rage or ‘youth’ radio JJJ biopower. At least complete ‘commercialisation’ (Hesmondhalgh, 2007: 305) failed to eventuate.

Kids don’t like me,

Moms are mad,

I’m going off the market,

‘Cause I look so sad

’Bitterness Barbie’ – Lunachicks, 1995

Figure 5: Polymer Vee

Figure 5: Polymer Vee

4 Barbie Rock and sexism: beyond good and evil

Social exclusion is a ‘multidimensional concept’ (Bonner, 2006; Hubert, 2000), that tends to ‘deflect attention from ever-increasing income inequality and class conflict’ (Daly and Silver, 2008: 554) or poor social capital. Poverty measures of ‘physical’ or ‘financial’ capital are enhanced, social exclusion theory argues, by non-participation indicators of ‘discrimination, chronic ill health, geographical location or cultural identification’ (Burchardt et al., 2002: 6). Women own one per cent of ‘physical’ titled land and are seventy per cent of the world’s ‘financial’ poor (MX, 2008)[10], making ‘cultural identification’ (in this case ‘genre’) indicators more relevant, in the context of popular music. Frith (2001: 46) and Hesmondhalgh (2007: 23) define genres as a record label method or ‘format’ for coping with risk. The ‘irrational’ behaviours of audience ‘taste’ or artistic ‘talent’ in the ‘star system’. Similarly Breen (1991: 193-4) refers to a ‘pre-existing system’ of market fact versus authenticity fiction. For Bayton (1998: 15) the ‘objectification of performers’ derives from this ‘star system’ (‘creating product loyalty’ and ‘simplifying promotion’) as ‘another record company strategy to secure profits’. Proof of powerless media in technophile times, Barbie Rock escaped notice because of such ‘cultural’, ‘irrational’, ‘pre-existing’ or ‘loyalty’ factors. Since the 1950s race-culture of Rockabilly and Doo-Wop, 1960s counterculture Psychedelia of Surf to Folk, 1970s subculture Punk fused with Reggae, and 1980s underculture Swamp joined to Hip-Hop, reconciliation of such opposites has defined rock’n’roll. By the 1990s, homieculture Riot Grrrl and Girl Power linked Barbie Rock through a series of ‘answer records’ (a ‘bizarre’ tradition for Dawson and Propes, 1992: 132), like ‘Twist Barbie’, ‘Barbie USA’, ‘Bitterness Barbie’, ‘Barbie Girl’, ‘Barbie Be Happy’ and ‘Career Barbie’.[11]

Not without precedent, the bridging phase between the fadeout of Rockabilly in 1959 and British Invasion by 1963 was important for more than Keitley’s sexist reference to a ‘lost spirit of rock‘n’roll’ (2001: 118). This was also a moment of ‘exciting and energetic “girl groups”’ (Gaar, 1992) with chart-topping songs, like The Bobbettes, The Chantels, The Chiffons, The Crystals, The Dixie Cups, The Exciters, The Ronnettes, and The Shirelles. Music primary definers tend to dismiss these ‘in-between years’ of rock‘n’roll, when popular music suffered a supposed ‘near death experience’ (Keitley, 2001: 116; Birch, 1987: 165). Devaluation of female success in ‘monetarily dry’ times hid a reality of controlled artists, withheld royalties and ‘black’ women excluded by secondary definer media (busy re-racialising a newsworthy Watts Riot). The Ed Sullivan Show, for instance showcased both a ‘white’ 1950s Elvis and 1960s Beatles. ‘Radio may have been colour blind but television was not: none of the black girl groups of the early ‘60s appeared on the Ed Sullivan Show no matter how many hits they had, whereas a minor contender like Britain’s Cliff Richard, who had only two U.S. Top 40 hits at the time, appeared three times’ (Gaar, 1992: 51). So much for confident claims that ‘rock and roll has always been on television’ (Grossberg, 1995: 371).

Viewed in this way, while Stras concedes ‘the genre’s seemingly contradictory historic importance as a fostering ground for feminine and racial equality’ (2010: 21), the Girl Groups of enlightened scholarship based on biopolitical race, class, age or sexual identities seem to reiterate an ‘evil’ problematic of mistaking effects for causes (Baudrillard, 1984: 13). Stras’ assent to agist ‘rites of passage’ from ‘girl’ to ‘woman’, for example leaves her idea of ‘adolescence’, now complicit with ‘adult’ moral panic, open to charges of propaganda for the very sexual inequality evils that she criticises. Launched during this Girl Group era and seen as the paradigm ‘of young girls’ aspirations and fantasies’, Barbie™ the doll ‘embodied a fixed ideal of emerging womanhood for the English-speaking world’ (2010: 15). This identity neutral ‘respectability’ was an unattainable norm. Stras argued that the only option open to Girl Groups was to ‘dissemble’ their nature. Pretending to do/think one thing, while actually doing/thinking another (2010: 19). Much later, a plastic era of Riot Grrrl lo fi, Girl Power hi tech or Barbie Rock technical skills emerges to decode that ‘essentialised myth of woman tied to nature’ (Toffoletti, 2007: 79).

Surpassing 1960s Girl Group success and sexist discrimination on a larger scale over a longer period, the 1990s saw another social exclusion of the new female Heavy Metal meets Girl Group emergent style. Increasingly eclipsed by new technology and over-invested ‘poor chic’ cinema, MTV-centric privatised ‘cable’ television shifted music priorities from audio to telegenic visuals. Homan (2007) and Mathieson (2000) note how less profitable radio turned to ‘standardised playlists’, while record companies kept to the research based ‘objective repertoires’ of 1980s Rap or 1970s Punk. Yet the female counter-practice, registered in both chart success and innovation, spanned Olympia’s early self-misspelt ‘Riot Grrrl’[12] to ‘Girl Power’ global hit ‘Barbie Girl’ by Denmark’s Aqua. A ‘homie’ (cf. Hip Hop curfew neologism, ‘homegirl’) glocalised genre cross-section would include the popular success of Aqua, Breeders, Cardigans, Cub, Destiny’s Child, Donnas, Echobelly, Elastica,[13] Killing Heidi, Shonen Knife, Skunk Anansie, Spice Girls, Superjesus and Tiddas. Male bands could only offer passé feeding contexts (Nirvana, Pixies, Suede, Living Color) of retro subgenres, such as Punk-Metal,[14] Grunge or Britpop. But if the experience was certain, like Punk, dissensus not consensus was a norm for many participants:

No, evil ha ha…I have a big problem with females in music at the moment, if you’re talking about your, in my eyes, Barbies – Ella Hooper, Killing Heidi (Hex)

The ‘Barbie’ part I can see, the horror [Heavy Metal] I don’t – there is psychological horror, but I think horror is the wrong word to put on it…terror is more like it – Greil Marcus, journalist (Hex)

‘Specular’ (Irigaray, 1985) Riot Grrrl’s many Heavy Metal cover-versions allocated bands like Heart, the Runaways and Girls School as significant influences, disturbing this alleged misogynist boysclub genre with intuitive ‘cool grrrls’.[15] Going further, Girl Power ‘overmimed’ media secondary definitions, untying the definitive gaze of MTV clips with a reflexive awareness that ‘to define “woman” is necessarily to essentialise her’ (Moi, 1985: 139; Straw, 2001; and coolgrrrls.com, 1998). A presentation of self, like symbolic interactionist stigma logic (Goffman, 1979), rock’n’roll genre formation from 1950s ‘Rockabilly’ (variant of hillbilly), 1960s ‘Psychedelia’ (mental hallucinations), 1970s ‘Punk’ (prison putdown), and 1980s ‘Swamp’ (country urbanism) to 1990s ‘Barbie’ (plastic female) displaced and reversed a once mild pejorative term.[16] Hebdige traces this progression back to inverted labelling of ‘black’, ‘funk’, ‘superbad’ and ‘jazz’ (1980: 62-3). Sensing the very real community demand for a subversive female genre in 1998, Rock’n’roll Highschool recording studio’s Stephanie Bourke suggested, ‘it would be great if someone could bring us all together…the scene is definitely there for it’ (Polymer ‘Stone’: 15). While rarely recognised by media secondary definers, pop culture slang and entrepreneur primary definitions do occasionally transmute into mass genre accreditation. Subcultures seeking publicity or ‘street cred’ are criticised for cultivating moral panic (McRobbie and Thornton, 1995: 572). Even Barbie™ doll copyright holder Mattel Corp, after unsuccessful litigation against Aqua (The Beat, 1997), now capitalise with authorised Barbie Party Mix CDs. Yet as Elastica discovered, ‘intellectual property rights are the motor driving much of the music business’ (Breen, 1999: 70).[17] According to Baudrillard, translation of evil (or sexual demonisation) into mere ‘misfortune’ can lead to, ‘a whole culture of misfortune, of recrimination, repentance, compassion and victimhood’ (2007: 145).

In the industry of romance,

New ways to enhance,

Her beauty,

My little doll,

Beauty comes from the soul

‘Barbie Be Happy’ – Essential Logic, 1998

Figure 6: Dissent, Linoleum [with permission from Universal Music]

Figure 6: Dissent, Linoleum [with permission from Universal Music]

5 Dissent: the compact disc

‘Of crucial importance’ to a sociology of rock, album reviews ‘seek simultaneously to provide a consumer guide, to comment on a culture, and to explore personal tastes’ (Frith, 2001: 174). In terms of a consumer guide’, Straw suggests, ‘the genre as the context within which records were meaningful accompanied the rise of the “serious” record review’. That ‘generic economy’, drawn according to Straw from ‘film criticism’ important to academia (2001: 103), might also be traced to Jazz writing. But because ‘most journalists are male’ reviewers, Bayton argues that ‘a hegemonic masculine view tends to predominate in the music press’ (1998: 3). Her comment on the culture’ notes 1990s music as, ‘a genuine female youth subculture with the explicit aim of moving in all areas of the rock world’ (3). A ‘lucky researcher in the right time and place’ (Negus, 1999: 29), I[18] too discovered ‘an attempt was made to create an organised network amongst all-girl bands, via fanzines’ (Bayton, 1998: 75). This distinguished Barbie Rock from the genre symbiosis of Heavy Metal for which, Straw (2001) argues, ‘audiences do not constitute a musical subculture’. Conversely, Metal’s devotion to rock‘n’roll cover versions (rebutting ‘consistent noninvocation of rock history/mythology’ charges – 102-3) is consolidated by the Grrrls own genuine delight in Metal covers. With such reference groups of Punk-Metal cementing Riot Grrrl to Girl Power, exploring personal tastes’ in the 1990s amounts to Barbie Rock Invasion of ‘the most male dominated of rock forms’ (109).

 

Yeah, I mean I’m not familiar with the Linoleum LP you’re talking about, so I can’t really comment on that – John Peel, journalist (Hex)

Barbie Rock?! Hahahaha…tell me, which exponents of Barbie Rock do you feature? – Caroline Finch, Linoleum (Vee)

As the title suggests, Linoleum’s first album (1997) is sexed dissent to rock’n’roll mythology. Summing-up Barbie Rock, singer Caroline Finch says, ‘Dissent is a different way of looking at things, and it’s a different way of song-writing and bringing up things that other people know but don’t usually bring up from a female perspective’ (Finch, 1999: 35). Linoleum encountered the same obstacle course experienced by more high-profile groups, like Elastica (settling out of court with flattered Punk bands, Wire and The Stranglers). ‘We had problems in Australia certainly because another band called “Linoleum” showed up out of the blue…I think they’d put a single out a while before us, and they started demanding huge amounts of money from us for our name. So I know we had quite a lot of difficulty in Australia getting press and things, because we recently weren’t really allowed to be called Linoleum out there. That was a bit of problem. Yeah I think the industry is in a bit of a mess at the moment’ (34). Riot Grrrls were also banned by popular music media (recalling a blanked out Sex Pistols at No. 1) for manifestos, ejected male slam dancers or heckler abuse texta scrawled on bodies. ‘I wonder, I mean there doesn’t seem to be a lot of coverage of female bands.’ Yes, the rock-press also practise social exclusion of music.

Musically, Linoleum’s Barbie Rock vector surrounded Caroline’s cutesy high-pitch to contralto vocals, delivered as rapping Girl Power overmime segués into room tipping specular Riot Grrrl in 4/4. An under-rated lead guitarist for these unidentified times, musician Paul Jones’ high-action manoeuvring wrung out the Heavy Metal power chord with duck walk riffs, surf licks, and free-form feedback. ‘Paul’s got a good set of new pedals and things, so there’s some interesting sounds on our new stuff. But they’re still coming out of the guitar, even if they don’t sound like it. I think that Paul’s particularly good at playing his guitar so that it doesn’t sound like a guitar sometimes. He plays not exactly in a traditional way, so I think that’s how come we get those results’ (34). Caroline’s rhythm converging on Paul’s created the classic dynamic duo of rock’n’roll guitars, only interrupted by Dave’s snap to deep rumble drum or Grunge fuzz faithful bass of Emma. A white noise, wall of sound energy that, ‘takes power out of the hands of the dangerous people…and puts it back into the people who are being creative.’

American producers Paul Kolderie and Sean Slade (also veterans of Hole and The Pixies) engineered and mixed in multi-layered studio flow a creation upgraded to vinyl quality audio depth. The Erik Nitschean-style Mod red, white and blue sleeve design, backed with fresco secco group miniatures, was initially packaged in real lino. ‘The fact that it’s quite tacky and it’s…I love linoleum because it’s not what it appears to be. Especially when it looks like a very glamorous floor and it’s not. I don’t know if you’ve ever seen our records that are packaged in linoleum? We didn’t make that many of them. Ever since we first started up all our flyers, and our first singles were actually packaged in floor covering, which is quite fantastic.’ (37). Track listings for this floored Barbie Rock genius explored matters of masochist masquerade [‘Marquis’], alienation shock [‘Dissent’], risk chic [‘Dangerous Shoes’], social snobbism [‘On a Tuesday’], remote control [‘Restriction’], substance abuse [‘She’s Sick’], post-modern angst [‘Unresolved’], and ad hominem [‘Smear’]. ‘It’s certainly a view of dysfunctional relationships, there’s a questioning of things that don’t work, and I find all these kind of issues more interesting’ (35).

Yeah I wanna be like her,

Ride the bus in my underwear

’Career Barbie’ – The Kowalskis, 2002

Figure 7: Polymer Hex

Figure 7: Polymer Hex

Concluding remarks

Recent music research on sexed subgenre revolutions has examined both Riot Grrrl (Schilt, 2004) and Girl Power (Martin, 2006; Strong, 2007). But this is the first to recognise their interconnection as phases in the key 1990s genre of Barbie Rock, while avoiding assimilation to established genres (but see Riley, 1994; Lumby, 1998; Chugg, 2005). The unidentified factors (Parts 1-3 above) behind social exclusion and social capital until now pre-empting full recognition as a popular music genre are, in a word (detected earlier by Hesmondhalgh), ‘technicism’. In this gatekeeper scenario, 1) a ‘computerisation’ of cultural intermediaries like rock-press hacks succeeded by primary definer world-wide web hackers, 2) accompanied by secondary definer media ‘moral panic’ implosion, and 3) 3D radio freeze-frame to 2D ‘video dominated’ freeze-out (if not ‘cool’), overshadows the all but not seen and not heard infant terrible genre. Technology has ‘inhibited creativity’ (Hesmondhalgh, 2007: 88). However, whether at odds with fashionable iconoclastic slogans like, ‘down with genres’ (Deleuze and Parnet, 1987: 17), this new technology has also empowered creativity. As the very fact of Barbie Rock demonstrates (Toffoletti, 2007).

Yet there is another tendency, apparently inescapable in even the most enlightened works of men, at times unheard by women (Part 4). The men are easy to find (check out Lester Bangs’ ‘Back Door Man and Women in Bondage’). But when Reynolds and Press portray Riot Grrrl’s Huggy Bear ‘in full awareness of its connotations’ as ‘“asking for it”’ (1995: 331). Or female Bayton argues a Girl Power audience would be, ‘only too delighted to give her [Courtney Love] an ironic f***’ (1998: 79), a sense that such projective hysteria stems from internalised sexism is hard to avoid. In that case, men who ‘determine the marketplace’ (Stras, 2010: 4) are uncontested (but see Part 5) rivalling ‘evil speaking evil’ or evil speaking good (including my mere maleness?) noted by Baudrillard (2010: 39). The theory of evil is a leitmotif for ‘the conspiracy of art’ or demonisation of Barbie Rock. In this ‘obstacle race’ of unsung female artists, indicating a ‘social exclusion’ by the arts, the ‘impotent’ (Baudrillard, 1996: 122) hysteria of male ‘rape’ appears the most fitting explanatory metaphor because, ‘it is no longer decency that is threatened with violation, but sex, or rather sexist idiocy, “which takes the law into its own hands”’ (122).[19] Nevertheless, as even their Satanic Majesties, The Rolling Stones signify with their latest best of Grrr, the lived experience of rock’n’roll – from Riot Grrrl to Girl Power – lives on in the music of Barbie Rock.

Most people would look at it as sarcastic right now; cause like no one really talks about it in the genuine sense anymore, even though I love mine – Donna A, The Donnas (Hex)

 

Acknowlegements: Thanks to Jo Grant from NEIS [New Enterprise Incentive Scheme]; Stephanie Bourke of Rock n roll Highschool; Jacqueline Gallagher at Monash University. Also special thanks to Farrago [University of Melbourne], and Rabelais [Latrobe University] for supporting Polymer.

 

References

Abbott-Chapman, J., Denholm, C. and Wyld, C. 2007 ‘Pre-service professionals’ constructs of adolescent risk-taking and approaches to risk management’, Journal of Sociology, 43(3), 241-61.

ABC 2006 ‘Aboriginal report’, Lateline, television program, 15 May.

Age 2006 ‘Urgent call: send troops now to NT’, 20 May, 1.

Age 2007 ‘National emergency: PM acts’, 22 June, 1.

Banks, J. 1998 ‘Video in the machine: the incorporation of music video into the recording industry’, Popular Music, 16(3), 293-309.

Baudrillard, J. 1984 The Evil Demon of Images. Annandale: Power Institute.

Baudrillard, J. 1996 The Perfect Crime. London: Verso.

Baudrillard, J. 2007 The Intelligence of Evil or the Lucidity Pact. Oxford: Berg.

Baudrillard, J. 2009 The Transparency of Evil. Brooklyn: Verso.

Baudrillard, J. 2010 The Agony of Power. Los Angeles: Semiotext(e).

Bayton, M. 1998 Frock Rock: Women Performing Popular Music. Oxford: Oxford University.

The Beat 1997 ‘Barbie Song Upsets’, 3 September, 33.

Beilharz, P. 1995 ‘Social Theory in Australia: a Roadmap for Tourists’, Thesis Eleven, 43, 120-33.

Birch, M. 1987 ‘“I’ve got class and I’ve got style”: the politics of popular music’, in Breen, M. ed. Missing in Action. Kensington: Verbal Graphics.

Bonner, A. 2006 Social Exclusion and the Way Out. West Sussex: John Wiley & Sons.

Bourdieu, P. 1988 Homo Academicus. Cambridge: Polity Press.

Bradbury, M. and McFarlane, J. Eds. 1976 Modernism. Harmondsworth: Penguin.

Bradshaw, J. 2004 ‘How has the notion of social exclusion developed in the European discourse?’, Economic and Labour Relations Review, 14(2), 168-86.

Breen, M. 1991 A stairway to heaven or a highway to hell: heavy metal rock music in the 1990s, Cultural Studies, 5, 193-4.

Breen, M. 1999 Rock Dogs. Annandale: Pluto Press.

Brett, J. 1991 ‘The Bureaucratisation of Writing’, Meanjin, 50(4), 513-22.

Brophy, P. 1987 ‘Avant-garde rock: history in the making?’, in Breen, M. ed. Missing in Action.

Burchardt, T., Le Grand, J. and Piachaud, D. 2002 ‘Introduction’ in Hill, J., Le Grand, J. and Piachaud, D. Eds. Understanding Social Exclusion. Oxford: Oxford University.

Chugg, R. 1989 ‘Rock’n’roll de/generation’, in Brophy, P. ed. Stuffing, Music: Image. Northcote: Stuff.

Chugg, R. 2005 ‘Reasons for dissent: the sociology of barbie rock’, TASA Conference paper University of Tasmania.

Chugg, R. 2007 ‘The beast box: recombinant television’, Continuum, 21(1), 91-106.

coolgrrrls.com. 1998 ‘The e-magazine dedicated to women and music’.

Cohen, S. 1973 Folk Devils and Moral Panics. St Albans: Paladin.

Daly, M. and Silver, H. 2008 ‘Social exclusion and social capital: a comparison and critique’, Theory and Society, 37, 537-66.

Davies, H. 2001 ‘All rock and roll is homosocial: the representation of women in the British rock press’, Popular Music, 20(3), 301-319.

Dawson, J. and Propes, S. 1992 What was the first rock‘n’roll record? Winchester: Faber and Faber.

de la Fuente, E. 2007 ‘The place of culture in sociology’, Australian Journal of Sociology, 43(2), 115-30.

Deleuze, G. and Parnet, C. 1987 Dialogues. London: Athlone.

Driscoll, C. 1999 ‘Girl culture, revenge and global capitalism: cybergirls, riot grrls, spice girls’, Australian Feminist Studies, 14(29), 173-193.

Fabbri, F. 1980 ‘A theory of musical genres’, http://www.tagg.org/others/ffabbri81a.html.

Finch, C. 1999 ‘Linoleum Wipe the Floor of Contemporary Style’, Polymer Vee, 5, 34-7.

Foucault, M. 1976/1981 The History of Sexuality, Volume One. Harmondsworth: Penguin.

Frankel, B. 2001 ‘The Rise of the Sociologist King’, Arena Magazine, 52, 20-4.

Frith, S. 2001 ‘The popular music industry’, in Frith, S. et al. Eds., The Cambridge Companion to Pop and Rock.

Frith, S., Straw, W. and Street, J. Eds., The Cambridge Companion to Pop and Rock. Cambridge: Cambridge University.

Gaar, G. 1992 ‘Girl Groups’, in She’s a Rebel: the History of Women in Rock‘n’roll. Washington: Seal Press.

Germov, J. and McGee, T.R. Eds. 2005 Histories of Australian Sociology. Carlton: Melbourne University.

Gibson, W. 2006 ‘Material culture and embodied action: sociological notes on the examination of musical instruments in jazz improvisation’, The Sociological Review 54(1), 171-87.

Goffman, E. 1979 Stigma, Notes on the Management of Spoiled Identity. Harmondsworth: Penguin.

Greer, G. 1999 The Whole Woman. London: Doubleday.

Grossberg, L. 1984 ‘“I’d rather feel bad than not feel anything at all”: rock‘n’roll, pleasure and power’, Enclitic, 8, 94-111.

Grossberg, L. 1995 ‘MTV: swinging on the (postmodern) star’, in Munns, J. and Rajan, G. Eds., A Cultural Studies Reader. London: Longman Chesire.

Hall, S. 2005 ‘New labour’s double-shuffle’, The Review of Education, Pedagogy, and Cultural Studies, 27, 319–335.

Hall, S., Critcher, T., Jefferson, T., Clarke, J. and Roberts, B. 1978 Policing the Crisis: Mugging, the State and Law and Order. London: Macmillan.

Harley, R. and Botsman, P. 1982 ‘Between “no payola” and the “cocktail set”: rock’n’roll journalism’, Local Consumption, 2(3), 231-63.

Hebdige, D. 1980 Subculture, the Meaning of Style. London: Methuen.

Herald Sun. 2007 ‘Child abuse alarm, teachers report thousands of cases’, 8 July, 15.

Hesmondhalgh, D. 1999 ‘Indie: the institutional politics and aesthetics of a popular music genre, Cultural Studies, 13(1), 34-61.

Hesmondhalgh, D. 2007 The Cultural Industries. London: Sage.

Homan, S. 2007 ‘Classic hits in a digital era: music radio and the Australian music industry’, Media International Australia, 123, 95-107.

Hubert, J. 2000 ‘Introduction: the complexity of boundedness and exclusion’, in Hubert, J., ed. Madness, Disability and Social Exclusion. London: Routledge.

Irigaray, L. 1985 Speculum and the Other Woman. New York: Cornell University.

Keitley, K. 2001 ‘Reconsidering rock’, in Frith, S. et al. Eds., The Cambridge Companion to Pop and Rock.

Linoleum 1997 Dissent. compact disc, London: Geffen.

Lotringer, S. 2005 ‘Introduction: The Piracy of Art’, in Baudrillard, J. The Conspiracy of Art. New York: Semiotext(e).

Lumby, C. 1998 ‘The Shock of the Image’, Polymer News, 1, 15-16.

Martinelli, A, 1994 ‘Entrepreneurship and Management’, in Smelser, N.J. and Swedberg, R. Eds., The Handbook of Economic Sociology. New York: Princeton University.

Mathieson, C. 2000 The Sell-in. St Leonards: Allen & Unwin.

Mathieson, C. 2006 ‘Death and the majors’, Meanjin, 65(3), 241-45.

Martin, F. 2006 ‘Women on this planet: globalisation and girl rock in Taiwan’, Perfect Beat, 7(4), 5-31.

McRobbie, A. and Thornton, S. 1995 ‘Rethinking “moral panic” for multi-mediated social Worlds’, British Journal of Sociology, 16(4), 559-74.

McRobbie, A. 2007 ‘Top girls?: young women and the post-feminist sexual contract’, Cultural Studies, 21(4-5), 718-37.

Moi, T. 1990 Sexual Textual Politics. London: Methuen.

MX 2008 ‘Unequal in law, society’, 7 April, 9.

Negus, K. 1999 Music Genres and Corporate Cultures. London: Routledge.

Raphael, A. 1995 Never Mind the Bollocks, Women Rewrite Rock. London: Virago.

Reynolds, S. 1988 ‘Against health and efficiency’, in McRobbie, A. ed. Zoot Suits and Second Hand Dresses. Basingstoke: Unwin Hyman.

Reynolds, S. and Press, J. 1995 Sex Revolts: Gender Rebellion and Rock‘n’roll. London: Serpent’s Tale.

Riley, V. 1994 ‘Women’s Lip’, Real Time, 3, 13.

Rimmer, M. 2007 ‘The Grey Album: copyright law and digital sampling’, Media International Australia, 114, 40-53.

Ritzer, G. 2004 The McDonaldization of Society. Newbury Park: Pine Forge.

Rolling Stone 1997 ‘Women in Rock’ special issue, December.

Schilt, K. 2004 ‘Riot-Grrrl is . . . ’: Contestation over meaning in a music scene.’ in Bennett, A. and Peterson, R.A. Eds. Music scenes: Local, translocal and virtual. Nashville: Vanderbilt University.

Shien, G. 1987 ‘FEMALE NOTES: women’s music’, in Breen, M. ed. Missing in Action.

Stras, L. ed. 2010 She’s So Fine: Reflections on Whiteness, Femininity, Adolescence and Class in 1960s Music. Farmham: Ashgate.

Straw, W. 1984/2008 ‘Characterizing Rock Music Culture:  The Case of Heavy Metal.’, in Michael Ryan, ed., Cultural Studies: An Anthology. Boston: Blackwell.

Strong, C. 2007 ‘Gender and memory: remembering and forgetting the women of rock’, TASA/SAANZ Joint Conference paper, University of Auckland.

Tanner, J., Asbridge, M. and Wortley, S. 2008 ‘Our favourite melodies: musical consumption and teenage lifestyles’, British Journal of Sociology, 59(1), 117-44.

Théberge, P. 1991a ‘“Musicians” magazines in the 1980s: the creation of a community and a consumer market’, Cultural Studies, 5, 270-93.

Théberge, P. 1991b ‘The “sound” of music’, New Formations, 8, 99-113.

Toffoletti, K. 2007 Cyborgs and Barbie Dolls. London: I.B.Tauris.

Virilio, P. 2005 The Information Bomb. London: Verso.

Walker, C. 1996 Stranded. Sydney: Macmillan.

Walker, C. 2000 Buried Country: The Story of Aboriginal Country Music. Annandale: Pluto Press.

Zappa, F. and Occhiogrosso, P. 1990 The Real Frank Zappa Book. London: Pan.

 

Notes

[1] A song by Captain Beefheart, ‘Rock’n’roll’s Evil Doll’ (1974) encapsulates male fear of women in music for even the most enlightened artist.

[2] Riot Grrrl is an ‘underground feminist punk rock movement’ (http://en.wikipedia.org/wiki/Riot_grrrl) according to Wikipedia; whereas Girl Power, ‘as a term of empowerment, expressed a cultural phenomenon of the 1990s and early 2000s’ (http://en.wikipedia.org/wiki/girl_power).

[3] Ten per cent of national income redistributed from labour to capital (Frankel, 2001: 24).

[4] Quotations are from Polymer back-issues: News (1998), Stone (1998), Inner City (1998), Thing (1998), Vee (1999) and Hex (2000-1).

[5] ‘It’s always useful to remember…that the history of popular music is really traced through the losers rather than the winners, because there are far more of those’, says John Peel (Hex, 2000-1: 30)

[6] For outstanding cultural studies typical of this period, see Grossberg (1984); Straw (2001); and Brophy (1987).

[7] See also Hesmondhalgh (1999: 36) on ‘sell out’ and ‘burn out’.

[8] See Chugg (2007)

[9] Alternately:

Australian avant garde rock then, starts and finishes with the fact that the people born and/or living in Australia make Australian avante garde rock. But such a subcategory carries no mysterious subcultural traits that can differentiate its content and substance from avant garde rock around the world. It is no wonder that groups from Seattle, Brussells, Cornwall, Vancouver and Canberra can provide remarkably similar work without ever having heard each other’s work, simply by plugging into the same historical sources and references from both the histories of art and rock. As in so many instances, “Australianism” might work as a qualification but not a description – (Brophy 1987, pp. 140-1)

[10] I argue that the concept of ‘social exclusion’ retains relevance in cultural contexts.

[11] By Shonen Knife (Japan, 1992), Gloo Girls (USA, 1994), The Lunachicks (USA, 1995), Aqua (Denmark, 1997), Essential Logic (UK, 1998) and The Kowalskis (USA, 2002), respectively. (Now even crossing-over to Country, with Unknown Hinson’s ‘Barbie Q’, and Jack Ingram’s ‘Barbie Doll’)

[12] For ‘grrl’ or ‘grrrl’ spelling, see Raphael (1995: xxiii).

[13] Destiny’s Child was the most successful US female group; Elastica’s album became the fastest selling debut in the UK, et cetera.

[14] Contrasting Punk, Straw highlights Heavy Metal’s ‘triumph of craft production…“empty” virtuosity and self-indulgence’ (2001: 100).

[15] Barbie Rockers covering Heavy Metal include: Babes from Toyland, Baby Animals, The Breeders, Belly, The Cardigans, The Clouds, Concrete Blonde, The Corrs, Daphne and Celeste, The Donnas, Lita Ford, Fur, L7, Linoleum, Ninety Nine, Nitocris, Rebeccas Empire and Superjesus.

[16] For another commodity research double entendre, see Ritzer’s (2004) ‘McDonaldisation’ (itself labelled ‘McWeberian’).

[17] A well-known example is Girl Group, the Chiffons’ ‘He’s So Fine’ plagiarised by ex-Beatle George Harrison – later disclaimed by Chiffons members citing label legal pressures (see also Rimmer, 2007: 40-53).

[18] A mere male.

[19] In this context, ‘sexist idiocy’ is determined by technicism: ‘has not rape perhaps become the unacknowledged by-product of a technological emergency that is becoming routinised?’ (Virilio, 2005: 70)

 

 

Biographical Note: Rock Chugg is a freelance sociologist from Melbourne, with recent research appearing in publications like Continuum, Refractory, and Meanjin.

Morality, Mortality and Materialism: an Art Historian Watches Mad Men – Catherine Wilkins

$
0
0

Abstract: In 17th century Netherlandish painting, artists employed a complex visual system to assign a symbolic value to everyday objects, in a sort of visual shorthand for lengthier moral concepts and narratives. Such “disguised symbolism” was often used to reflect as well as express concern about the period’s wealth, hedonism, and habits of consumption through the accoutrements and, thus, the terms of the material world. This article will explore the adoption of disguised symbolic iconography in the television series Mad Men.  The article will focus on certain objects – the mirror, the watch, the egg, and the oyster – that have acquired and accumulated meaning over time, based on art historical traditions and the past experiences of their collective viewers.  The article will argue that the way in which these objects were used, both in 17th century Netherlandish painting and in Mad Men, promotes contemplation of morality, mortality, and materialism. Ultimately, they transcend the function of a visual narrative device and allow viewers access to “Truth” about consumption that is relevant to their own lives, as well as to those of the show’s characters.

Figure 1: AMC’s Mad Men, ‘Smoke Gets In Your Eyes’ (2007)

Figure 1: AMC’s Mad Men, ‘Smoke Gets In Your Eyes’ (2007)

AMC’s critically-acclaimed Mad Men (creator: Matthew Weiner) is a television series in which style and visual aesthetics are regularly praised for both enhancing the depth of the fictional narrative and contributing to the historical authenticity of the program. For instance, Jim Hansen has argued convincingly about the importance of appearances and the layers of meaning to be unpacked in regard to the corporeal image of Don Draper (Hansen 2013, 145-160). Meanwhile, Jeremy Butler has written about the “profound nature of things:” the objects that comprise the mise en scene as signifiers of 1960s culture and evidence of Weiner’s attention to historical visual literacy (Butler 2010, 55-71). So far, however, there has not been an attempt to combine these two valuable perspectives – Hansen’s search for deeper meaning through style with Butler’s focus on material paraphernalia – in regard to Mad Men. Yet, for art historians, it is a familiar practice to interpret the appearance of objects as a means of unveiling deeper meaning and creating an experience of revelation for the viewer. Employing the tradition of 17th century Netherlandish painting as a point of comparison, this article will explore the use of disguised symbolic iconography in Mad Men. The frequent insertion of historically symbolic item such as the mirror, the watch, the egg, and the oyster promotes contemplation of morality, mortality, and materialism. Ultimately, the inclusion of these objects allows viewers of Mad Men access to “Truths” about consumption that are relevant to their own lives, as well as to those of the show’s characters.

The basic tenets of semiotics are essential in drawing a comparison between the use of symbolic objects in both Mad Men and Netherlandish painting. A simplified semiotic approach to the subject would argue that images (whether painted or motion picture) are signs, that signs are designed to represent something beyond themselves, and that an inherent part of human experience is to interpret the signs that we encounter in our daily lives. The act of creating any image is an act of arranging compositional elements to convey meaning; and the act of viewing an image is an act of translating and deciphering such elements to create significance. Hence, representations of objects – whether depicted in a painting or in a television program – always contain layers of potential meaning and symbolic value.

In Derrida’s semiotics, this process is constantly being negotiated and both sign and signified are always in flux, with authorship becoming less significant than viewership for determining value and meaning (Derrida 1982, 313-323). According to Bryson and Bal, an image is “by definition repeatable…[o]nce launched into the world, the work of art is subject to all of the vicissitudes of reception; as a work involving the sign, it encounters from the beginning the ineradicable fact of semiotic play…works of art are constituted by different viewers in different ways at different times and places” (Bal and Bryson 1981, 179). In his influential text Ways of Seeing, art historian John Berger argues that images acquire and accumulate meaning over time, based on the past experiences of their collective viewers. The reason that certain historic images may still convey specific symbolism or meaning to us today is because our current experiences are not altogether unlike those of the individuals who first made and viewed the images in question (Berger, J. 1991, 24-33). Therefore, to argue that certain “signs” (symbolic objects) convey a similar meaning in both 17th century painting and on Mad Men in the fictional context of the 1960s, we must investigate potential similarities in context that would affect the production and reception of these images, all the while keeping in mind how the context of this article’s writing influences our interpretation (Derrida 1979, 81).

Madison Avenue in the mid-twentieth century was, in many ways, not so different from the Grote Markt district of Antwerp in the seventeenth century. Over the course of the previous hundred years, Europe had become socially and economically modernized and, proportionately, the Low Countries had financially benefited much more than any other area of the continent during the Renaissance (Snyder 2004, 433). Better methods of transport there led to massive increases in trade, empire, resources, and urbanization. These factors caused a shift in the distribution of wealth, as the flourishing of commerce elevated the social and material status of merchants, bankers, and those involved in the service industry, giving rise to a true middle class that was capable of consuming and patronizing the arts. On a visit to Antwerp in 1520, the renowned German printmaker Albrecht Dürer noted the socioeconomic diversity of the urban environment, commenting upon his interaction with “workmen of all kinds, and many craftsmen and dealers…shopkeepers and merchants…horsemen and foot-soldiers…Lords Magistrates…clergy, scholars, and treasurers” (Durer 1889, 96). Among these various strata of the population, the new influx of material wealth was readily apparent despite vocalizations of concern about the sins of gluttony and greed (Schama 1987, 3-15). Northern European cities were becoming renowned for every conceivable type of consumption, from “business [to] bourse…breweries [to] brothels,” even though members of the population expressed anxiety about the way in which these practices countered traditional religious beliefs (Snyder 2004, 433). Art itself became a commodity, treated by some as an investment, by others as pleasurable decoration for increasingly large homes (Alpers 1983, xxii).

Figure 2: Simon Luttichuys, Vanitas (c. 1655)

Figure 2: Simon Luttichuys, Vanitas (c. 1655)

It was in this context, of increasing material wealth combined with Christian guilt at overabundance, that disguised symbolism became prominent in the paintings of the era. As described by Erwin Panofsky, disguised symbolism was a complex visual system that was, nevertheless, understood by a wide audience. Artists employing this strategy assigned a symbolic value to everyday objects, in a sort of visual shorthand for lengthier moral and historical concepts and narratives (Panofsky 1953, 131-148). Disguised symbolism was used as a vehicle to “create an experience of revelation” (Ward 1994, 12) through the accoutrements and, thus, the terms, of the material world (Lane 1988, 114).

Intended as a means to “celebrate the triumphs of the Dutch culture of commodities [yet…] moralize against consumption” (Berger, H. 2011, 37-38), familiar objects were transformed into “emblems of mortality to remind the viewer how transitory and fragile his pleasures are and how easily beauty and life are broken” (Slive 1962, 488). In 1984, the art historian Ivan Gaskell convincingly argued that the inclusion of disguised symbolism is ultimately about a desire to reveal Truth. In his article, “Vermeer, Judgment, and Truth,” Gaskell compares the iconography of a particular painting – Woman Holding a Balance – to Biblical verses, other period artworks, and 16th century guides to visual symbolism in order to demonstrate that the painting can be read as a contemplation of Truth as an antidote to the imbalance of worldly vanities (Gaskell 1984, 557-561).

The notion of balance that was integral to the emergence of disguised symbolism in 17th century painting was once again relevant in post-World War II America, when conflict about consumption again reared its head. Then, the United States experienced extreme economic expansion, fueling – and, in part, fueled by – a growing advertising industry with its locus of power on Madison Avenue in Manhattan. Between 1945 and 1960, America’s GDP doubled, and grew by an additional 46% in the ensuing decade, with major growth taking place in the service sector and in the manufacturing of consumer durables (Vatter and Walker 1996, 129). Plentiful jobs, higher wages, and better educational opportunities led to the rapid growth of the middle class and an increasing market for everything from processed foods to refrigerators, cars to a home in the suburbs. Television sets, in particular, became much more affordable to many Americans, and ultimately were made a ubiquitous part of post-war cultural life, with nearly 90% of all US households owning one by 1960 (Jordan 1996, 798).

Yet, many Americans were uneasy with such newfound wealth and increasing consumption. According to historian Elaine Tyler May, the fear that spending would lead to decadence was rooted in a long-standing sense of pragmatism and Christian morality that was skeptical of luxury and opulence (May 1988, 148). The advertising industry was highly influential in challenging these doubts, and took advantage of television’s ability to transmit messages straight to the living rooms of a public with more leisure time and greater disposable income than ever before. Corporate spending on advertising doubled in the decade immediately following the end of World War II, then doubling again within the course of just the next five years (Vatter and Walker 1996, 129). Representations of conspicuous consumption abounded in both print ads and television commercials of the era, demonstrating the power of the image by effectively encouraging Americans toward increasingly materialistic values and practices (O’Guinn and Shrum 1997, 278-294). As the market became saturated and consumers became younger and savvier over the course of the 1960s, a creative approach that would continue to attract and influence Americans was necessitated. Wit and humor, narrative and personality were incorporated into advertising, and subtly began to define not only the products being sold, but also the advertising industry itself. More and more Americans began to associate the trade with modernity, youth, and hipness, in addition to its traditional connotations of wealth, luxury, and indulgence (Meyers 1984, 122; Frank 1997, 132-167).

With the actual historical and locational context of the fictional program Mad Men aligning with that of the Netherlands in the 17th century, we may now begin looking in earnest for shared “signs,” recurring images that appear in the television program and in historical Dutch paintings, encouraging deeper analysis as examples of disguised symbolism. Though there are many incidences of potential disguised symbolism to be found throughout Mad Men, this article focuses on four symbols – the mirror, the timepiece, the egg, and the oyster – that were of especial significance in the 17th century, and whose historical symbolism resonates particularly well in the 1960s storylines that Weiner constructs.

The very first episode of Mad Men (“Smoke Gets In Your Eyes”) introduces the mirror as a symbolic element, of reflection and confrontation with a true image of self that is often denied. The opening scene finds Don sitting alone at a bar table, an empty book of matches and three snuffed-out cigarettes in the ashtray before him and a mirror above him, reflecting the bar’s patrons (Figure 1). Trying to come up with a new advertising strategy for Lucky Strike cigarettes, Don asks the barman about his smoking habits. After a moment’s hesitation, he explains that he loves smoking, but then goes on to state that his wife “hates it” and that “Reader’s Digest says it will kill you.” The camera then pans to the bar, at which the dozens of happy patrons, male and female, who were reflected in the mirror above Don’s head are seen puffing away on their cigarettes. Whether the smokers are oblivious to the dangerous nature of their habit, or just unwilling to confront it, the viewer is put in the position of an omniscient observer of Truth, privy to the reality and self-deception taking place in the scene, and the balance between the pleasures of the physical world and their potential relationship to death.

Even a casual observer might detect the symbolic possibilities that a mirror possesses, and their potential applications in Mad Men. A mirror is a vessel of verisimilitude, confronting the user with an image of self that might not correspond to the user’s perception of self. The mirror’s ability to strip away vanities and deception and reveal truth lends power to this object, as well as symbolic value. The fact that the image that appears in the mirror is temporal in nature, subject to change or vanish altogether when the subject himself moves or disappears, adds another layer of meaning to the mirror, suggesting the transient nature of self-awareness, if not life itself. In the disguised symbolism of the 17th century, this object was used to suggest the impermanence of youth, beauty, and earthly delights as well as the foolishness, the deception, in valuing such ephemeral things.

When watching Mad Men, the viewer often occupies a position similar to that of a spectator of such a 17th century Netherlandish genre scene. The century prior had seen the development of a fascinating new artistic convention: the inclusion of reflective surfaces that depicted people or objects outside of the actual area of the representation, oftentimes the artist or symbols of his craft. For instance, Simon Luttichuys’ painting Vanitas (Figure 2, c. 1655) contains several elements historically representative of the passage of time and inevitability of death, including the hourglass, the fading flower, the blank page, and, of course, the skull. A mirror is also included in this assemblage, demonstrating that the symbolic value of this object is aligned with that of the others. Indeed, it reflects the back of the skull, suggesting that, though we often turn to the mirror for superficial reasons, to satiate our vain impulses, the mirror can actually reveal the fleeting nature of youth and beauty and demonstrate, instead, the true end that awaits all things of this world. In addition to the skull, another object appears in the mirror: an artist’s easel, with a canvas upon it. This inclusion invites the viewer to not only identify painting with a reflection of truth, but to identify personally with the artist, the omniscient maker of meaning, by seeing the scene from his point of view (Stoichita 1997, 186-197). The invitation to witness a moment in time from this vantage point – from which we can see both an object and its reflection, perception and truth, and are left to draw our own conclusions about the difference between the two – is also extended to viewers in Mad Men.

Figure 3: AMC’s Mad Men, ‘Maidenform’ (2008)

Figure 3: AMC’s Mad Men, ‘Maidenform’ (2008)

Figure 4: AMC’s Mad Men, ‘Maidenform’ (2008)

Figure 4: AMC’s Mad Men, ‘Maidenform’ (2008)

Mirrors abound in every season of the program, offering views of several characters that both reveal the characters’ focus on superficiality and self-perception, yet allow us to also see a more critical, corrupted, “realistic” view of their true moral character – from an overweight Betty tightening her stomach in front of a mirror in Don’s new apartment, to Megan, sobbing in front of a bathroom mirror after her acting abilities are criticized, to Don, peering into his mouth with a mirror to look at his rotting tooth, a potent symbol of a decaying soul. “Maidenform” (Season 2, Episode 6) is an episode particularly dominated by reflection and repeatedly offering the viewer the opportunity to stand outside of the scene being depicted and to see the dual function of the mirror: as an object involved in vain pursuits as well as in the revelation of the fraud involved in such self-deception.

The episode opens with the three primary female characters – Betty, Joan, and Peggy – reflected in bedroom mirrors as they are donning undergarments. Here, the mirror represents the superficial focus of women striving to satisfy the gaze of self and/or other. Each woman attempts to construct a youthful and beautiful façade, to appear trim, well-formed, and tan (respectively), whether for their own approval or for that of those who might observe them (Figure 3).[1] Similarly, Peter Campbell uses a mirror in the same episode to seemingly overlook his own lusty sins and see instead a successful and desirable spouse. This scenario arises when Pete returns to the apartment he shares with his wife after a fling with a potential Playtex bra model who still lives at home with her mother. Creeping into a darkened house, Pete puts down his briefcase and catches a glimpse of himself in the hall mirror. After looking himself in the eye for a moment, he gave a slight, smug smile and looked away, perhaps relishing his newfound studly self-perception while ignoring the obviously roguish dimensions of his character. In all of these cases, the mirror is engaged in its primary function, serving as a reflective surface that allows those who stand before it to see what they want to see – a flattering version of themselves that is nonetheless constructed of fleeting qualities such as youth, beauty, and sexual desirability.

Later in the episode, the mirror takes on its second function, revealing to the omniscient viewer both the superficial reflection as well as the often-unpleasant Truth about what or who is shown in its surface. For example, midway through the episode, Don meets Bobbi Barrett in her hotel room, where she pours two glasses of champagne in front of a mirror. She looks up at Don, who approaches her from behind, and has a verbal exchange with him that ends in his request for her to “stop talking,” perhaps because her speech disrupts the desirable image that he saw reflected in the glass. They kiss and turn away, while the viewer continues to watch their lusty embrace in the mirror. Though the couple is surely attractive and desirable on the surface, the viewer sees in the mirror two individuals who are violating their marital vows as they head toward the bed that lies in shadow. Indeed, the scene quickly goes sour when Bobbi, in that bed, verbally reveals this truth that Don doesn’t want to face: he has developed a reputation around town for being a connoisseur in the field of extra-marital affairs.

Figure 5: AMC’s Mad Men, ‘Doorways’ (2013)

Figure 5: AMC’s Mad Men, ‘Doorways’ (2013)

In the final scene of the episode, the exchange with Bobbi is revisited, in a sense, as the viewer witnesses Don shaving before a mirror. His daughter Sally comes into the bathroom to observe him, reassuring her father that she “won’t talk” so as not to disturb him in the process – a verbal reminder of his own request to his mistress. He stops to look at her, then back at himself in the mirror. He is visibly shaken, and cannot continue his act of grooming, first looking himself in the eye, then averting his own gaze before sitting down on the toilet, with his back to the mirror above the sink, demonstrating an inability to confront his own infidelity as reflected back to him. However, the viewer is still privy to a mirror image of Don for, as the camera pulls back, we see the image of the “real” person slipping off-screen to the left, as a reflection of him in the mirror on the back of the bathroom door comes into view on the right side of the screen. For a moment, both images inhabit our field of vision, before the scene fades to black (Figure 4). Just as in Luttichuys’ painting, the viewer is put in the position of the omniscient meaning-maker, confronted with dueling dimensions of reality: Truth and perception. Here, the deceptive nature of the glass now becomes visible not only to Don, but to the viewer as well, whose ability to stand outside the scene and observe that which is simultaneously represented and that which is not, highlights the discrepancy between reality and reflection and drives home the need to challenge the acceptance of superficial appearances with honest assessments of cost and benefits.

The appearance of the wristwatch and its significance in several episodes of Mad Men also speaks to self-deception, particularly in regard to the denial of one’s mortality. The wristwatch, like the hourglass in Luttichuys’s painting and many other Netherlandish genre scenes, reminds the viewer that time is passing and will eventually catch up with us all, that the pleasures of indulgence and consumption will pass, and that we – like all material things – will ultimately age and die. This symbolism is driven home in Season 6, Episode 1 (“Doorways”), for example: an episode almost entirely about death. The episode opens dramatically, with the camera pointed toward the ceiling, a woman screaming, and a doctor’s face looming large up above, performing chest compressions on the person whose perspective the viewer occupies, putting us in position to contemplate the potential for an imminent and unexpected death. The scene fades to black, as the voice of Don reads the first sentence from Dante’s Inferno: “Midway in our life’s journey, I went astray from the straight road, and woke to find myself alone in a dark wood.” A scene comes into view, of Don reading and Megan sunbathing on Waikiki Beach. Just a moment later, he picks up his wristwatch to check the time for Megan, only to discover that the watch is not working (Figure 5). He holds it to his ear, scrutinizes the dial, then hands it to Megan, who insists that Don must have gotten the watch wet. After a moment of consideration, she hands it back to him saying, “Who cares what time it is?”

Figure 6: Peter Claesz, Vanitas Still Life (1630)

Figure 6: Peter Claesz, Vanitas Still Life (1630)

Figure 7: AMC’s Mad Men, ‘Ladies Room’ (2007)

Figure 7: AMC’s Mad Men, ‘Ladies Room’ (2007)

Figure 8: Jan van Eyck, Lucca Madonna (1436)

Figure 8: Jan van Eyck, Lucca Madonna (1436)

The meaning behind this opening sequence revolves around an interpretation of the stopped watch as a symbol of death. In paintings, all watches and clocks are, of course, stopped, frozen by the artist at a distinct moment in time, for all time. Yet it is equally obvious that, in the disguised symbolism of 17th century paintings, the appearance of a “stopped” timepiece was not a mere side effect of the painting process, but a deliberately symbolic inclusion. In Peter Claesz’s 1630 Vanitas Still Life (Figure 6), a skull and bone rest atop a table, next to an empty and overturned glass, a common symbol for a life brought to an abrupt end. Beside the glass is an elaborate pocketwatch, open, with a mirror in its lid reflecting nothingness and its hands frozen in time. The watch, viewed in conjunction with the skull and glass, can be read as implying the end of time that precedes the darkness, the truth we wish to avoid, seen in the mirror. As the watch stops ticking, so, too, we are left to gather, does the heart of its owner.

This interpretation is particularly relevant in light of the juxtaposition of images – heart attack victim and Don’s stopped watch – seen in “Doorways.” The fact that Megan disregards the stopped watch may be a sign of her youth and naivety, her sense that there are still many more moments to be experienced, despite the fact that one never knows when he will run out of time, so to speak. Yet the observant viewer likely can recognize that this is a symbol not to be ignored, especially since it has appeared before – in Season 2, Episode 3 (“The Benefactors”), in which Don’s watch is again described as having stopped, only later to be repaired by Betty. Perhaps there, the broken watch was more metaphorical, a representation of Don’s marriage nearing its expiration date (this was, after all, the episode in which he begins his affair with Bobbi Barrett that later led to his divorce). In “Doorways,” however, the close visual link between the stopped watch and stopped heart suggest a more fatalistic reading in keeping with the vanitas imagery of the 17th century, which overtly drew parallels between timepieces and the end of all time.

The second-ever episode of the series, “Ladies Room,” seems to likewise directly reference the disguised symbolism popularized by 17th century Netherlandish painting. The opening sequence of the episode is a beautifully composed still life activated by the presence of the motion picture camera. In the first shot, an egg is cracked and broken, with the yolk poured out into a small bowl. A disembodied hand then reaches for a dish in which a lemon sits, halved. One half is taken away, placed in a cloth napkin, and squeezed over the yolk. In the background, the viewer can discern other bowls, piled high with croutons, cheese, and lettuce – the makings of a Caesar salad (Figure 7). Yet the focus is clearly on the egg, a highly symbolic food item that makes its way into dozens of 17th century paintings, as well as several future episodes of Mad Men.

Eggs have, in many cultures and for many millennia, been considered a symbol of fertility, reproduction, and regeneration. The egg’s physical characteristics and actual function – a vessel for a new life hidden within a rounded form that has been compared to both a testicle and a breast – are clearly responsible for these attributes, which occur in the art and mythology of societies as diverse as that of ancient Egypt, China, and Finland (Newall 1967, 3-32). The Christian artists of the Northern Renaissance imbued the egg with additional levels of disguised symbolism that were carried on in the works of 17th century Netherlandish painters. Within the Christian tradition, the egg became representative of the Resurrection of Christ (Jesus rising from the dead as the chick springs forth from a seemingly inert object) as well as the Incarnation (with the egg representing the womb of Mary, impregnated with the Christ child).

Jan van Eyck’s Lucca Madonna (Figure 8, 1436) illustrates this latter interpretation of the egg, demonstrating the new, distinctly religious dimension to the pre-Christian association of the egg with regeneration and fertility. In this painting – as well as other images of the Madonna by van Eyck, such as the Ince Hall Madonna – the Virgin Mary is enthroned in a domestic interior, nursing the newborn Christ child on her lap. To the right are objects such as a basin filled with water and a clear glass vessel that are traditionally read as symbols of baptism and of Mary’s intact virginity, respectively.[2] On the left side of the picture plane is a large leaded glass window through which light passes: a potentially symbolic inclusion, as Christ is often called the light of the world, the means by which divine truth is illuminated. The light beams pass through the intact glass (again, symbolic of the undisturbed virginity of Mary) to illuminate not only the Madonna and child, but two eggs that sit directly on the windowsill. The symbolism of the eggs parallel that of the other objects represented in the scene, in that they speak to the idea of new life being created and contained within something unbroken and intact. The fact that there are two eggs – mirroring the two human figures present in the painting – suggests that van Eyck is referring not only to the Incarnation of Christ, but to the Immaculate Conception of Mary herself. In both cases, fertility and reproduction were stripped of the “sinful” stain of sexuality; instead, they were connected to ideas of purity, wholeness, and the sacrality of familial life, which the egg, as a versatile symbol, took on.

Figure 9: AMC’s Mad Men, ‘Three Sundays’ (2008)

Figure 9: AMC’s Mad Men, ‘Three Sundays’ (2008)

In Season 2, Episode 4, “Three Sundays,” the egg appears in a way that corresponds with the disguised symbolism just described. This episode focuses on Peggy Olsen and her relationship with religion in the aftermath of giving birth to Peter Campbell’s son. Despite her struggles with spirituality and the institution of the Catholic Church, Peggy agrees to do some pro bono advertising work for her home parish and develops a friendly relationship with the young pastor in the meantime. Through dialogue with her family, the pastor, Father John, finds out about the child Peggy bore out of wedlock. At the end of the episode, the third Sunday – Easter Sunday – has arrived, and an egg hunt is taking place on the church grounds. Father John approaches Peggy and the camera zooms in, then holds the image of the priest pressing a blue egg into the palm of her white gloved hand, “for the little one” (Figure 9). The Easter interpretation of the egg – as a sign of the risen Christ – is here eclipsed by the other symbolic value of the egg, as a sign of fruitful femininity and the proliferation of family life through new birth. Though Peggy’s virginity is, of course, no longer intact, the priest’s gesture seems to suggest a desire to restore wholeness to Peggy herself, and to her family life. Peggy, who ultimately gives her child up for adoption, seems uncomfortable with the gesture, perhaps not only shocked by Father John’s knowledge of her illegitimate child, but unwilling to adhere to the model of femininity and family life implied by the intact egg and visualized in van Eyck’s Madonna paintings.

Understanding the egg as a symbol of fertility and family with sacred undertones helps inform an understanding of an even more frequently seen image, repeated in several episodes of Mad Men: that of the broken egg. If the intact egg is a symbol of wholeness, virtue, and the life-giving role of sexuality in a familial context, the broken egg represents the shattering of innocence, integrity, and the familial bond. Broken eggs can frequently be found in Netherlandish paintings representing the cost of intemperance, particularly as it relates to sexuality and family life. For example, Jan Steen’s raucous Interior of an Inn (Figure 10, c. 1665) depicts a barmaid whose skirt is being lifted by an inebriated patron. Though she places one hand on his arm as though to restrain him, with her other hand, she touches her breast in a sensual manner. Two other men look on, one openly laughing and the other provocatively stuffing the bowl of his long-stemmed pipe with his pinky finger while staring at the female subject. On the floor before them all are opened mussel shells (the symbolism of mollusks to be discussed at length later in this article), an empty frying pan with an exceedingly long handle, and many broken eggshells.

While there might be a logical interpretation of the eggs’ appearance – surely, in a scene of such boisterous behavior, its reasonable to assume that a fragile food item might be disturbed – their staged appearance on the floor, in conjunction with the other objects that surround them and the previously established symbolic value of intact eggs, suggests disguised symbolism at work. The indulgence in the vices of smoking and drinking are overtly pictured in the work. Additionally, sexual indulgence is implied through the gestures of the painting’s characters and the phallic allusions of the pipe stem and pan handle juxtaposed with the receptive voids of the pipe’s bowl and pan’s bottom. The broken eggs add a moralizing component to this otherwise festive scene, alluding to the destruction of purity in the domestic ideal expressed by the intact egg, and signifying especially fallen womanhood (Thomson and Fahy 1990, 11).

Figure 10: Jan Steen, Interior of an Inn (c. 1665)

Figure 10: Jan Steen, Interior of an Inn (c. 1665)

Figure 11: AMC’s Mad Men, ‘Red in the Face’ (2007)

Figure 11: AMC’s Mad Men, ‘Red in the Face’ (2007)

Breaking eggs serve a similar iconographic function in Mad Men, and are especially abundant in Season 3, appearing in Episode 4 (“The Arrangements”) and Episode 5 (“The Fog”) alike. In both cases, the egg’s historical value as a symbol of incarnation, birth, and familial life is relevant, but the fact that the egg is shown breaking seems to indicate the fracturing and dissolution of family life. In “The Arrangements,” the premise by which the breaking egg is introduced is, again, the making of a Caesar salad. Gathered around the table this time are Don, Peter Campbell, and Horace Cook, Jr. (“Ho-Ho”), one of Pete’s old school friends. Ho-Ho is presented as a spoiled rich kid, about to waste his inheritance by investing in jai alai. The dinner conversation revolves around father-son tensions – particularly, the distaste Horace Cook, Sr. has for his son’s leisurely lifestyle and investment choices – a theme that will repeat throughout the episode. As the men talk around the table, a new camera angle is presented, which places the salad preparation table – and, significantly, the breaking of an egg – in the foreground of the shot. Perhaps not coincidentally, this same episode features Gene, Betty’s father, dying, as well as an insomniac Don getting up in the middle of the night to stare at a picture of his father and step-mother: families shattered, just as the egg was.

In “The Fog,” Betty Draper gives birth to Gene, the child conceived even as her marriage with Don was falling apart. Again, the egg is given a place of importance, visually and, this time, in regard to dialogue as well. Don, taking care of Sally and Bobby while Betty is hospitalized, is shown cooking a late night snack of eggs and corned beef hash. He holds an egg up to the light, “checking for a chick” as Sally astutely guesses, before the camera closes in on the frying pan to show Don breaking the egg over the sizzling meat. Sally says that her teacher taught her that eggs from a store could never become a chicken, “even though they came from a chicken.” Sally then awkwardly transitions into a question about whether or not the baby will live in Grandpa Gene’s old room. The new life of the Gene, Jr. is juxtaposed with the death of Gene, Sr., visualized by the breaking of the egg. Perhaps as well, Don’s futile search for new life in the egg (the lack of a chick and Sally’s statement that he will never find a chick in eggs like that) represents the inability for his family to be reborn and recover from his adultery.

The image of the broken egg again reflects generational conflict and families divided due to intemperance in Season 4, Episode 2, “Christmas Comes But Once a Year.” Taking place in the aftermath of the Drapers’ divorce, this episode finds Sally’s close friend Glen Bishop breaking into the family home (now occupied by Betty and her new husband, Henry Francis) and vandalizing it with the help of his friend. Glen, the product of a broken home himself, has been motivated to engage in this behavior by the confidences of Sally, who confesses extreme unhappiness in her new familial environment. The two young boys focus their destructive energies on the kitchen, and are shown dumping out the refrigerator contents on an empty counter. While there are a few food elements that are clearly visible (such as ketchup and cereal), there is again a focus on eggs, with the boys removing a carton and breaking individual eggs all around the kitchen. It is a scene of simultaneous abundance and squander, infused with anger. These representatives of the younger generation lay waste to the fruits of capitalism and consumerism as they express frustration with the seeming selfishness of their parents. The boys’ unhappiness with the divided Draper household – and their own, as well – is reinforced by the symbolic connection between broken eggs and broken families that echoes through the seasons of Mad Men.

A final symbolic element to appear in Mad Men is the oyster, verbally and visually alluded to in several episodes of the series. Throughout, the oyster serves as a symbol of luxury and indulgence, often with intemperance as an undertone. For example, in Season 2, Episode 2 (“Flight 1”), Peter Campbell finds out that his recently deceased father was insolvent, having spent his money on “oysters and travel club memberships” – seemingly frivolous and fleeting pleasures. This verbal association of the oyster with self-indulgence is enhanced by imagery in the series as well, such as that seen in Season 1, Episode 7: “Red in the Face.” Don and Roger, out for a bender of a lunch, are enjoying copious amounts of oysters and martinis. At the 34:33 mark, as Roger is ordering another pair of drinks and more oysters, the camera focuses in on the table before them. On the red and white striped tablecloth sit two platters piled high with the remains of a decadent lunch. Each hosts a dozen empty half-shells and several squeezed lemon wedges nestled on a bed of lettuce alongside a partially-full bowl of cocktail sauce. On a small plate between the two main dishes are two lemon wedges, one untouched and the other appearing squeezed. Directly adjacent to this plate is an ashtray, full of cigarette butts and ashes. Two small carafes of water appear half-full, and Roger’s martini glass, nearly empty (Figure 11). The visual connotes both extravagance and waste, abundance spent, with symbolism and a moralizing tone that are both borrowed from Netherlandish painting.

Ubiquitous in 17th century genre scenes, oysters were – then and now – a food that connoted luxury and indulgence, yet also perhaps danger. The European fascination with the oyster can be traced back at least as far as Greek mythology, which initiated the association of the erotic with the shellfish by situating the birth of the goddess Aphrodite on an oyster shell. “The oyster remained a symbol of Aphrodite throughout antiquity, the Middle Ages, the Renaissance, and continued into the Baroque era,” bringing with it through the ages an association with the characteristics of “fertility, pleasure, and sex” that were attributed to the goddess herself (Cheney 1987, 135-6). The rise of the Dutch empire in the 1600s added another layer of symbolic meaning to the oyster. As the Dutch conquered the Portuguese in the waters of the Indian Ocean around the mid-century mark, pearl fisheries and oyster-rich waters came under the domain of the Netherlands, and the oyster became an accessible – if rare – edible indulgence that simultaneously represented the increasing power and possessions of the nation.

When seen through a moralizing lens, however, the positive identifications of the oyster with abundance, wealth, and sensual pleasure can be correlated to the deadly sins of greed, pride, gluttony, and lust. Indeed, many Dutch genre painters employed the image of the oyster in their works as a strategy for documenting the increasing luxury and materialism of their time while suggesting the fleeting nature of such earthly passions. In his 1618 tome Symbolorum ac Emblematum Ethico-Politicorum, Joachim Camerarius explained that paintings that illustrated the drinking of wine and eating of oysters spoke not only to pleasure, but to the dangers of over-indulging in physical desires (Camerius 1618, 120). The scholar claimed that the inclusion of oysters in an image were a means by which a nouveau-riche society that was, at the same time, historically abstemious could grapple with such conflicting aspects of its character (Camerius 1618, 121).

The duality of the oyster’s symbolism has much to do with the potential peril involved in its actual consumption. Raw shellfish carries a risk for bacterial contamination that can cause serious illness in those who consume a tainted sample. Illnesses such as hepatitis, typhoid, or even death by septicemia were not uncommon in centuries past, when refrigeration was lacking and proper handling techniques were not always employed. The literal link between indulgence and the possibility of an untimely death made the oyster a symbol ripe for exploitation by Netherlandish vanitas painters.

Figure 12: Jan Steen, Easy Come, Easy Go (1661)

Figure 12: Jan Steen, Easy Come, Easy Go (1661)

As such, it became “the focus of many genre paintings…frequently the principal food depicted, serving as a vehicle for moral comment” in scenes both superficially merry and serious (Cheney 1987, 135). For example, in Jan Steen’s Easy Come, Easy Go (Figure 12, 1661), the oyster meal is at the center of a seemingly festive gathering loaded with disguised symbolism. In the dining room of a lavishly appointed home, an elderly servant opens oysters for her laughing master, while a beautiful young woman offers him a full glass. The contrast between these two female figures is enough to indicate a negative interpretation of the oyster; associated with old age, not youth and beauty, with death rather than the full life that the brimming cup signifies. Other aspects of the painting add layers of meaning to the oyster’s symbolism. Behind the master of the house is a statue of the goddess Fortuna mounting a die and surrounded by strange sculptural elements. For example, beneath her are two cornucopias: the one on the right (the side of the young woman) overflowing with fruit and coins, while the one on the left (the side of the elderly woman and oyster) full of wilting brambles. Similarly, on the left side of Fortuna is a weeping putto, while on the right, a cheerful one reigns, scepter in hand.

The duality of the imagery here is reinforced by the inscription on the mantelpiece beneath Fortuna that reads “Easily Won, Easily Spent.” A simple explanation for this composition might relate the message and symbolism to the gambling that is taking place in the background of the scene: the viewer sees two men standing around a gaming table through a doorway painted in the upper left corner of the canvas, rolling dice in a game of backgammon. However, the action taking place in the foreground of the scene may be even more pertinent to understanding the fullness of the moralizing message Steen wished to express. Here, an oversized lemon is place on the seat of a chair in closest proximity to the viewer, its peel twisting off in a way that was meant to imitate the inner springs of a clock, symbolizing the passage of time. The lemon is placed next to a juicy oyster, though several empty shells also occupy the seat of this chair, as well as the floor. To the left of this chair, also in the foreground of the scene, is a representation of a boy adding water to wine, suggesting not only religious ritual, but the tempering that is desperately needed in this scene. Behind him on the ground are two empty glasses, perhaps representing the emptiness – even the death – that can be avoided through moderation. Overall, the oyster here serves as a symbol of indulgence and wealth, but is simultaneously associated with chance and even death. The contrast suggests the transient nature of earthly delights and urges moderation and restraint.

Steen’s intention, and that of the other painters of the 17th century symbolic genre scene, was to visualize the decadence of their culture that many members of the up and coming middle-income class of patrons idealized or perhaps even experienced. Yet, these artists sought to simultaneously problematize indulgence by reflecting on the true nature of mankind’s mortal existence and the ultimate cost of consumption by using symbols and signs that a contemporary audience could understand. Thus, Dutch artists could access the often complex, conflicting feelings – of both desire and guilt – that their audience possessed through images that were equally layered with duality of meaning. A similar sort of disguised symbolism works in Mad Men not only because the cultural climate of the 1960s had much in common with that of the Netherlands in the 17th century, but because our contemporary 21st century context does as well.

Since Mad Men debuted seven years ago, its Western audiences have enjoyed the highest standard of living the world has ever known — but with the knowledge that this abundance of riches often comes at the expense of other people, animals, and the environment. Like the Dutch as well as their American predecessors in the 1960s, the 21st century audience of Mad Men is experiencing the attraction of material pleasures (made all the more alluring by the ubiquitous advertising of our age) while simultaneously witnessing the social and environmental cost of consumption. The continuing conflict between consumption and conscience suggests that the use of disguised symbolism in Mad Men can transcend the function of a visual narrative device, and may also point viewers toward a course of action that is as relevant today as it was in the 17th century: to seek Truth and strive for balance in the complex relationship between mortality, morality, and materialism.

 

Works Cited

Alpers, Svetlana. 1983. The Art of Describing. Chicago: University of Chicago Press.

Bal, Mieke and Norman Bryson. 1991. “Semiotics and Art History.” The Art Bulletin 73/2: 174-208.

Berger, Harry. 2011. Caterpillage: Reflections on Seventeenth-Century Dutch Still Life Painting. New York: Fordham University Press.

Berger, John. 1991. Ways of Seeing. London: Penguin Books.

Butler, Jeremy. 2010.”’Smoke Gets in Your Eyes’: Historicizing Visual Style in Mad Men.” In Mad Men: Dream Come True TV, edited by Gary Edgerton, 55-71. London: I.B. Tauris.

Camerarius, Joachim. 1618. Symbolorum ac Emblematum Ethico-Politicorum. Middleburg.

Cheney, Liana De Girolami. 1987. “The Oyster in Dutch Genre Paintings: Moral or Erotic Symbolism.” Artibus Et Historiae 8/15: 135-158.

Derrida, Jacques. 1982. Dissemination. Chicago: University of Chicago Press.

Derrida, Jacques. 1979. “Living On: Border Lines.” In Deconstruction and Criticism, edited by H. Bloom, et al., 75-176. New York: Continuum Publishing Group.

Dürer, Albrecht. 1889. Literary Remains of Albrecht Dürer. Ed. W.M. Conway. Cambridge: Cambridge University Press.

Frank, Thomas. 1997. The Conquest of Cool: Business Culture, Counterculture, and the Rise of Hip Consumerism. Chicago: University of Chicago Press.

Gaskell, Ivan. 1984. “Vermeer, Judgment, and Truth.” Burlington Magazine 126/978: 557-561.

Hansen, Jim. 2013. “Mod Men.” In Mad Men, Mad World: Sex, Politics, & Style in the 1960s, edited by Lauren Goodlad, Lilya Kaganovsky, and Robert Rushing, 145-160. Durham, N.C.: Duke University Press.

Jordan, Winthrop. 1996. The Americans. Boston: McDougal Littell.

Lane, Barbara. 1988. “Sacred vs. Profane in Early Netherlandish Painting.” Simiolus: Netherlands Quarterly for the History of Art 18/3: 107-115.

May, Elaine Tyler. 1988. Homeward Bound: American Families in the Cold War Era. New York: Basic Books.

Meyers, William. 1984. The Image Makers: Power and Persuasion on Madison Avenue. Los Angeles: Times Books.

Newall, Venetia. 1967. “Easter Eggs.” The Journal of American Folklore 80/315: 3-32.

O’Guinn, Thomas and L. J. Shrum. 1997. “The Role of Television in the Construction of Consumer Reality.” Journal of Consumer Research 23/4: 278-294.

Panofsky, Erwin. 1953. Early Netherlandish Painting (Cambridge: Cambridge University Press.

Schama, Simon. 1987. The Embarrassment of Riches: An Interpretation of Dutch Culture in the Golden Age. New York: Alfred Knopf.

Slive, Seymour. 1962. “Realism and Symbolism in Seventeenth-Century Dutch Painting.” Daedalus 91/3: 469-500.

Snyder, James. 2004. Northern Renaissance Art. Upper Saddle River: Prentice Hall.

Stoichita, Victor. 1997. The Self-Aware Image: An Insight into Early Modern Meta-Painting. Cambridge: Cambridge University Press.

Thompson, James and Everett Fahy. 1990. “Jean-Baptiste Greuze.” The Metropolitan Museum of Art Bulletin 47/3: 5-54.

Vatter, Harold and John Walker, eds. 1996. History of the United States Economy Since World War II. Armonk, NY: M.E. Sharpe.

Ward, John. 1994. “Disguised Symbolism as Enactive Symbolism.” Artibus et Historiae 15/29: 9-53.

 

Notes:

[1] This idea is reinforced by Don’s pitch to the Playtex company later in the episode, in which he says, “its about how they [women] want to be seen by us, their husbands, their boyfriends, their friends’ husbands.”

[2] While the basin’s likeness to a baptismal font makes for a relatively straightforward comparison, the importance of the glass carafe is understood best in light of period sources, such as this Nativity song quoted by van Eyck in another composition: “As the sunbeam through the glass passeth but not breaketh, so the Virgin, as she was, Virgin still remaineth.” See Panofsky 1953, 144.

 

Bio: Dr. Catherine Wilkins is a faculty member in the Honors College at the University of South Florida. She received a M.A. in Art History (2003) and an interdisciplinary Ph.D. in Cultural History (2008) from Tulane University. The subject of her doctoral dissertation was post-World War II German landscape painting and its use as a vehicle for social critique. Dr. Wilkins’ research on this subject was recently published as Landscape Imagery, Politics, and Identity in a Divided Germany, 1968-1989 (London: Ashgate, 2013). Her current research interests cover a range of topics in modern and contemporary art and popular culture. Additional publications include “Performing Art History’s Problems with New Media: ‘Capitalist Realism,’ the Northern Renaissance, and Gerhard Richter.” NMediaC, Vol. 6, No. 1 (Summer 2009) and “Constructing Individual Identity in a Cybernetic Age.” Gerard Lange Cartograph/Jared Ragland Apropos (Rocky Mount, N.C.: Barton College Press, 2010).

Playing At Work – Samuel Tobin

$
0
0

Abstract: People play games at work, especially digital games, rather than asking “why” this paper starts with “how”? To do so the game Minecraft and its players are used as a focus to address how people manage to play while at work and in workplaces. This data is drawn from public online forums where hundreds of Minecraft players offer tips for circumventing technical, bureaucratic, social and ethical play constraints and share there feelings, experiences and successes. In these specific and detailed accounts of media practices constrained and engendered by the demands and expectation of workplaces we see the shifting nature of public and private, of work and leisure.

Minecraft-Banner

Playing at Work         

This paper focuses on the way people play the game Minecraft (Mojang, 2001) at work and the ways in which they talk about that play and the practices that support it.  The Minecraft players I study write about this play and the tactics needed to engage in it at work as a combination of subterfuge, escape, challenge, invention and guilt-ridden compulsion. I show how this complicated relation to their play is informed by the ways in which play, games, work and the work place are continuously redefined through these players’ practices and discourse. I focus on adult players of the game Minecraft and the ways they manage to play while they are at work. This data is drawn from the forum reddit.com, where hundreds of Minecraft players offer tips for circumventing technical, bureaucratic, and social play constraints. These online discussions detail a range of technical and play practices constrained and engendered by the demands and expectation of workplaces. In these practices and discourse we see the continuously shifting nature of public, private, work, leisure, mobility and most of all play.

From the Minecraft subreddit on reddit:

thread title: who plays Minecraft at work?

I’ll be honest, this game has pretty much destroyed my productivity recently. I work in IT so I’m on the computer all day. I also have my own office so people cruising by and catching me building really isn’t an issue. Since I bought thisgame 2 weeks ago I’ve wasted more time at work than I even care to admit. Everyday I tell myself I will focus and do actual work, and everyday boredomsets in and I am drawn to Minecraft like a moth to a flame. I am a sad pathetic   individual. Who else is with me? – Apt Get

The short answer is “lots of people.” But what these people mean when they say “Yes, I play Minecraft at work” and refer to themselves as “sad pathetic individuals” is complicated. To address these complicated and complicit issues, I focus on the central problem for these players: “How do you bring your game to work?”  In the sections that follow I rework the phrase “bring your game to work,” stressing different words to expose what is at stake in these spaces and practices of work and of play. First, however, we need to ask what people might or could mean by Minecraft. In exploring how people play Minecraft at work (or any game), we are asking “how” not just in the sense of “How do you manage” but also “In what manner” do you play at work.  The manner or way of playing changes the nature of the game, redefines it, pushes certain aspects of the game forward while eliding others.  As we will see below, players redefine Minecraft, sometimes radically, as they need to in order to play it.

Here at the outset are some general observations and caveats.  At the time of this research (2011-2012) few posters in the subreddits (as the forum threads of reddit are called) mentioned mobile or “Pocket” versions of the game when discussing how to play it at work. This may be due to a kind of self-selection of Minecraft fans in the threads.  People who like the game enough to read and write about it on an online forum may not be interested in playing it on platforms other than the PC or laptop.  In any case, the issue of mobility for most reddit users is not as much about buying a Minecraft app for a smartphone, as getting Minecraft onto their work computers. What we see when we look at the responses people gave to the question “How do you play Minecraft at work?” is a move to redefine what Minecraft play can be while referencing a core experience and object: PC-based Minecraft play.

playing at work

Foregrounding the “at” in the phrase “playing at work,” focuses our attention on “work” as “workspace,” a space constituted by labor, and also by architecture, furniture, expectations, routines, and movements.  We need to attend to the implications of bringing play materials and practices into the workspace, and to the movements such play demands.  The workplace context and the practices it demands make mobile a game which otherwise might not be. This complicates definitions of mobile games, while reinforcing the importance of space and situation to the understanding of game play.

To play Minecraft at work, players need to find ways to bring the game with them to the office. The barriers to this are technical, securitized, cultural and practical. In order to access saved games through workplace firewalls, players trade tactical tips on online forums on how to load Minecraft files onto thumb drives, email zips to themselves, and to otherwise convince their work networks that no barriers have been breached and that nothing is amiss.  Commenters discuss issues of visual surveillance and subterfuge, with extensive discussion of monitor tilting, lines of sight, glare, minimizing routines, hotkeys, and ways to arrange a play mis en place that looks like work (a point we will return to).  These commentators are not always employees contriving to avoid being caught by their boss: the thread at the top of this piece was originated by a boss, Apt Get, who wants to hide his play from his peers as well as from his underlings, and ultimately from himself. These techniques of circumventing lines of sight and firewalls allow people to play at work and at the same time shape and define what that play can be. This play is both proscribed and defined by the context. While details and the differences are important, what these players in all sorts of work contexts share is an array of needs, worries and techniques developed in order to play at work.

It is easy to see how work could be a hostile environment for Minecraft play. Yet in many cases, for some players, work is a less fraught play space than other alternatives. As Apt Get’s comments later in his thread on playing Minecraft at work shows:

Glad to know I’m not alone. I am also guilty of sketching things on graph paper during meetings when I am without a computer. I am married and have 2 small kids, so work is about the only time I get to play.  – Apt Get

This comment reminds us not to assume a neat split between workplace and labor on one side, and domesticity and leisure on the other. The nature of the relation between work and play is a key issue for any study of play or games. In Games of Empire (2009), Greig de Peuter and Nick Dyer-Witheford discuss the historical transformation of this relationship through their critical account of games, capitalism and immaterial labor. In Free labor: Producing culture for the digital economy (2000), Tiziana Terranova makes similar points. Julian Kücklich’s account of computer game modification, or “modding” as a strange mixture of labor and play or “playbour” (2005), also helps us historize the shifting relationship between play and work, and the new hybrid modes that emerge from these categories. In “Alienated Playbour: Relations of Production in EVE Online” (2015),  Nicholas Taylor et al. show how what we might assume is “just” play can in fact be work.

These questions of the work-play relation predate contemporary developments in game studies. We see the relation and separation enforced to different degrees in classics such as Roger Caillois’s Play and Games and Johan Huizinga’s Homo Ludens: A Study of the Play-Element in Culture. Hannah Arendt writes in the Human Condition: “(A)ll serious activities, irrespective of their fruits, are called labor, and every activity which is not necessary either for the life of the individual or for the life process of society is subsumed under playfulness.” (1998) Similarly, in the “Sociology of Sociability,” Georg Simmel expresses an interest in play because of its very apartness from other ‘material’ spheres of life, including work (1949). Separation for these authors is a matter of motivation, economics and necessity.

For Apt Get and many responders at reddit, work is defined spatially. Apt Get asks: “Do you play Minecraft at work?” rather than asking, “Do you play Minecraft instead of work?” Work, for Apt Get and many other players, work is defined more as a place than an activity. This definition of work as a location means that itretains its status as a place for labor even when play is brought in to it. To understand what is at stake when people bring their Minecraft play to work, we need to pay attention not just to what they mean by “work” and “at work” but also to what they mean by “home” and what they do “at home.” For Apt Get, his job is where he can play, even if it is difficult to do so. He can’t or won’t play at home. His posts suggest it is more difficult for him to play at home than it is a work. No doubt many of us recognize ourselves and our workplaces in these posts. What we find in the threads is a complicated and contradictory range of attitudes, experiences and ideas about the appropriateness, pleasures, worries and requirements of playing Minecraft (and other games) at home as well as at work.

Is playing at work always a modified, compromised form of Minecraft play? Not necessarily. This author, who has the luxury of work of an office with a door, a personal computer, and students who rarely take advantage of office hours, is able to play Minecraft in an as unfettered manner as one could hope for. Indeed like Apt Get, time at work was the only time I really could play Minecraft or, for that matter, wanted to. Game play is never “free-play,” as it is always defined and constrained as well as afforded in by the exigencies of everyday life (office, door, computer, students, teaching preparation, publication pressures). Play is always in relation to the everyday, to the rhythms of leisure and labor and socialization and movement and the un-freeness of free time.  Playing at work then is not (or not just) a more constrained or diminished form of play, even if it is often viewed this way.  Playing at work is a compromise, but a compromise that can lead to new and interesting permutations of play.

Minecraft play is typically described as open-ended, free, and creative, in short as the kind of play celebrated by ludic utopians of every stripe.  But what we see in the Minecraft “subbreddit” is a discussion of an even more expansive and “free” play, one perversely bounded by the space of work, as we can see in following two examples, which while specific and personal, are not outliers, and which give glimpses of play tactics and techniques shared in the Minecraft forums, when responding to the question: “How do you play at work?”

Figure 1. A player shares an image of how he "plays at work."  He has used his companies’ Maple computer algebra system to model a possible Minecraft construction.

Figure 1. A player shares an image of how he “plays at work.”  He has used his companies’ Maple computer algebra system to model a possible Minecraft construction.

Figure 2. A player shows how he "plays at work" by stacking shipping boxes in a recognizably Minecraft manner.

Figure 2. A player shows how he “plays at work” by stacking shipping boxes in a recognizably Minecraft manner.

These are very different ways to play Minecraft.  They are different from each other and different from our expectations of how people play Minecraft. These (seemingly) radically different approaches to Minecraft play result in part from differences between these two posters’ workplaces.  However what these two players share are places of work filled with tools and objects of labor. Both players use the stuff of their jobs to build things as Minecraft play practice.  Each is playing Minecraft, but in a kind of play that arises out of and reflects the specific contextual affordances and constraints of their work. Each plays in a way that is both in contestation and conjunction with work and its boundaries.  These two images of Minecraft play would not exist without the work and work places that shaped them. These are just two examples of how the space and tools of one’s work shapes the kinds of play needed to fit those contexts. For every job site and set of tools or materials, we might expect to find different play practices. These examples point to the need to account for a thicker, messier kind of play for playing not just Minecraft, but for all kinds of games played at work.  And while Minecraft may be especially suited to these ludic perruques, it is not unique in being a game people like to play when they are otherwise expected to be working. For each game, as for each work site, we can expect new play practices, cultures and experiences.

With these new practices we get new discourse. How do the Minecraft forum commentators talk about their relationship to this place into which, against which, and with which they forge new play practices?  Many commentators, Apt Get included, use negative and loaded language borrowed from substance abuse and addiction to describe their relationship to their Minecraft play. In my book, Portable Play in Everyday Life, I found that Nintendo DS players use these same metaphors to describe games that they play intensely (2013). We see similar language going back all the way to David Sudnow’s Pilgrim in the Microworld. The angst conveyed by this fraught language seems to go against the perspectives we see in work by researchers like Leonard Reinecke (2009) or Paul Mastrangelo et al. (2006), who argue that play at work is useful or salutary in the sense that it helps one to work better or longer. This perspective may seem managerial or exploitive, but it is also useful for those of us who are invested in a critical approach to games, and to further examining the connections between play, work, playbour and leisure. The reddit commentators are much more likely to talk about their play as transgressive, criminal or pathetic, than restorative. Their discourse is not one of break-rooms and recharging, but of addiction, subterfuge, and tricks. I suggest that this kind of player language should be read as not (only) about compulsion or addiction, but also as code for a particular kind of pleasure and awareness of the larger cultural context for understanding and describing that pleasure. It is a compliment to call a game addictive. It is not only a self-flagellating or distancing remark.

play at work

This discourse brings with it the habitus of the addict: of secret drunks and self-deception.  In order to keep the activity going, there needs to be subterfuge, evasion, cover and camouflage. This is a different kind of playing, a “playing at.” This is playing at in the sense of playing as make believe, as “acting as though.”  This approach to playing at work owes much to Johan Huizinga’s sense of play as always secretive, even in plain sight, and as having a “pretend” quality (Huizinga 1955). It also carries a bit of the sense of calling out something as deceptive, as bullshit, as in the phrase: “What are you playing at?”

The kind of pretense most essential for these commentators is pretending to work while “really” playing.  The hidden or furtive aspect of playing games at work is neither new nor endemic to Minecraft.  Older Macintosh users may remember the “quick the boss is coming” feature from games such as Othello, a command which would instantly bring up a mock spreadsheet to hide your game.  The personal computer’s WIMP interface (windows, icons, menus, pointer), with its layers upon layers of windows, allows a kind of slight of hand and easy hiding of games or other NSFW (Not Safe For Work) activities.  Digital games can be harder to sneak into work outside of white-collar office settings. But as desktop and other types of computers increasingly are used in stores as point of sale systems, in entry ways, and at front desks, we can suspect that many are being used to play games, although it is impossible to know how many, how often — I know that I played a lot of web based games while a clerk at a wine store.  This kind of video game playing at work has clear connections to la perruque (“the wig”) as described by Michel de Certeau, except instead of “a worker’s own work disguised as work for his employer” it is a worker’s play disguised as work. This has more in common with how de Certeau mobilizes la perruque to describe a whole range of practical détournement (s) of time and space (2002).  The time and the spaces which are constituted by work are not our own but spaces of the other. As de Certeau writes:

(A) tactic is a calculated action determined by the absence of a proper locus. No delimitation of an exteriority, then, provides it with the condition necessary for autonomy. The space of a tactic is the space of the other. Thus it must play on and with a terrain imposed on it and organized by the law of a foreign power (2002).

We should take literally de Certeau’s suggestion that the tactic “must play on and with a terrain imposed on it.”  In the case of Apt Get and his interlocutors, this terrain is one of cubicles and frosted glass, but also of box trucks, servers, firewalls, nosey neighbors, and if we merge terrain with time, deadlines, lunch breaks and all-nighters.

If we look back to the algebra-derived model above, or think of design doodles in meetings, or other more expansive ideas of what constitutes Minecraft play, we see that these are perhaps unsurprisingly also potentially cases of perruquesque tactics.  The shipping box stacking, while not at all subtle, takes advantage of the fact that the thing being used in play is also the thing used at work, here not just boxes, but also the act (and skill) of stacking them.  This points not just to a flexibility inherit in the tactics needed to play at work, but also to a more fundamental relationship and tension we see in cases where objects of labor are used for play or pleasure.  This is especially common in digital play (think of the keyboard and mouse of pc gaming), but as the shipping box example demonstrates, not unique to digital play. The tools employed in the case of Minecraft play at work, whether the PC in orthodox forms of Minecraft play, or the diverse workplace materials (graph paper, algebraic software) of the more outré tactics, are always ready to shift back and forth between ludic and mundane. Whatever is reworked towards play always shifts back.  This too, is consistent with de Certeau’s understanding of the tactics of everyday life; whatever tactics of subterfuge might win us, we must be willing to readily discard (2002).  Minecraft play at work is a kind of playing at the level of mimesis and pretense as well as duration; it is, in the best sense of the word, improvised.

To close, let’s return to the two images presented above of work-place play (or work/place/play). If we (mis)read these as being about Minecraft play, and not forms of Minecraft play itself, we leave behind these practices and these players. If we leave exclude these players and their play from our definition of what Minecraft play really is, we must then face the realization that there is no center to hold on to in defining Minecraft play: When is it real, really? In adventure or survival mode? When one is playing alone, or only in groups? Networked or not? To better understand all forms of digital play we must take the limit or fringe cases seriously.  Stacking real boxes at work at first may seem like a strange way to play Minecraft, but it is also somehow the most Minecrafty practice one can think of.  This is due to the creativity of the player, but also to the centrality of space and context for determining what play looks like and what play can be.  We move then from the ideal to the possible, from the discrete to the situated, from the simulated workspace of the mine to the real and contested work place of the player.

This is a move that we need to make when we study games in general, a move towards the world of the player rather than just the world of the game. This is important not only for understanding work-themed games played at work, or mobile games played on the go, but also for understanding more seemingly stable arrangements between player and place, from the historic arcade, to the tavern, to the couch and TV coupling of the home.  These spaces are in many ways as mysterious and as contested as any mine, dungeon, or alien galaxy. When we listen to players talk about how they play rather than just what they play we can begin to attend more to the nuances of these mundane spaces to understand the situated, contextual and contingent nature of play and to see play as always complicated and complicit.  We may well then arrive at an understanding of play as more like the rest of our lives: complicated, compromised, and vital.

 

References

apt_get. “How Many of You are Playing Minecraft at Work?” http://Minecraft.reddit.com/r/Minecraft/comments/dtbiz/how_many (accessed November 2011)

Arendt, Hannah, and Margaret Canovan. 1998. The human condition. Chicago: University of Chicago Press.

Caillois, Roger, and Meyer Barash. 2001. Man, play, and games. Urbana: University of Illinois Press.

De Certeau, 2002. Michel The Practice of Everyday Life. Berkeley: University of California Press.

Dyer-Witheford, Nick, and Greig De Peuter. 2009. Games of empire: global capitalism and video games. Minneapolis: University of Minnesota Press.

Huizinga, Johan. 1955. Homo Ludens: a Study of the Play-Element in Culture. Boston: Beacon Press.

Mastrangelo, Everton, and Jeffery A. Jolton. 2006. “Personal Use of Work Computers: Distraction versus Destruction,” CyberPsychology & Behavior. 9, no. 6, 730-741.

Reinecke Leonard. 2009. “Games at Work: The Recreational Use of Computer Games

During Working Hours” CyberPsychology & Behavior 12, no. 4 461.

Simmel, G. and E. C. Hughes. “The Sociology of Sociability.” The American Journal of Sociology 55, no. 3 (1949): 254-261.

Sudnow, David. 1983. Pilgrim in the microworld. New York, N.Y.: Warner Books.

Taylor, N., Bergstrom, K., Jenson, J. & de Castell, S. 2015. “Alienated Playbour: Relations of Production in EVE Online,” Games and Culture 365-388

Tobin, Samuel. 2013. Portable Play in Everyday Life, Basingstoke and New York, Palgrave Macmillan.

Terranova, Tiziana. 2000. “Free Labor: Producing Culture for the Digital Economy.” Social Text, 63 (Volume 18, Number 2), Summer 2000, 33-58

 

Bio

Samuel Tobin is an Assistant Professor of Communications Media and Game Design at Fitchburg State College in Massachusetts who studies play, media and everyday life. He is the author of Portable Play in Everyday Life: The Nintendo DS (Palgrave Macmillan, 2013.)

 

 

Viewing all 76 articles
Browse latest View live