Audio-visual Rhetoric and its Methods of Visualization: Introducing visual notation systems for film analysis

Abstract In this paper, we use information design of rhetorical
theory in order to investigate the structure of audiovisual
media. From a design research perspective, we are
especially interested in the power of visualization. In this
respect, we compare different methods of film analysis
referring to their representation mode and evaluate them
referring to their ability to visualize the cinematic structure.
Within the realm of film analysis, we ask :
• How can the use of information design enhance the
traditional analysis of film?
• What is the influence of the selected semiotic code
(text, image, or animation) on the interpretation of
the medium and its structure?
To answer these questions, we introduce examples of visual
notation systems that were applied to analyze film
sequences within a project. We discuss their method and
value for the purpose of analysis and reflect on the role
of the different semiotic codes that are applied. Referring
to the traditional method of analysis in the form of
written film protocols, we hypothesize that this method
is attended by a high loss of structural information when
transferring the audio-visual, dynamic code of film into
written text. While film protocols are more appropriate
for content, respectively plot analyses, we state that
static and dynamic visual notations are suitable forms
for making structural and rhetoric phenomena of film
explicit. Visual notations might cause the least loss of
information in the analytical process, and might of high
value for academic film analysis.
Background
Audio-visual rhetoric is a knowledge domain for designers
in theory and practice that was introduced by Gui
Bonsiepe in the 1960ies (Bonsiepe 1961). Its theory and
communication system is valid for all communicative
actions aiming for persuasion. Richard Buchanan described
the whole body of design practice as rhetorical
argumentation (Buchanan 1989) and opened the way
for a rhetorical design theory on a broad basis. In the
following, visual rhetoric was established within the
design education mostly in the US, teaching the analysis
of information design on the basis of rhetorical patterns
(Ehses 1984, 1986, 1988), (Kostelnick 1989, 1998,
2003), (Poggenpohl 1998). These concepts were transferred
to dynamic, audio-visual media by Bonsiepe in the
late 1990ies, and he introduced the term “Audio-visual
Rhetoric” (Bonsiepe 2008). Today, this theory combines
the ancient communication theory and its huge body of
knowledge with New Rhetoric (Joost 2008, 2008a) and
applies this knowledge to the design domain and its visual
approaches.
The Notation System for Film Analysis
Exploring audio-visual patterns of film is an analytical
endeavor that is still in search for a useful method. In
film theory and practice, there are different models to
describe and analyze filmic structures. For film scholars,
the most established method is using a written film
protocol to elaborate on formal aspects as well as on narrative
structures. This method was questioned by many
scholars because of the lack of potential to reflect on the
specific dynamic and audio-visual quality of film (Lehner
1987). Audio-visual rhetoric provides an approach to
overcome this problem by introducing a notation system
for film. In academic context, there are some examples
of visual film protocols (Hahne 1992), (Ramsbott / Sauter
1988). Nevertheless, none of the visual systems has
been established for film analysis in a broader context
and has not yet reflected the progressing nature of film.
A general issue is that there is no interdisciplinary collaboration
between study of film, film production, rhetoric,
and design research to come up with an applicable
system. Bringing together all these competences, one
could design a comprehensive system that could be used
in various contexts – academic as well as applied. In this
context, we discuss the impact of the semiotic code on
the film analysis (Table 1).
Comparison between different notation
systems for analysis
When applying rhetorical knowledge to the analysis of
film, we take the following steps. We raise the hypothesis
that a visual analysis of film conveys a visual knowledge about the rhetorical structure of media. This form
of knowledge can hardly be expressed verbally. Ernst
Gombrich claims what the diagram presents in front of
our eyes can hardly be expressed by words – as a succession
of statements (Gombrich 1984). An example for this
hypothesis is the visualization of the spatial information
on a map. This complex information that is accessible
parallel on one sight can hardly be expressed verbally in
a cognitively efficient process. Based on this hypothesis,
we introduce different notation systems to visualize recurring
patterns in film: a static visual film protocol (Fig.
1), an animated film protocol (Fig. 2), and a written text
protocol. audiovisual
signs into a written text means to change the semiotic
code in a radical way – from image and sound to
text. This process involves a loss of information, particularly
of the audio-visual and dynamic quality of the
sign system. The idea of using a visual protocol as tool
for film analysis is based on the hypothesis that a visual
diagram can be processed cognitively much more efficient
than language (Bonsiepe 2008). This is especially
true for the visual aspects of film and not for re-telling
its storyline. It is not new to say that the tools and methods
that are used for an investigation clearly influence
the research results. This is also true for the method we
suggest in this paper. To set up a visual diagram of film
focuses much more on visual and structural aspects than
on the storyline. The aim is to reduce a loss of information
that occurs when the audio -visual texture of film is
transferred to written film protocol. Therefore, the audiovisual
signs are translated into a visual structure. This
method has additional advantages: the graphic displays
information on one sight so that the recipient can process
the data in parallel. Written text can communicate
information only in a successive way – one word after
the other. In the notation protocol, information about the
whole clip in each of the channels can be visualized at
the same time, allowing a parallel interpretation of data
and of relationships among the various audio -visual elements.
Here, the pattern structure can be easily identified
on the basis of a graphical representation. For example,
repetitions or climax patterns can be singled out quickly
on the basis of their visual form. With this approach,
large amounts of data from audiovisual
media can be efficiently
processed for analytical purposes. In the next
step, we compared the different systems according to
their semiotic code, their mode of information and the
kind of perception they require. Discussion
One finding is that a verbal analysis, on the one hand,
is much more focused on content and storyline of a film
and translates the audio-visual medium into a verbal
narration. It uses the symbol as semiotic code and requires
a successive perception. Its mode of information
is temporal. The visual analysis, on the other hand, is
more focused on the visual structure of film and its recurring
patterns. It uses icon, index, and symbol as semiotic
code and requires a synchronic perception. Therefore, it
displays spatial information. An animated protocol adds
dynamic information about the development of film and
includes the analyzed film itself into the animation. It
also refers to icon, index and symbol in its semiotic code
and requires synchronic as well as successive perception.
Its mode of information is rich and includes spatial,
temporal and acoustic data. Through visual approaches
such as static visual protocols as well as animated ones,
a different kind of knowledge is gained compared to a
mere verbal analysis: a specific kind of visual knowledge.
This study argues in favor of a semiotic autonomy
of visual signs.
Conclusion
The visual film protocol can serve developers of audio
-visual media in various ways. First of all, it is a helpful
tool to analyze and interpret film and understand its
composition. Moreover, the static visual film protocols
make different films structurally comparable. Using the
notation system one gets a visual protocol comparable to
music notations that can be used for reproduction purposes.
This leads to the third point: the notation system
as a software could be used as tool for film design and
planning in addition to the technique of storyboarding,
which is still a standard tool for the production process.
With this visual aid, film makers can compose their texture
beyond sketches of the scene and visual description.
Based on these insights, we suggested the notation system
as a new tool for film analysis for designers as well
as for film scholars.
References
Bonsiepe, G. (1961). Persuasive Communication: Towards
a Visual Rhetoric. In Crosby, T. (Eds.) Uppercase
Nr. 5, London, UK, 19–34.
Bonsiepe, G. (2008). Audiovisualistische Rhetorik in
zeitbasierten Medien: Über die kognitive Relevanz diagrammatischer
Visualisierungen. In: Joost, G., Scheuermann,
A. Design als Rhetorik. Birkhäuser Verlag Basel.
217-232.
Buchanan, R. (1985): Declaration by Design: Rhetoric,
Argument, and Demonstration. In Design Issues, Volume
II, Number 1. 4-23; Reprint in: Margolin, V. (1989)
(Ed.): Design Discourse. History, Theory, Criticism,
Chicago. 91-109.
Gombrich, E. (1984): Representation and Misrepresentation
(Reply to Murray Kreiger) In Critical Inquiry, Volume
11, 2, Chicago. 195-201. Hahne, D. (1992). Komposition und Film: Projekt nach
Motiven aus Camus „Der Abtrünnige“ für Chor, Orchester
und Spielfilm. (Volkwang-Texte III. Bd. 5. Hrsg. v.
Josef Fellsches).
Lehner, C. (1987). Einige zentrale Probleme der
neueren Filmsemiotik. In Bauer, L., Ledig, E., Schaudig,
M. (Eds.) Diskurs film: Strategien der Filmanalyse. Bd.
1 , München, Germany, pp. 59–72.
Joost, G. (2008): Die rhetorische Pattern-Language des
Films. In Joost, G., Scheuermann, A. (Eds.) Design als
Rhetorik, Birkhäuser Verlag Basel, 233 – 249.
Joost, G. (2008a) Film-Sprache. Die audio-visuelle
Rhetorik des Films. Bielefeld. See also:
http://www.geschejoost.org/AVRhetorik/
Ramsbott, W., Sauter, J. (1988). Visualisierung von
Filmstrukturen mit rechnergestützen Mitteln. In: Helmut
Korte, Werner Faulstich (Hrsg.): Filmanalyse interdisziplinär.
(Zeitschrift für Literaturwissenschaft und
Linguistik. Hrsg. v. Helmut Kreuzer. Beiheft 15). Göttingen.
pp. 156–165.