A sketch of a theory and modelling notation for elucidating the structure of representations

Peter Cheng


A structural theory of visual representations and an accompanying modelling notation are outlined. The theory identifies three types of fundamental representational components, specified as cognitive schemas, that span internal mental and external physical aspects of representations. The structure of diverse and complex example representations are analyzed. Twenty-three requirements that a general theory of representations must address are presented. The new theory satisfies the large majority of them. The theory and notation may provide a basis for future methods to predict the efficacy of representations.

7 Replies to “DDPC1”

  1. This is quite interesting to see. We (Clive Richards and Yuri Engelhardt) have proposed a structural theory of visual representations and an accompanying notation – which we refer to as the ‘DNA of visualization’ – that has basically the same goals and the same main components as the one proposed here. The majority of the requirements that are listed here for such a theory are the same that we based our approach on. It is nice to see that some of our earlier work is referred to.  At the Diagrams 2006 conference we proposed a basic structural theory and a tree diagram notation for visualizations: “We propose a limited set of possible ‘building blocks’ … Based on these ingredients, and the rules for their combination, the syntactic structure of any visual representation can be drawn as a hierarchically nested tree. We claim that the presented visual syntax applies to all types of visual representations.” (full paper at: http://yuriweb.com/VisDNA/objects-and-spaces-engelhardt-2006.pdf ).  We have meanwhile further developed this structural theory of visual representations and the accompanying notation into a more thorough ‘DNA of visualization’. Various materials illustrating this approach, including DNA tree diagrams of numerous example visualizations, are at http://VisDNA.com . We greatly welcome any comments, ideas or thoughts on our current Diagrams 2020 poster page: http://www.diagrams-conference.org/2020/index.php/program/posters/p15/ .  

    1. Hi Yuri and Clive, thanks for your comments.
      I expect some readers will appreciate the pointers to your own papers. There are some key differences in our approaches that I think are useful to note.
      First, my approach theorises that the most appropriate entities for the analysis of representations should be things that fundamentally have a representational function. Hence, the proposed schemas explicitly distinguish and combine information about the target domain concepts and information about the graphical objects.
      Second, understanding and using representations is fundamentally a cognitive activity, so my approach adopts cognitive schemas as elementary representation components. This means they are easier to link what we know about human information processing, which lends some weight to the validity of the approach in terms of psychological plausibility.
      Third, this concerns scope. The approach is intended to cover any abstract symbolic representation, not just visualizations. Any general theory should be encompass linear sentential notations (e.g., written natural language, mathematical formula, indexing systems and so forth), but also in principle be extendable to non-visual media (e.g., haptic and aural).
      Fourth, the approach takes as the starting point of analysis the conceptual structure of the to-be-represented topic relative to a specific user (or class of user) of a representation, because a given instance of a representation (a display) will be interpreted differently by users with alternative levels of domain knowledge or different amounts of experience of the class of representation. What constitutes basic representing objects in a display is not fixed by the display, in itself, but is dependent on the act of reading. Many of the important features of the monster thermodynamics graph (Fig. 3) will not be apprehended by a novice but will be salient to an expert, so the structure of models for the different users will necessarily diverge. Of course, one might restrict analysis to common representations or typical users, but then many interesting representational phenomena that pivot on such differences would be missed.
      Fifth, the “DNA” approach focusses on graphical composition – how visual displays are constituted from visual encodings of types of information. In contrast my theory and model focuses interpretations, which might be viewed as alterative readings of particular constructions, so demands a fuller acknowledgement of the rich conceptual structure of the topics that many representations are attempting to capture.
      It would be interesting to discuss which of the 23 criteria I propose are satisfied by various approaches and also their relative theoretical importance.

    1. Hi Chipp,
      Thanks for the comment. One important difference between Wickham’s grammatical approach and my structured approach is that I’m taking a cognitive perspective rather than a computational constructive one. This means that our starting points have an importantly different starting points. Wickham considers _data_ to be plotted (similar to Yuri and Clive’s _types of information_), whereas my approach starts with the _rich conceptual structure_ of a topic. In my view, many interesting representational phenomena cannot even be properly described, let alone explained, without reference to the full conceptual structure of the target domain. Some of these phenomena are described in refs [7,8] in the paper, but I would also point you to:
      doi: 10.1111/j.1756-8765.2009.01065.x

  2. Great question, Chipp! As Hadley Wickham describes at the beginning of the paper that you are linking to, Wickham’s approach is based on Leland Wilkinson’s ‘Grammar of Graphics’, which in turn is based on Jacques Bertin’s ‘Semiology of Graphics’.

    It seems worthwhile indeed to try to combine, on one hand, such ‘grammar of graphics’-related bodies of work from the data visualization literature (e.g. Bertin, Wilkinson, Card, Mackinlay, Ware, Munzner, Vega-Lite) with, on the other hand, a structural-theory approach like the one that Peter is proposing here – as you express it in your comment, “to use a diagram (graph) to present another diagram”. An early paper (Diagrams 2006) attempting this is at: http://bit.ly/metadiagrams and a more thorough development of our attempts to do this is described in our poster at this conference (‘The DNA framework of visualization’: http://bit.ly/DNAposter ), which can be further explored at the accompanying website: http://VisDNA.com

    Naturally, we are curious to hear what you – and others – think of our attempts to combine such approaches.

  3. Thank you for the interesting talk, Peter! I browsed through the paper as well, and I have the impression that your framework is mostly intended to analyse already existing representations.

    Do you think it could help during the design phase of new representations/visualisations and if so, in what way do you expect it to be useful?

    1. Hi Sven, thanks for the comment. Yes, the theory and modelling notation was created to analyse existing representations. But once you have built a model it reveals the underlying structure of the given representation, so you can make judgements about its strengths and deficiencies, which in turn suggests how one might improve it. Also, using the notation, I find that some representations that are seemly quiet differently graphically actually have similar representational structures, which implies that one could readily transpose the conceptual content between representations. I am planning to explore how the approach can be used for design.

Leave a Reply

Your email address will not be published. Required fields are marked *