Spectacles, sieves and filters

Diagram from page 132 of Gray, C, & Malins, J. (2007) Visualizing Research: A Guide to the Research Process in Art and Design, Taylor & Francis Group, Abingdon, Oxon. Available from: ProQuest Ebook Central. [14 November 2023].

Interpreting the map: methods of evaluation and analysis

Visualizing Research: A Guide to the Research Process in Art and Design

“Analysis is not about adhering to any one correct approach or set of right techniques; it is imaginative, artful, flexible and reflexive. It should also be methodical, scholarly, and intellectually rigorous. (Coffey and Atkinson, 1996, p. 10)”

Validity and reliability: towards research quality

“validity and reliability. These concepts (and others, as we shall see later) are concerned with establishing research quality”

“In scientific methodologies, objectivity, validity, reliability and replicatability are the cornerstones of research quality”

“Spectacles and sieves: criteria”

More related to looking at the outcomes themselves:

“• to ‘evaluate’ is to ascertain the value of something and to judge or assess its worth;
• to ‘analyse’ is to examine something in detail in order to discover its meaning.”

“However, nothing can be evaluated or analysed without criteria”

“For example, what makes ‘good’ design could be articulated in
relation to three key criteria: effectiveness, efficiency, economy. These, in turn, could be ‘unpacked’ to provide more focused criteria, for example effectiveness in relation to context and aesthetics; efficiency in terms of function and use; economy in terms of cost and use of materials.”

“It is essential that the criteria you develop relate to the aim and
objectives of the research.”

“Criteria are like spectacles and sieves”

“they are the means by which we focus, capture and distil value and meaning. Different spectacle lenses allow us to see in various ways – to see some things whilst not being distracted by others”

“Different meshes in sieves allow us to capture some things while discarding others, for example in panning for gold. Conversely, paper coffee filters capture the unpalatable grounds leaving us with the essential distilled liquid.”

“These different lenses, meshes, filters are metaphors for the sets of criteria by which we evaluate, analyse and make sense of research outcomes”

I used content analysis to look at the images of cultural identity that the students provided in order to “search for pattern and meaning” (Collier, 2004)

Van Leeuwen, T., Jewitt, C. & Collier, M. (Eds). ‘Approaches to Analysis in Visual Anthropology’. The Handbook of Visual Analysis. Available at: https://doi.org/10.4135/9780857020062.n3 (Accessed 2 January 2025).

Steps in Thematic analysis

– Familiarisation (note initial impressions)
– Coding: Assign descriptive codes (significant or recurring, relating to potential themes
– Theme generation: Group similar codes together to identify broader themes
– Theme refinement: Review and refine themes, consider relationships between themes
– Interpretation: Analyse the identified themes in the context of the research question, considering the social, cultural and historic background of the images

Gray, C, & Malins, J (2007). Visualizing Research: A Guide to the Research Process in Art and Design. Taylor & Francis Group, Abingdon, Oxon. Available at: ProQuest Ebook Central. (Accessed: 14 November 2023).

Analysing data

Now that I have gathered data relating to my intervention I will look into methods of data analysis.

I read Chapter six: Analysing Data, from Kara, H. (2025) Creative Research Methods in the Social Sciences: A Practical Guide to gain an overview on the types of approaches to data analysis.

Up to this point my understanding of data analysis is that it falls into two categories (quantitative and qualitative) and I knew very little beyond these general terms.

The text helped open up my understanding to the kinds of approaches available, how they might suit different kinds of data and provided some examples that gave context to the methods.

My understanding is that I’ll be using the project outcomes as data: the typefaces that students have designed in response to the brief – to explore type design in relation to an aspect of their own identity. Presumably I will be using qualitative methods to do this. I also plan to collect data at the end of the intervention, in the form of interviews, questionnaires, and/or conversations between the students where they have the opportunity to reflect on what they have learned, explored, how their understanding of the canon, decolonisation, diversity in relation to typeface design has changed as a result of the intervention. With the latter, there is potential for quantitative analysis too, in terms of how many students mention particular themes, subjects, etc.

The article gives an example:

“Suppose that you hold
a focus group with eight first-generation immigrants from different countries of origin. You begin by having each person share some basic demographic data by way of introduction: where they have lived, how old they are, their occupation(s) before and after immigration, who and where their family members are. Then you facilitate a discussion of their experiences of emigration and immigration around themes drawn from the academic literature, including wealth and poverty, coercion and freedom, belonging, emotion, status, togetherness and separation. The resulting data would be amenable to quantitative and qualitative analysis”

This is very similar to what I might do with the discussions / conversations at the end of the intervention. With the themes drawn from the academic literature being: decolonisation, the western canon, diversity, etc. The article suggests that “The resulting data would be amenable to quantitative and qualitative analysis”, something I hadn’t fully considered.

The article talks about how conversations can also be analysed in other ways: “consider any silences, pauses or omissions in order to try to uncover what might have been left unsaid and why (Frost and Elichaoff 2010: 56, in Kara, 2015: 99). I’m not sure if this will apply to my data but really interesting to read about these alternative methods of analysis.

The article also refers to “more creative approaches to data analysis” but also the importance of “understanding where rules must be applied” (Kara, H 2015: 100). Firstly I’d like to learn more about possible creative approaches but this is perhaps also referring to how data analysis needs to be rigorous and guided by rules.

In terms of ethics the piece talks about the responsibility of the data analyst and that “It is essential that you do not invent or distort your data, or misuse statistical techniques” (Poon and Ainuddin 2011: 307, in Kara, 2015: 100).

I intend to record conversations and discussions with students and there is some useful information here about ‘data preparation and coding’, specifically about “large variety of decisions to be made about transcription” (Kara 2015: 100). Again, there is reference to “should you record non-speech sounds that people make”, something I had not previously considered.

So there is potential that I will use both qualitative and quantitative data analysis methods.

“Quantitative and qualitative data need to be analysed separately, using different techniques, and in research where both quantitative and qualitative data have been gathered, the datasets will be analysed separately before the analyses are integrated to produce the research findings.” (Kara 2015: 101)

The article later lists several kinds of qualitative data analysis, a few that I found most relevant to my research are

  • content analysis – a semi-quantitative technique for counting the number of
    instances of each category or code (Robson 2011: 349)
    • thematic analysis – identifying themes from coded data (Robson 2011: 475)
  • conversation analysis – detailed analysis of the verbal and non-verbal content
    of everyday interactions (Bryman 2012: 527)
  • phenomenological analysis – analysing participants’ stories from, and
    descriptions of, their ‘life-worlds’, or individual experiences and perceptions,
    with a focus on meaning (Papathomas and Lavallee 2010: 357; Mayoh, Bond
    and Todres 2012: 28)

The last method in particular as potentially being connected to my project about identity and typeface design, how the projects are, in a way a kind of story about individual experiences, cultural backgrounds, etc.

A key part of the text was around Discourse Analysis (DA), “based on the concept that the way we talk about something affects the way we think about that phenomenon.” (Kara 2015: 105). Particularly as the author talks about how DA can be applied to other kinds of data, such as images.

Other potentially useful bits of information from this text were:

– idea of recording participants and how this might affect what they say, if they know they are being recorded. Some researchers looked at recording conversations in more natural settings, for longer periods (Gordon 2013: 314)

– Use of diagrams and maps to “help you visualise your data and the ideas and relationships that develop as you work through the analytic process (Kara 2015: 107). The example given here of researchers Charles Buckley and Michael Waring, “they found that creating
diagrams helped them to generate, explore, record and communicate insights
about their data” (Kara, 2015: 107) and “using diagrams in data analysis can help to uncover some otherwise hidden parts of the research process ((Buckley and Waring 2013: 150 in Kara 2015: 107).

– Also key to my data analysis is an example given on p.109 of researchers in Australia who used Mixed-method analysis and “developed an analytical procedure using three different methods to analyse children’s artworks” (Kara, 2015: 109). They used quantitative techniques (Content analysis, where they categorised visual elements in the drawings and counted number and frequency, and also used two qualitative methods: Interpretive analysis (looking at mood or atmosphere in the drawings), and developmental analysis (looking at correlating development with age). “The researchers conclude that this combination of analytic methods can ‘provide deep insights into young children’s understandings’
(Sorin, Brooks and Haring 2012: 29, in Kara, 2015: 109).

Another interesting example given is how researchers in the US looked at “how video could be used to represent young people’s identity” (Kara, 2015: 109), a very similar topic to my intervention. “They describe video data as ‘multimodal’ because it contains still and moving images, colour, a range of sounds and silences, sometimes text and so on” (Kara, 2015: 109).

“Halverson et al originally approached video analysis
by starting with dialogue, but then they encountered a film that had no dialogue, which engendered their decision to develop a multimodal approach. Their aim was to create a multimodal analytic framework, not to analyse data in different chunks, but to reflect how the interaction of different chunks of data can create new meanings. Following the work of Baldry and Thibault (2006), they divided the film into ‘phases and transitions’, which were units of analysis that had some kind of internal consistency, for example through a type of shot, a consistent voiceover or the same music. Then they devised a coding scheme, based on the work of Bordwell and Thompson (2004), for each unit of analysis. This involved four broad categories based on filmmakers’ key cinematic techniques:


1 mise-en-scène: anything visible within the camera’s frame, such as setting
and characters
2 sound: anything audible, such as dialogue and music
3 editing: the filmmaker’s interventions that create the film
4 cinematography: the filmmaker’s techniques for altering the image from that
seen through the camera’s lens.

Within each category, more detailed codes were developed, such as facial
expressions, clothing, sound effects, flashback, freeze frame, lighting and close-up.

Halverson et al say that using this system ‘to describe the phases and transitions of the films resulted in the creation of multilayered filmic transcripts that allow us to consider each mode individually, as well as how they connect to one another to help youth consider issues of identity in their films’ (Halverson et al 2012: 8).”

Lots to consider here, moving forward with my data analysis.

References

Kara, H (2015) Creative Research Methods in the Social Sciences : A Practical Guide, Policy Press, Bristol. Available from: ProQuest Ebook Central. [14 November 2023].

Intervention: week 5: Focus Group Discussions

During week five we asked the students to reflect on the intervention in their table groups (4–6 students).

I provided the students with discussion topics to help frame the conversations. These acted as starting points.

• Western canon of typeface design

• Diversity

• Decolonising design

• Importance of technical, practical, theoretical knowledge

The focus groups worked really well and created “a direct, intensive encounter” (Vaughn, Schumm & Sinagub, 2013). The conversations were lively and the students helped each other to articulate thoughts and reflections, providing “synergism, snowballing, stimulation, security, spontaneity” (Vaughn, Schumm & Sinagub, 2013).

Conversations with GCD Course Leader, Kira Salter

During my intervention I had several conversations with GCD Course Leader, Kira Salter, who was able to offer me some great advice on Action Research and kindly shared her previous ARPs with me to read.

Some notes below:

I had several conversations with BAGCD Course Leader Kira Salter who really helped me make sense of what I was doing and how to move forward with practical steps.

Kira provided me with her own ARPs and MA Teaching and Learning projects and these have been hugely helpful in gaining insight into the steps and structure of an ARP.

Kira’s ARP on Inclusive Curricula provides some interesting insights:

“provide an inclusive curriculum that anticipates the needs of all students regardless of their ethnicity, gender, their social background”
(Salter, 2020:3)

“By adopting student centered learning strategies that instil a
culture of inclusivity, that not only recognises but also utilises diversity, we can
provide a learning environment that delivers equality in both the access to learning and one that encourages, and nurtures, students contributions.”
(Salter, 2020:3)

This quote in particular really chimed with my intervention ideas, that the diversity of the student cohort provides an amazing opportunity to “utilise diversity” that “encourages, and nurtures, students contributions.” In this way the subject of typography can be made more diverse through the students research and ideas.

These ideas fed into my planning to use Padlet as a way to document the student’s work on the intervention but also so that it could be a communal learning resource for them during the sessions and beyond.

Information about Kira’s ‘Community Resources List’ initiative

“transform a closed discourse into open conversation in pursuit of a better representation of diverse historical, geographical and cultural contexts, as well as underrepresented and marginalised voices.”
(Salter, 2020:1)

Salter, K. (2020) How can participatory approaches impact, re-imagine and decolonise the curriculum? MA Academic Practice in Art, Design and Communication. University of the Arts London

Salter, K. (2020) Action Research Project. PgCert in Academic Practice. University of the Arts London

Focus groups

After talking to Catherine during my tutorial, she mentioned using group discussions instead of individual questionnaires to gather data from the students. She mentioned these would probably yield more results.

After reading:

Vaughn, S., Schumm, J. S., & Sinagub, J. (2013) ‘Why Use Focus Group Interviews in Educational and Psychological Research?’. Focus group interviews in Education and Psychology. Thousand Oaks: Sage. Available at https://methods-sagepub-com.arts.idm.oclc.org/book/focus-group-interviews-in-education-and-psychology/n2.xml (Accessed 27 November 2024).

I found some useful information to back this up and evidence why focus groups might be more effective:

“Perhaps the focus group interview’s closest relative is the individual interview. Hess (1968) noted that the focus group interview offers researchers distinct advantages over the individual interview. These include the following:

1.synergism (when a wider bank of data emerges through the group interaction),
2.snowballing (when the statements of one respondent initiate a chain reaction of additional comments),
3.stimulation (when the group discussion generates excitement about a topic),
4.security (when the group provides a comfort and encourages candid responses), and
5.spontaneity (because participants are not required to answer every question, their responses are more spontaneous and genuine)."

These five points demonstrate the positive attributes of a focus group over individual interviews or questionnaires.

I’ve decided to use this method for my method of gathering data from the students about their reflections on the project.

“Focus group interviews offer researchers in education and psychology the opportunity to garner qualitative data (gathered in small, interactive groups) regarding the perceptions and opinions of purposively selected individuals”

“If conducted properly, the researcher can elicit substantive information about participants’ thoughts and feelings on the topic of interest in relatively little time. Unlike more impersonal approaches to data collection (e.g., paper and pencil surveys), focus group interviews have the potential to bring the investigator closer to the research topic through a direct, intensive encounter with key individuals.”

https://methods-sagepub-com.arts.idm.oclc.org/book/mono/focus-group-interviews-in-education-and-psychology/chpt/why-use-focus-group-interviews-educational-psychological

On designing questions for data collection

In preparation for writing a questionnaire to gather feedback on my ARP intervention, I read two articles:

Writing Survey Questions
Pew Research Center
https://www.pewresearch.org/writing-survey-questions/

and

The Tools at Hand
Jean M. Converse & Stanley Presser
https://methods-sagepub-com.arts.idm.oclc.org/book/survey-questions/n3.xml

In Writing Survey Questions, the author presents various aspects of questionnaire writing and how these aspects should not only be considered but also how question order, context, types of answers can affect the answers and data from these surveys.

Creating questions

Questions should be used to accurately measure the opinions, experiences and behaviours of the group. Questions shouldn’t be ambiguous or biased. Essentially that there is a science that underpins good survey questionnaires.

The author talks about using the same questions at different points in the process, to measure change over time. Something I hadn’t thought about and perhaps too late to add to my process now but I could ask questions that require the respondent to look at how their work and ideas have changed over the course of the intervention.

“Asking the same questions at different points in time allows us to report on changes in the overall views of the general public (or a subset of the public, such as registered voters, men or Black Americans), or what we call “trending the data”.” (Pew Research Center, 2021)

“When measuring change over time, it is important to use the same question wording and to be sensitive to where the question is asked in the questionnaire to maintain a similar context as when the question was asked previously” (Pew Research Center, 2021)

The use of open- and closed-ended questions is a key component of questionnaires and “One of the most significant decisions that can affect how people answer questions” (Pew Research Center, 2021). Open-ended questions are where the respondents provide a response in their own words, whereas closed-ended questions are where respondents are asked to choose from a list of answer choices. As with questions, the order of answers can influence how people respond to them, “Answers to questions are sometimes affected by questions that precede them.” (Pew Research Center, 2021)

Question wording

Part of the design of the questions, “The choice of words and phrases in a question is critical in expressing the meaning and intent” (Pew Research Center, 2021). Questions should be clear and specific, and written in such a way that each respondent is able to answer, using simple and concrete language and avoiding unfamiliar abbreviations or jargon.

One aspect that might be important to my particular survey is something called ‘Social desirability bias’ which is about how people want to be accepted and liked, and may provide inaccurate answers to questions that deal with sensitive subjects and may understate / overstate answers to fit in. As my project is about identity, which can be a sensitive subject for some people, I should be mindful of this.

Question order

Once the questions have been designed, I will need to develop an order for the questions.

“Surveyors must be attentive to how questions early in a questionnaire may have unintended effects on how respondents answer subsequent questions. Researchers have demonstrated that the order in which questions are asked can influence how people respond; earlier questions can unintentionally provide context for the questions that follow (these effects are called “order effects”).” (Pew Research Center, 2021).

Respondents are much more likely to mention concepts or considerations mentioned in earlier closed-ended questions, so again, this is something to be mindful of.

“A questionnaire, like a conversation, should be grouped by topic and unfold in a logical order. It is often helpful to begin the survey with simple questions that respondents will find interesting and engaging” (Pew Research Center, 2021).

In the article ‘The Tools at Hand’, the authors state that “Every questionnaire must, finally, be handcrafted” (Converse, J. M., & Presser, S. (1986)) and that the designer must “cut and try, […] see how people react to it, […] and try again” (Converse, J. M., & Presser, S. (1986))

Like any piece of design, I need to allow time to test, get feedback and make changes to the questionnaire.

Exploration
– Have a clear set of research purposes
– Knowledge of work on the problem
– How a survey could shed some new light

Exploratory study “should take investigators out beyond their own academic or industrial subculture, to new “experts”” (Converse, J. M., & Presser, S. (1986))

“Survey questions, finally, must seem fair to people of widely different viewpoints” (Converse, J. M., & Presser, S. (1986))

The article talks about how inquiry might involve interviews with members of the target population rather than consulting ‘the literature’ to gain a more authentic understanding of members of the group.

“At this exploratory stage, there is little prospect of formally sampling the target population, but interviewing even a few individuals can enrich the researcher’s perspective. Another useful procedure is to assemble somewhat more formally the insiders of a given subculture in a “focused discussion group.” This can be of special value when a target population is likely to have special perceptions, problems, and idioms that may be relatively foreign to the investigator—youth culture, gambling, drugs, prisons, and so on.”

Does this tie in to what I’m doing? As essentially I’m trying to understand the identities of people fro different backgrounds to my own?

Bibliography

Converse, J. M., & Presser, S. (1986). The tools at hand. In Survey Questions (pp. 48-75). SAGE Publications, Inc., https://doi.org/10.4135/9781412986045

Pew Research Center (2021). Writing Survey Questions. https://www.pewresearch.org/writing-survey-questions/