Methodological Challenges to Measuring Transformation

When I set out on my dissertation research, my main question was whether and how interfaith dialogue functions to transform people. I had a hypothesis that people do interfaith dialogue because when doing it they experience a shift in perspective, behavior, or identity construct. I wanted more detailed data about the transformation that occurs through interfaith encounter.

But after a year of research, I have found that transformation does not concern all dialoguers. My starting hypothesis has to be broadened because, in the interfaith reconciliation and dialogue world, not everybody is focused on transforming the world or even achieving a huge social impact. I generally see two categories of investment.

One is the people who strongly claim that interfaith engagement must accomplish something, must be productive, must make a difference. I see this as a very worthy and crucial impulse, although I become more skeptical of its credibility as I delve more deeply into the incredibly numerous methodological complications of identifying, measuring, or guaranteeing the presence of such progress or efficacy when it comes to matters of subjective understandings, relations, consciousness, and transformation of long-held beliefs and thought patterns. That said, I don’t think these people should stop trying. I think I naturally tend toward membership in this group—concern about efficacy. I say I incline this way “naturally” because I do think this preoccupation is a matter of deep-seated temperament, and even though I am well-versed in the empirical challenges of establishing activistic efficacy over anything other than a very, very longitudinal study (like “the arc of history” or “since Vatican II”), I still stubbornly cling to the idea that people who are worried about the world should be doing something to improve it. I think this inclination is my temperamental response to my existential anxiety. Despite the trouble inherent in developing firm quantitative, material indicators of progress, dialoguers focused on efficacy tend to plan their activities in contexts that would be more likely to make an impact, such as political meetings, local school curriculums, and local hospitals.

There is a second category of interfaith engagers: those who are not overly invested in efficacy. They are concerned about mourning. Deepening relationships. Following their curiosity. Supporting each other. Enjoying meals together. Getting through another day. Creating joy. Articulating values. Human being. And when I liberate myself from a productivity-oriented hunger for efficacy and dwell in human-merely-being (to quote ee cummings), I glimpse the necessity and gift of this position too. Methodologically, it is a more sound investment to allow the dialogue itself to be the product of dialogue. Personally, I believe that it is probably healthier and more honest.

Overall my sense is that we need both approaches. In fact, I believe that interfaith dialogue is more likely to last as long as both approaches are intact, because the healthiest, most sustainable ecology is predicated on diversity and hybridization. 

Now, I’ll delve more deeply into the difficult task for interfaith activists who want to inspire transformation through dialogue.

The first methodological challenge of measuring dialogue-inspired transformation is to decide what exactly should be measured. If one could identify concrete legislative or policy shifts that arose directly as a result of dialogue efforts (such as the peace dialogues conducted by Sant’Egidio officers in Mozambique and the Balkans), these could be used to indicate a type of transformation. With a very longitudinal study, say, “since Vatican II,” we could demonstrate that more people are attending interfaith programs. We could count the number of participants at certain events, and compare the demographic of religious belonging at interfaith events to the general local religious demographic. But I have found that many interfaith organizations do not count heads or record such numbers. Moreover, it is notoriously difficult to measure religious belonging—how much harder would it be to measure the percentage of religious belongers participating in interfaith dialogue? It is very hard to develop a concrete, quantitative index of efficacy for the interfaith movement. 

If we want to measure personal, subjective paradigm shifts or whether people feel they achieved “mutual comprehension” (a frequent goal of dialogue), there are a number of psychological tests available, such as The Intercultural Development Inventory (Hammer), the Attitudinal and Behavioral Openness Scale (Caligiuri and Jacobs), the Interpersonal Reactivity Index (an empathy scale by Mark Davis), or scales that measure capacities for encountering difference, such as Differentiation of Self (Skowron). These are quantitative measures for personal change that introduce all the limits of using surveys and ratings to express personality. Quantitative personal narratives of change can be collected, but this form of data also introduces confounders regarding self-reporting, memory and consistency. Both quantitative and qualitative research on personal transformation primarily reveal impressions or ideas of change, or changed perspectives, but say little about changed behavior. Most of my subjects describe or rate their experiences of transformation in cognitive terms, describing attitude and perspective shifts. This may show that people may feel differently but do not always do differently, or perhaps (among my subjects) that their behavioral starting point was not religious violence to begin with. Amongst the majority of interfaith activists, the personal movement is not from violence to peace but from ignorance to education, from insularity to friendship.

Therefore the efficacy of dialogue is very hard to demonstrate quantitatively, or more concretely than as an object of the narratives and discourse of the interfaith society. The fact that interfaith dialoguers often relate narratives and discourses of transformation reflects the intensity of their interest in and commitment to transformation—even if this data is not a concrete indicator of progress, and may not even clarify what constitutes “progress” in dialogue.

When communities speak of transformation, are they merely finding solidarity in their shared values, in the content of their conversation? What will become of their community if they realize that the very product of their dialogue about transformation may, in the end, be their dialogue about transformation? Is “transformation” more of a dynamically motivating Platonic ideal than a reality?

Anybody over 30 will probably say no. We know we’ve changed over the years, in some ways for the better, and some of us as a consequence of direct and focused efforts to transform ourselves. Maturity is a real phenomenon. How can we count it? Most of us can’t—but we still know it is happening. Measuring transformation remains a mystery, seemingly one of faith and private experience, of personal testament, of certain happening but constantly shifting substance. Its elusive aspects do not allow us to deny its reality.

I’m drawing a not-so-subtle parallel between the experience of faith and the experience of transformation. Both are predicated on things gone unseen and unmeasured. Most faithful people find it immaterial that their faith experience cannot be measured, and might say their faith is not subject to empirical demonstration. After all, it wouldn’t be very faithful if a faithful person abandons their faith in the absence of evidence–or if their faith is predicated on generating empirical evidence (for example, continuously seeking proof that prayer affects healing, even though the amalgam of well-conducted, controlled studies are inconclusive or detect low correlations). Mature faith actually takes in the observable world and makes sense of it in the terms of their elastic, embracing faith. Most forms of faith evade measures and proofs. The presence of faith in God is no proof of the existence of God.

And yet–and yet–experiences of faith or transformation can determine everything about a person’s quality of life and their level of motivation to keep engaging with–and striving toward–these constructs of faith and transformation. Even if God and social change are “merely” dynamically motivating ideals, the journey of standing in relation to these ideals–and sometimes experiencing them to be as real and true as any documentable piece of material evidence–can make life rich and meaningful.

So between those dialoguers who aim for change and those who aim for relation, may there be understanding that we need both of you.

Image courtesy of Wikimedia Commons.

7 thoughts on “Methodological Challenges to Measuring Transformation”

  1. I enjoyed this essay very much. Like you I believe we need people in both categories of interfaith dialogue, ideally a blend of the two in my view. But I think I differ in appreciation for the significance of measurement possibilities in transformation. Identifying and utilizing such measurements would mean our efforts have a real world impact, extremely significant in a post-9/11 world. Surely such indicators should be of interest to non-profits working in this area, as well as grant funding organizations who might fund them. I’d love to see your dissertation on this if it’s finished. Thanks again.

    1. Thank you, John! The dissertation hasn’t even begun! You’ll have to wait on the edge of your seat for about a year, I’m afraid 🙂

      I think we share an interest in generating methods and methods to detect transformation and social impact. I think a well-design longitudinal study could reveal a lot. If nothing, my diss research has shown me that fieldwork raises more questions than answers, and I know what I would do differently and that I don’t have the resources to do it. I think this measurement project is the next thing to tackle……

  2. I enjoyed reading your article. Thanks for the careful thought. You might be interested in a detailed typology of different forms and purposes of dialogue that is available in an unpublished paper I researched and wrote. Let me know. Best wishes, Randall Paul

Comments are closed.