First MMDL episode dealing with Qualitative Data in detail – but it won’t be the last!
Check out this wonderful conversation with Professor Helen Dickinson about rigorous Qualitative Research, and the questions it can answer for us.
“one of the challenges is you know We all have our own histories and we all see the world in a particular way and sometimes we assume that everybody else kind of has those sorts of experiences and the reality is they don’t”
“One thing that sometimes frustrates me is qualitative research being seen as kind of not as scientific or more soft than kind of you know Quote-unquote kind of hard Research”
“one of the other things that sometimes gets said is that qualitative research is subjective because it’s based on people’s thoughts and perceptions And to that I guess I’d say well, you know to a degree is but I don’t think there’s any approach at all in research that doesn’t involve subjectivity”
Transcript
Linda: Welcome back to another episode of make me data literate. you’ve often heard me say that data can tell you what but it can’t tell you why, and That’s not entirely true because qualitative data can tell you the why, and To date I have sadly neglected qualitative data on this podcast. So we are here today to try to start rectifying that with Professor Helen Dickinson. Welcome Helen
Helen: Hi Linda
Linda: Thanks so much for being here. This is exciting. We’ve got some new ground to explore
Helen: Thanks for the invite
Linda: So tell us who are you and what do you do?
Helen: Yeah, so I’m a professor of public service research at the University of New South Wales in Canberra and So what that means is I do research and I teach Mostly in the space of public policy and public Administration and I’ve got a particular interest in disability policies and services. So how they’re designed and how they’re implemented and what impacts they have for people in practice. So I spend a lot of time talking to Policy folk and people with disability who use those services, to really understand them in more detail
Linda: That’s really interesting and I know you have strong feelings about things like the NDIA and and Robo debt and things like that. How did you get into that kind of line of work? I didn’t warn you about that question So you can dodge it if you want to but I’m fascinated to know how that happened
Helen: Um a fairly long and winding road really. so My first degrees were in geography And through that I got an interest in in health and in policy. Went away and worked outside of academia for a bit and then decided I wanted to do a PhD. managed to get a kind of a scholarship to do one in in Broadly in health but interested in how health organizations Interacts with others, like kind of education and social care and things like that. and through that Really kind of got an interest around particular groups who interact with lots of different kind of service systems. So like older people and people with Disability, and really I just yeah, it just kept on growing and I am always interested in different sorts of Things and so I just followed the path and ended up where I did. I’m a accidental academic
Linda: That is it’s such a theme on this podcast and I think actually it needs to become a permanent question that Nobody I’ve interviewed has kind of set out – almost nobody set out to be where they are – These long and winding career paths fascinate me and it’s not the way we teach careers at school, but that’s that’s a separate rant! What did you have to learn to do your work was there anything you feel was missing from your formal education? particularly with the data focus
Helen: Oh look heaps and you know one of the great perks of this job is continually get to inform how you work I guess one of the big things for me between my formal education and what’s really important to know in the work that I do now is kind of the difference between what like the books and the theories say happens and what actually happens kind of in practice so You know spent a long time getting to know kind of what’s the reality for policy makers? So, you know, we know that you know the books tell us there’s a policy cycle and that’s how policies get made but it doesn’t really tell you kind of how stuff actually happens in practice, and what some of the constraints are for Policy makers when when they’re working and it’s really important to know that when you’re thinking about kind of design and implementation of policy.
But the other big area that’s important for me is kind of the lived experience of people who are on the receiving end of policies so I Guess one of the challenges is you know We all have our own histories and we all see the world in a particular way and sometimes we assume that everybody else kind of has those sorts of experiences and the reality is They don’t, and so I spend a lot of time talking with advocacy Organisations and people with disability to kind of really understand how the world might be different for them And so what other sorts of issues that they might might face. so I mean this to some degree is something you can’t really You know always teach in formal teaching, you need to kind of get experience and practice doing, but one of the biggest things have been about understanding What are some different ways to ask questions and and to frame them to understand what you need and really importantly? How do you listen? And that’s something that needs a lot of practice, but how can you listen to make sure that you really understand where somebody’s coming from?
Linda: That’s I love that with the design of systems so often stops at the you know at the release phase you design the system you push it out there and then you stop the design process and you stop looking, but actually that’s where the design process really becomes even more important to go what hang on how how are people actually experiencing this? We have a family member in hospital right now and you know the descriptions on the website and in the literature about how the the patient experience of that hospital is supposed to go are not at all reflected in our experience or his experience of what’s happening and and so that disconnect between this idealized you know textbook version and the lived experience is something that doesn’t get enough attention.
Helen: Absolutely, and that’s been a core that’s really kind of the core focus Of the work that I do and I lead a research group That you know we have a number of different researchers who are interested in that issue because you know what you can carefully package up in a design phase you know you can you can think about a bunch of things But you know the really interesting bit is what happens when you get out into the messy world And people don’t behave in the way that you expect them to and so yeah, that’s the thing. I’m really interested in
Linda: I Love that. That’s so interesting. What are the challenges that you face working with qualitative data
Helen: I Think sometimes the perception is a big challenge of that so One thing that sometimes frustrates me is qualitative research being seen as kind of not as scientific or more soft than kind of you know Quote-unquote kind of hard Research so more than once when I’ve had Reviews on a journal article where I think it’s probably gone out to more quantitative researchers I’ve got comments saying oh this seems to be anecdotal or a form of journalism and not scholarship, and you know one of the other things that sometimes gets said is that qualitative research is Subjective because it’s based on people’s thoughts and perceptions And to that I guess I’d say well, you know to a degree is but I don’t think there’s any approach at all in research that doesn’t involve subjectivity so you know That’s it. Yeah
In quantitative research you’re thinking about you know, how am I gonna operationalize variables? You know what models am I gonna run? Which outliers do I exclude? You know all of those use a form of judgment, you know So there’s a degree of subjectivity to them
Yeah, so probably in qualitative research, we don’t see subjectivity as a flaw, right? So it’s a part of the method, And what we want to do is we want to explore human meaning experience and context and they are all you know subjective and we want to study them them directly so You know if we take a health example because we you know, you would just raise one there Linda You know if you want to know about how people experience chronic pain for example The only way you can know about that is through the subjective account And so, you know good quality Qualitative research acknowledges all of this, you know, it’s systematic and follows quite rigorous methods. And so, you know, we have different ways of understanding what’s meant by reliability and validity when we compare it to quantitative studies. And you know, we take particular approaches to make sure that you know we’ve got kind of transparent systematic and rigorous research.
So, you know, we don’t pretend that researchers are detached observers, you know we recognize their role and and build it into our practices.
And so when we get accused of being a softer kind of you know, anecdotal science, I guess that makes me frustrated
Linda: yeah, I’m right there with you on that and it’s really interesting I didn’t plan this but the last few episodes of the podcast they have it keeps coming up the idea that the myth really that quantitative data is objective that that any research that we can do is objective We even in what we choose to study there is subjectivity and then the particular questions we ask and the categories that we create I just did an interview for another podcast which hasn’t come out yet but it’s going to be super interesting about the idea of the categories and that we have this idea that categories are their objective and they’re clear and they’re scientific and you know, we’re making this stuff up. The category of academic what does that mean? like it’s we’re so you know, even if you look at race, the category of Caucasian or the category of British, or the category of Indigenous Australian: What do they mean and are they clear, you know, are we talking about genetics? are we talking about percentage of you know Pure blood. What does that even mean? And it you know, I think it’s highly offensive But also like it’s meaningless! you do these DNA tests and they give you this percentages of where you’re from and it’s all you know, it’s yeah, so the idea that quantitative is Objective and qualitative as subjective is nonsensical. It’s all you know, people are involved in all research, and that means that it is inherently subjective and I think we we It’s very dangerous to say Quantitative data is objective because then you get the idea that you can’t question it and that’s That’s a really dangerous road
Helen: Yeah, that’s right and all research can be critiqued, you know, there’s no perfect – with all the money in the world, There’s no perfect research, kind of you always have to make a series of value judgments And there’s things that you look at and there’s things that you don’t look at and you’ve got particular ways of doing analysis And I think with all research and I mean this comes through in your podcast really well Linda is that you know being clear about what you’re doing and why and being transparent about those those Judgments that you’d made is really important.
Linda: Yeah, and that means you have to know that they’re there and and and when we teach quantitative data is this pure objective thing we gloss over that and try to pretend it’s not there. And I you know, it’s the problem I have with third-person Papers paper writing in science as well, you know, this was done. It wasn’t I didn’t do it. There wasn’t a person who did it. It was done by some pure, you know God of Science,
Helen: Absolutely
Linda: Yeah, it’s crazy. It’s Fascinating to me how this theme is coming through without me deliberately setting out to chase it. Have you ever seen data deliberately misused?
Helen: Yeah, of course and and there’s always examples with any method of data being used Kind of badly whatever the approach and you know largely often, you know relates to people not following the methods Right, I think it’s probably important to kind of separate out What’s just kind of sloppy use of of qualitative data and where it’s been deliberately kind of misused for things like political or Organizational ends.
Linda: Yeah
Helen: So, you know generally it’s a problem where you know, it’s used as a casual kind of commentary around a topic rather than it being rigorously kind of analyzed so, you know, if you read something you can’t find out who was involved why they were chosen or how they were recruited or how the data was analyzed and then generally that’s something that should make you worry,
And the sorts of things I guess we see when people misuse qualitative data would be stuff like cherry picking data. So, you know, people might just pick out the odd quote or experience that fit a kind of predetermined message and ignore any contradictory kind of account Or people could kind of decontextualize quotes so they you know pick out a bit that that seems to suggest one thing when they’re actually talking about a very different issue.
The thing you probably see quite a bit of is over generalization. So where people have a quite small or specific kind of group and they make claims as if they reflect a much bigger group
So sometimes in the policy space, you know, we might want to think about what the impact of a policy is but there’s some groups that are really seldom heard from in research So yeah, I think in a disability space that might be people who Don’t communicate with with words with with kind of verbal language. People with significant intellectual disability or people who are kind of houseless And there are examples of groups that that need kind of specific skills and resources to find and to engage well. And so sometimes they get missed in research processes and so, you know, we might say oh, yeah, generally a policy is well received, But that might be because we’ve only actually spoken to a small group of that population and sometimes they’re not the people with the kind of more complex kind of support needs and and lives and so that might not be completely kind of accurate.
Linda: Yeah, that’s a it’s a bit like choosing the people who chose to respond to the survey, you know, there’s a big gap. Yeah.
Helen: An opt in survey or something like that.
Linda: Yeah
Helen: And they’re often the groups who kind of you know will have the greatest challenges with those sorts of things because you know, the reality is most people who make policy, make policy for people like them and you know, the reality is in across most kind of Australian federal and state and territory governments. They’re people who are predominantly middle class They’re people who have gone to university, Have a relatively high level of income, and probably haven’t you know used homelessness services, or Maybe not have interacted with welfare payment services and and things like that, and so they make a bunch of assumptions about people’s lives.
and so You know, you mentioned robodebt earlier and and I think one of the things that Fascinates me in some of the accounts around that is the point at which the government Realizes that that’s something’s happening, So there’s been all these reports in the papers and people saying I’ve been getting these debts, And I shouldn’t have got them and there’s there’s kind of a point in time where basically somebody in the government goes to one of the kind of main players in this and says I’ve received this Debt and that doesn’t seem right and they’re like, oh, wow. Well if you got this debt, you’re not the sort of person who should have got one of these and that’s the point at which they go Oh, maybe some things kind of going wrong here.
Linda: Yeah,
Helen: and so we really need to understand those kind of you know the full range of experience of people’s lives to make good policy.
Linda: Yeah, that was it’s really hard to see or even Understand there are issues for people whose experience is very different to your own. I’ve done a fair bit of work in in the usability space with technology but we had a keynote at PyCon in In September from Laren Le-Gassick talking about accessibility and she talked about her dad’s experience of using a microwave, because he was very low vision and in the end he’s I think he’s completely blind now, and She talked about they used to have a microwave with a dial so you turn the dial and you get a certain amount of time, and you can feel where the dial is and I had never thought about the fact. They blew up that microwave and they couldn’t get a new one The only Dial that they could find was the dial turned continuously and you had to be able to read the time on the on the system, so you couldn’t tell by where the dial was how how much time you’d set it for. There are these cute little things the tactile stickers that you stick on the button so you can tell where the buttons are, But you still can’t see what it says on the timer, you know things that you just don’t think about so these wonderful shiny smooth panels that we have where you press buttons that don’t really exist, they’re just sort of sensitive areas on a perfectly smooth panel, and It’s it’s mind-blowing when you think, There is a there is a An entire experience of the world that I have no idea about and that I’m completely ha blind to. You just unless you know someone who’s had that experience you don’t even know that it’s a thing
Helen: Totally and I’ve you know, I’ve got interesting kind of you know tech around caring as well and What it’s the most me I’ve gone around to a few kind of Exhibitions where people have kind of been developing some of this sort of stuff, And what we’ve seen is I mean a lot of that development particularly when robotics and stuff like that used to come out of kind of defense funding, but interestingly more people are moving into kind of health and disability spaces because actually kind of funding has you know the bigger market around those, because we’re living longer and with kind of more disabilities and we’ve had before,
Linda: more importantly Rich people are having more disabilities
Helen: Exactly. We go around and see these kind of things that people were developing in the health or aging in the disability or aging spaces and They would say this is a solution to this issue, and I would say to them So which you know which older people or which people were disabilities have you engaged with to identify that and to kind of work through, And and they hadn’t basically.
you know that space was a vehicle for the development of the thing that they wanted to do and And and you know, that’s problematic I mean it’s problematic for the thing that they’re developing but the results are problematic for them because if they don’t understand how people’s lives work. They’re not gonna develop the things that work for them. So that engagement element is is so crucial.
Linda: You just described almost every tech device. I’ve ever interact with Yeah, they’re all designed for young straight cis white men
Helen: Having said that though, I mean there is such a growing and I mean people with disability have always been incredibly Innovative with technologies It might not be necessarily kind of a bigger scale. They were starting to see some scale of that I mean people take use of things around us all the time like you know and smartphones and things like that and develop really really great kind of Assistive kind of supports in terms of in terms of their lives But it would just be great to see kind of the industry recognize that and build on it rather than you know starting from a different place
Linda: Yeah, and so often when you build technology that works for a broader range of Differently abled folks you wind up with something that works better for everybody in fact
Helen: Absolutely, it’s like the whole debate around inclusive education, right? So if we kind of make You know if you think about education and how that works, you know, there’s quite a lot of literature that suggests that you know if you think about You know how you design your curriculum for folks with different disabilities you’ll also often have the impact of people who don’t have English as a first language or other kind of Diverse needs will be able to engage with that more as well.
Linda: Yeah Yeah, that I mean, that’s a whole other podcast I could talk to
Helen: That’s a different rant.
Linda: Yeah. Always have to try to keep myself at least a little bit on track If someone presents you with a qualitative study What questions should you ask what questions should we ask when someone comes to us with qualitative data and says this shows X Y Z, how do we How do we sort of interrogate that to know that it’s good quality?
Helen: what you’re looking for is quite a bit of you’re looking for transparency. What did they do, and why? what was the kind of justification for that? So, you know, I want to know how the data were collected and why? who participated? Why was that group kind of approached? Why would they have a view on the issue that you’re looking at? How many people were Involved? what context did they engage in in that research in? what questions were asked and why? so a bunch of stuff about how kind of the design and how data were collected
And then when you get through to the analysis, you know, what analytical technique did they use to interrogate the data? So there’s quite a few different ones in qualitative data So was it, you know, thematic analysis, narrative analysis, grounded theory? You know, why was that the most appropriate sort of approach? did you use software in analyzing the data and how did you how did you kind of use that? One of my real bug bears in qualitative research and I think you talked about this before, is where somebody says We looked at this data and these themes emerged, you know, they didn’t they didn’t just rise up and kind of stand up and say, “hey, we’re over here!, you know, they’re interpretations.
So how did they get to that? Was that you know, it was an inductive process where, you know You looked at the data and you found everybody was talking about broadly kind of similar things, Or was it a deductive approach where you kind of know stuff from the literature, and so you went you went looking for that so really a kind of an exploration and a Justification for that. And then when you kind of get into the right of I want to see kind of a link between the data and the findings. So I mean what I mean by that is, you know, I want to see direct quotes or observations or field notes So we’re really using those those if it says if it’s quotes as kind of evidence
And not the researcher saying oh, yeah, everybody was generally kind of satisfied with this issue and didn’t use any kind of evidence to demonstrate that. But similarly I also don’t want to just see kind of a quote dump, So where you’ve got like a quote page of just you know different sorts of quotes. without any kind of Analysis so, you know, you really want kind of balance of the participant voice with the Interpretation. I want to know that the research was done ethically, so Was there a formal ethics process? did people consent? Are the data being treated confidentially, and if not why? are they being stored appropriately? Will you be kind of Informing the participants of what you found so, you know, there’s in university research, There’s kind of formal university ethics and then there’s I see in a different thing of kind of broader ethics of research.
But if people are giving their time to be involved in something generally, you know I want to be communicating that back with them and checking that we’ve kind of got the interpretation Right. and then you know in you presenting that data, Are you doing it with enough context so I can understand it? But also so that people can be treated kind of confidentially, so, you know, you can not use people’s names, but you can report identifying details.
So if you’re doing research with a kind of a multidisciplinary team and there’s like one nurse and one teacher and one social worker, right? It doesn’t take a lot of identifying detail to be able to work out whose quotes are which. And then I guess finally some sort of acknowledgement of the researcher role and reflexivity. So what’s the position of that individual in the background, and the assumptions that they come to the research with? That might have influenced the data collection and and interpretation
So really it’s just kind of stepping through all of those different bits and making sure that you know, you got a full understanding of what happened and why those decisions were made along the way.
Linda: Yeah, it’s it’s that it’s that idea of fundamental transparency, isn’t it the idea of you know the whole idea of of the The peer review process and and publications and stuff is that You can replicate it and and if you replicate it you should get the same results, and we actually haven’t done that in quantitative data very well, and we’re still not doing it very well in that often people don’t have access to the data, or the the method is not sufficiently detailed, or the you know, you just, you don’t actually know how the research was done in enough detail to replicate it.
And I think that’s at the base or one of the things at the base of of our broader replication crisis that you can’t replicate something if you don’t really know what was done, but you know with a six-page conference paper or even a you know 20 or 30 page journal article it can be difficult to get the detail in while still discussing the actual bigger picture stuff.
But you would think in the age of the internet that you could have you could you could have a systematic way of of storing the details even if they’re not in the publications, but they could be you know accessible. we’re just not very good at systematic things.
Helen: With more online with more with the journals kind of being less focused on the you know when they were printed, and that’s the only way you got access to them. There were real pagination issues around that, so when you had kind of additional kind of content that was it that was a challenge. There’s you know often, you know, well now because they’re online we’ll use a lot more supplementary tables to explain how we got there. So you might have a you might kind of put in there your kind of coding framework, all your parent and your kind of child nodes in there, and talk about how you got those in more detail? I mean there are challenges. I mean with qual data kind of like there are sometimes challenges with making your data accessible to other researchers to be able to do the same sort of analysis on because of the kind of stuff around confidentiality and privacy and things like that that’s right.
Linda: Yeah, exactly.
Helen: And so but even you know, I mean generally in the sorts of fields that I publish in you Probably got about a thousand one and a half thousand words at the most for your kind of methodology So it can be tricky to kind of fully explain it in that, But there’s enough space to broadly justify a number of those things particularly with the supplementary material and it’s in it’s about making sure all those different components are in there.
Linda: Yeah Yeah Yeah, it’s that it’s really about being transparent and being systematic, isn’t it?
Helen: Mm-hmm, and it’s so for every method, right? It’s not just qualitative research. It’s all no.
Linda: Yeah, 100% I’d like to say how does qualitative data inform policy, but that might be an overly optimistic question. How should it inform policy?
Helen: But yeah, I guess it you know like all things it depends and I think it can play a really important role in a number of Ways, and the sorts of stories that you can gather from qualitative data can be really compelling, right? So they’re real people’s experiences and they can be quite emotionally engaging and So, you know what we’ve talked about today is qualitative research really about understanding why people think or act in particular ways or perceive things. And so it could be really good way to understand the lived experience behind numbers. So it can help design interventions that fit people’s real-world Experiences,.
You know, if we find that there’s a low uptake of a service, you know one interpretation of that could be well, that’s not needed, right? Actually, it might be but it’s difficult to assess for particular reasons. So, you know people can’t get to it. There’s no bus that goes there or something like that, And it might also tell us about unintended consequences, you know, though policies often have Side effects that we don’t capture in statistics, and so we need to know about those. But it also tells about you know Who misses out, and why, and some of those reasons, So there’s another way that it can inform as well, which is pretty a big topic in the disability space at the moment, and that’s around the issue of kind of co design and making sure that people with disability shape policies, So that they’re more relevant and more kind of legitimate. And qualitative research can play a really important role there.
In most of my research as a qualitative researcher, I work a lot with quantitative researchers So when we want to kind of really compel policy makers to do something, We kind of bring together quant and qual data, ideally some economic data. We know that different people respond to different types of data and so having a package of that evidence can be really compelling but it can give us a more full kind of understanding of policies and their impacts in kind of depth and breadth.
So, you know While it doesn’t always feel that the data informs policy, So I think it does in quite a few times, and there’s a space to do some of them.
Linda: That’s encouraging, that that gives me hope. It sometimes feels like there’s a lot of you know You see governments doing a lot of stuff where you go, we know that doesn’t work. Why why are you still doing that? So it’s good to hear that often it does
Helen: And I guess that’s because, While we talk about wanting to have evidence policies and that is true, there’s also a series of constraints and a series of values that underpin how we make policies and so, you know while governments talk about wanting to be evidence-based, actually they don’t want to be wholly evidence-based, right? So and you find yourself in real conundrums around evidence.
So I remember working with a health organisation in England years ago, and you know, essentially their kind of primary target was to improve kind of life expectancy, was what they decided to try and sum up in one kind of measure, what they were trying to do is to do, and then we said okay, but if that’s your aim, essentially, you probably want to stop spending money on services for older people and put all the money into kind of trying to prevent childhood accidents and injuries, that’s the only way you’re gonna get there, right?
Yeah, and and so this is the challenge, you know often evidence – something might be a really good evidence base, but it doesn’t chime with the values of how we think about things. I mean in kind of in drugs in around pharmaceuticals and and things like that we often have rules about ee’ll spend a certain amount of money on a drug that proves this kind of effective
But then when you get to people who’ve got very rare kind of conditions, we might be willing to spend a lot more on something that’s not haven’t got the same level of efficacy because we know there’s not a lot of money that’s put into researching those particular issues, so evidence intersects with the number of different things in different ways, but it’s certainly in the mix in there.
Linda: I like that it’s a more nuanced way to look at things and um often… One of the things that keeps coming up again and again in the in the podcasts and also in life is you know if you if you’re really certain about something, if you’re getting really strong results, that’s a big red flag.
Helen: It can be yeah. Yeah, sometimes.
Linda: Yeah, yeah, not always but often certainly a point where which you look back and and you know examine things closely and check your assumptions.
This has been so interesting and I’m just fascinated by the themes that keep Coming up and and and the way The directions it points definitely going to have to do some more qualitative interviews on the podcast. What is it that excites you about data?
Helen: Is it too nerdy to say everything?
Linda: No I love that!
Helen: I don’t know, I mean probably the most is where I see a really kind of good example of a well done project that finds something new, and it changes the way that somebody like sees a problem or thinks about an issue. So you know the role that that data can play in the way that people make sense of the world, and they think about kind of their priorities and and you know and how they’re going to go about doing things. I think yeah that that that’s probably the stuff that excites me the most
Linda: That’s awesome and that’s exactly why I do the real projects in schools where I get them to solve real problems and learn these skills in that context because then the students say that AH! these are skills that are not just for solving textbook problems. These are skills that I can actually use to make change and That’s power. That’s amazing. I love that
Helen: Yeah, absolutely. No, and and that’s what’s great about it. It gives people The power to do things in different ways or see things in in different ways and yeah, no, that’s that’s incredible and especially when you I mean either in my role don’t teach kind of much around the the younger end, but certainly I do a lot of work with disability advocates who have children and young people and and seeing when you give them knowledge around Kind of something how they can go away and make a difference with that is just huge
Linda: Yeah That’s awesome. Thank you so much. It’s been a fantastic conversation I’m really grateful to you for sharing your time.
Helen: Thanks, Linda
