A fantastic chat with Dr Jake Clark, Principal Evaluation Advisor | Impact & Evaluation for STEM Education at CSIRO. Jake talks about using evidence to make STEM programs the best they can be, among many, many other things!
“There’s so many great initiatives out there and so it’s a matter of like – How is it working and more importantly, who is it working for and who is it not working for?”
Transcript
Linda McIver (00:00)
Welcome back to Make Me Data Literate. I’m very excited today to be delving into the area of STEM education and rigorous evaluation of programmes, which is something bizarrely I haven’t talked about yet on the podcast. So I think this is gonna be really fun. And my guest today is Jake Clark. Welcome, Jake.
Jake (00:19)
G’day Linda. Thank you so much for having me. I’m really excited for this conversation. And I know, I know it’s probably gonna go on tangents, but I’m excited for those tangents. I’m so prepped for them.
Linda McIver (00:31)
Me too. The tangents are always where it gets interesting. So tell us, who are you and what do you do?
Jake (00:38)
Yeah, ah so my name is Jake Clark and I work as a principal evaluation advisor for CSIRO’s education and outreach team. And so CSIRO’s education outreach team is the national science agency’s education arm and we’re supporting young people all across Australia to build their STEM capability.
And so as a part of my role I work in this sort of space of data science and research and evaluation to try and better understand why fewer students are choosing STEM in their tertiary education and in secondary education and then also working out what works when it comes to STEM education programs. There’s so many great initiatives out there and so it’s a matter of like how is it working and more importantly, who is it working for and who is it not working for? And so we can better work out better ways to engage young people in their education and to expose them to better STEM pathways.
And then also, when you go and collect all this information and find out all these amazing insights, then it’s a matter of, as we’re sort of discussing the before, sometimes the bit of the boring bit the but the most important thing is writing it all down and then being able to share and showcase that evidence with the broader STEM outreach ecosystem. So whether that’s people like yourself, Linda, who are working in the sort of STEM space and trying to work out well how do I, what’s the best way to engage teachers in terms of building up their capability or capacity in data literacy or it could be around, you know, what are the best ways to embed inquiry based workshops into regional schools, or how do we bridge mentors, industry mentors into low SES classrooms or whatever the case may be.
So, it’s a matter of being able to find out what is working and for who and to bring the, I think of it as a sort of rising tide, and always try and be a strength a sort of a strength based approach as best you can. Because we’re all trying to do, especially in sort of STEM education and outreach space where people are so passionate in in the in this in this ecosystem and it’s sometimes it’s a bit challenging to work out what is and isn’t working. And so to be able to have the space and energy to to be able to do that, to then say, “hey look, we’re doing amaz- you’re doing amazing things. What we might need to do is little bit of tweak here and there and bring that all up”, because in some instances smaller organizations don’t have that capability or have that energy or have that person that they can just dedicate to do all this to do all this work. So I’m very excited to be a part of CSIRO’s education outreach team to be able to do all that as my day to day job.
Linda McIver (03:40)
That’s so awesome. I love the idea of evidence based education innovations ’cause I talk about that all the time, that so often the schools implement new programs or they get very excited about a particular technique or a particular approach, but the capacity and skill to actually properly evaluate it is is often missing. So it’s really exciting to see.
Jake (04:07)
Definitely. And I th- I think what we need to understand is particularly with the work that we have been doing and the exciting work that the impact and evaluation team, which I’m part of, is doing is looking at as the ecosystem as a whole and and what we could do differently, is thinking about: What works? Why is it working? And what is the evidence base for that?. And then as I mentioned a little bit earlier, like: Who is it actually working for?
And particularly in in data, you’ll get like a sort of a single number that says, oh well, confidence in math has increased by thirty percent and you go, okay, well that’s cool, but is that significant? And you know, is that because of who you’ve actually been delivering that intervention for? Is it the fact that you’ve been able to deliver that intervention to schools that already have that capacity or the teachers are already on board with that type of pedagogy that they can then instill that confidence in them? Or is it the fact of, oh actually we have been specifically targeting teachers who don’t necessarily have the background or skills and we can see in government reports that, you know, some a lot of teachers are teaching out of field. And so not having the confidence and self efficacy to then teach and be confident about what they’re teaching and then so that then also, I wouldn’t say ooze ’cause ooze doesn’t sound right, but then kids can feel that. Kids can feel that confidence and go, well if you’re if –
Linda McIver (05:39)
Yes. Yeah, they can.
Jake (05:41)
If you’re not confident, then how can I be confident within my own within myself and be able to be confident in the in these skills? So and so you know if you’re targeting these specific people for this specific purpose, then that’s great, but then you sh- your evidence should be showcasing that and go, okay, well going back to that sort of single number, well that’s great, but if we actually split this up into teachers that have low capac- who have had low capacity before or high capacity, actually it seems like this intervention is working as intended and the lo- the lower capacity teachers have like a twofold increase in their confidence as opposed to the higher capacity ones because they already had that sort of baseline to begin with or whatever the case may be. So again, it’s about sort of working out like why why is it working and and who is it actually working for.
Linda McIver (06:39)
That’s a super important insight as well. The idea that something that works in one school or with one teacher isn’t necessarily gonna work with other teachers or in other schools and that idea that context matters and oh boy, the teachers teaching out of field thing is such an issue. It was actually one of the things that shocked me the most when I moved from academia into teaching was this idea that teachers could be asked to teach anything at all and and without any qualifications. And indeed in digital technologies that’s so often the case because we just don’t have enough people with digital technologies qualifications to fill the positions. So it’s it’s it’s a really interesting problem, particularly because they don’t do that in in in all of the other countries. So France, for example, I know doesn’t, because I have friends in the teaching system there, does not allow people to teach out of field. It’s just not a thing. They don’t do it. So it’s obviously, it’s a solvable problem. But we we certainly haven’t solved it. It raises some questions, I think. Yeah. Okay. So moving right along. What did you have to learn to do your work?
Jake (07:35)
Wow. Yeah.
Linda McIver (07:49)
Your, all your evaluation stuff. Was that was that something that you came to the job like fully equipped with from your formal education? That’s a that’s a big no on your face there.
Jake (07:57)
No. No. And previously listening to Lauren and sorry, Laura and Helen speaking about their experiences and the sort of nonlinear pathways that they’ve come to in order to be in their roles and I’m sort of in a in a similar boat and so my background’s actually in astrophysics and astronomy, so my undergrad I went to University of Adelaide to do a Bachelor of Space Science in Astrophysics and was really excited about that. And so as a physicist or an astrophysicist, you’re sort of like a glorified computer programmer and data scientist and sort of apply that to a in a sort of astronomical stamp or lens. And so that really sort of taught me about data modeling and being able to grab grab sort of I wouldn’t say clean data because it’s data is never never clean. And so you have to then work out why is it behaving or why is it looking like that? And being able to sort of correct the data based upon sort of, especially with my background, your sort of physical constraints that you’re that the that the universe is is is de- is dealing with, right? However, with social science data, the world is a lot messier. And as I tell people, like finding planets around stars is so much easier than this type of work because people’s, people’s backgrounds, their values, their interests, who they are as an individual, all of their experiences-
Linda McIver (09:45)
Plus just what kind of day they’re having, you know, you think you might figure one person out, but no, that can change from moment to moment as well. I always say technology is easy, people are hard.
Jake (09:48)
Exactly. Ex Exactly. And then we go on. A a hundred percent.
And so there’s all that complexity and then I go off and say to someone on a likert scale, likert scale being, you know, rate something from one to seven, one being really bad, seven being really good. How is that? Or, you know, people trying to can condense down their whole life experience and and understanding into this one little one little number is is bonkers, but that’s that’s that’s what we do. And so when I got the amazing opportunity to do a a Fulbright over in over in the States and to work on machine learning, which was really, really cool, and to look at using a technique sort of similar to Netflix when, you know, you watch some great TV shows and then it keeps recommending the same things and we’re looking at, well, if we can look at the especially with astronomy, you’re looking at it’s an observable science. You know, you just get what you’re what you get, you point your telescope out out and then hope that you get to see something cool. And and if you don’t, well that’s just part and parcel parcel of it, right?
And so we were thinking, well, could you be a little bit smarter with the current data that we have and sort of forward predict what types of planets are likely to find around particular stars because we have the physical or chemical properties of it? Unfortunately it sort of was null and void and in terms of there wasn’t anything conclusive. And so I came back home and particularly during that was during COVID and so it was pretty hard to be like, well, let’s get a job in in research. I mean that’s difficult in the best of times because people are-
Linda McIver (11:37)
Yeah.
Jake (11:37)
so clever in that space and I got the opportunity to go back to Questacon, actually did my Masters with Questacon and ANU with a Master of Science Communication and which was amazing. I got to travel around to different parts of Australia doing science shows for the in the Questacon Science Circus. And they’re like, ‘hey look, we need someone to look at the evaluation data for like a month. Would you be keen to come back and use your skills?’. And I was like, ‘Yeah, no worries like would love to do so’ and then that month turned into three years.
Linda McIver (11:50)
Nice. Ha ha.
Jake (12:14)
Which was absolutely amazing and my manager, Jenny Booth, is a fantastic mentor and so the work that obviously was missing a lot from my formal education ’cause I I mean I’ve a evaluated in terms of I would run programs up in up at USQ where I did my PhD and run workshops to get kids to build planets out of Play-Doh and ask sort of evaluative questions around did you or the teachers around did they enjoy it? Sort of more like program quality control type stuff, but never in terms of the work that I do. So I learnt a lot from my in terms of formal education through my mentors and networking and I mean Stack Overflow is your best friend, especially in data science, in in the best of times.
And so I think with the great opportunities I’ve had with the mentors that I have had in the sort of evaluation space, because again, you know, I come from a quantitative background and then you’re also dealing with a lot of qualitative research as well. And I’m so glad you had the likes of Helen on to talk more more about that.
And that research is incredibly complex and difficult and so subjective and all research and all data is subjective to a degree. But it’s really difficult to have something set up in a qualitative manner that you have a sort of control for or a baseline, and again with the work that we do –
Linda McIver (14:07)
Yeah.
Jake (14:10)
Kids are all coming from a different, a different background, you have different interventions coming in and out. And so that baseline is so hard to be like, okay, well, if you come in do hour-long hour-long intervention, are you actually moving the needle? – type thing. So I think in terms of my, what was missing from my formal education, it was that more qualitative data collection and-
Linda McIver (14:24)
Yeah.
Jake (14:37)
I think with a lot of formal education you understand a lot of you get taught a lot of the theory of how things work, but not necessarily the soft skills of how it you communicate with with people, how you especially if you’re doing an interview as we are right now. How do you get people comfortable in the space that you’re in for them to actually give you an honest opinion about something w- without their guard without guardrails on or hackles up because well what are you actually gonna do with this information and how is that gonna be reflective of me? and so that sort of stuff that you just gotta learn through experience for for better or for worse, but definitely those types of soft skills –
Linda McIver (15:05)
Yeah.
Jake (15:26)
-are incredibly valuable and stuff that you sort of learn as you’re plodding along.
Linda McIver (15:32)
I think they can be taught though and I think we don’t do a good job of that. You know, every university I’ve ever looked at says we do, you know, group work so that we teach our kids, our students to collaborate and and to communicate. And ah, while they all do group work, very few of them seem to actively teach the communication and the collaboration. They’re just like, ‘Well throw them in there and hope they don’t drown… or kill each other.’ And anyone who’s done a group-
Jake (15:40)
Mm-hmm. Yeah.
Linda McIver (16:02)
-project at school or at university is probably sitting there listening to that at the moment, just kind of teeth clenched, fists tight, thinking about what a terrible experience the group work was. And and it doesn’t have to be that way. We could be teaching it and we could actually be fostering much better communication skills. They do it quite well often in primary schools. They have, you know, communication circles and a whole lot of different techniques, but we seem to-
Jake (16:10)
Yeah.
Linda McIver (16:31)
We haven’t let that filter through the rest of the education system. I think you might have pressed a button. I get a bit I get a bit a angsty about that.
Jake (16:36)
No. No.
No, y you’re you’re fine. And I think it’s also when in that space what is often missed out on is when you are in academia and someone who’s come from that space as well is that you ne- you don’t get taught how to teach people or if you want to go into high school and be a high school teacher or primary school teacher, you have to do a bachelor’s or even a master’s at s- in some places to be qualified as a as a teacher. And similarly sorry and then when you are-
Linda McIver (16:59)
Yeah.
Jake (17:13)
-tertiary educator, it’s like, well, you’re great at this research, so therefore you get thrown in, and this person who hasn’t had that formal qualifications or training, they get put into out of field and they get put in in an out of field world and they go, all right, here you go, sink or swim. And being able to sort of cut that cycle of hazing isn’t again, isn’t the right word to use, but being able to give people proper training and bringing people along for the ride. And again, like what is good what is a good group projects look like? What is good group communication or or leadership look like? What are the sort of skills that they’re looking for that you’re trying to instill into students-
Linda McIver (17:41)
Ha ha ha.
Jake (18:04)
-with those types of projects? And that’s the type of stuff that we’re also looking at with the work that we do at CSIRO, particularly with the STEM Community Partnerships Program, where we’re working with industry and educators to bring in real life problems into these into these schools that kids can actually contextualize and see see that problem in their everyday life and then be able to then go off and solve them and so solve it. And so yet again they’re also within that sort of group project environment.
Linda McIver (18:39)
I liked your use of the term hazing there, even though you kind of pulled back from it, but it’s not wrong because it’s that similar vibe of I went through this, this is how the system works. It was you know, I’m fine, you know. It’s like my mother hit me and it didn’t do me any harm. Like it’s not it’s not that different, you know. It’s like this is this is this is my experience of education, this is my experience of group work, that’s how we do it. And it’s really difficult to shift that to to steer that into-
Jake (18:53)
Mm.
Linda McIver (19:09)
-more constructive processes, I think.
Jake (19:12)
And and particularly with within education and outreach where again, well, the system people think, well the system worked fine ’cause I’m I’m i I’m I’m doing great and I’ve come out of it fine. It’s like, well, w- how was it set up for you in that experience? Did you come from again a a high what’s known as a high ICSEA school where ICSEA is this it’s-
Linda McIver (19:25)
Yeah. Yeah.
Jake (19:40)
-produced by ACARA, which is the Australian Curriculum Reporting Authority. I think that’s I’ve got I’ve got that right. And ICSEA is a value of socio education economics economic index for schools and so you can rate schools from the lowest percentile from say bottom one percent all the way up to the top one percent. And so, you know, for the schools that might be in that sort of top of top of echelon and you’ve gone through that, then you’ve had those resources around you. You’ve had those teachers that have that aren’t teaching out of field. You have role models around you. You have mentors again to look up to and go, this is what I aspire to be.
And you go down the other end, which is actually where I’m from. I’m from the s- northern suburbs of Adelaide, where the most of the people that you grow up around haven’t finished high school, let alone gone off to do university. And so what you are expected to be in society and where you where you choose to go after school is very limiting. Even in I think year 10, we were shown by the career advisor at the time that-
Linda McIver (20:38)
Yeah. Yeah.
Jake (20:58)
And at the time I was like, that’s rub, like, why on Earth would you say it like that? But I end being a little bit older now, I understand what he was trying to say, but he was like, hey, this is what you guys want to be able to do after school. But these this is the ATAR that you need. And actually, no one in this school has achieved beyond this ATAR to to go off and do this stuff. And at that point it was a very sort of-
Linda McIver (21:19)
boy.
Jake (21:28)
-massive punch in the gut and I think it would have probably been better pal- it could have been more palatable if there was a sort of better messaging around it. I think it was a bit sort of terse messaging, but I think he was just trying to say like please prep yourself because it is a hard road to get to where you are. And then even when I was in uni, I remember just passing my first year of physics first year of physics and math and the head of astronomy at the time was like, ‘hey look, you should probably think about studying something else because your grades are probably not where they need to be’. and-
Linda McIver (22:04)
Who
Jake (22:10)
-and so that’s hard because again, like your education, the the education system that you’ve grown up has only limit has limited you by that and in particular the environment you grow around or that you’re grow up around where you need to work two, three jobs in order just to get through uni at that time was was challenging, and so-
Linda McIver (22:31)
Those messages from those two authority figures in your life make me so angry. That is so wrong. That’s devastating.
Jake (22:39)
I mean, spite, spite is a great motivator. And to go ahead and say, like now I have a PhD in astrophysics and have helped discover a dozen planets amongst the cosmos and now being able to use that data literacy to now be able to empower the STEM education and outreach ecosystem is is amazing. But again, like-
Linda McIver (22:44)
Yeah.
Jake (23:09)
-If you go out and say, well, this is all working well, it’s like, well, is it really working? And more importantly, who is it actually working for? And well, sorry, most importantly, who is it not working for and what can we do about it? And actually us and I think in terms of the sort of questions around future questions that you have-
Linda McIver (23:20)
Yeah. To fix that.
Jake (23:32)
-of thinking about who your audience actually is and what do they need in order to lift themselves up and what does actually what does success actually look like to them. I think that’s yeah, having that sort of humility and compassion within data is so so important.
Linda McIver (23:44)
That’s so important.
And and the ability to look for what’s not reflected in the data. So we see we talk about students going to university who are the first in their family to go to university, but it’s a whole new level when it’s the first in their community, the first in their school, the first, you know, like that’s i it’s a level of you know, th th there are these these levels of privilege that we don’t even realise we can’t see, you know, when you’re swimming in it, you can’t see your own privilege. And so, you know, I come from ev- many everyone all of I’m the youngest of four and everyone my sisters all went to university. My my mum didn’t, but you know, a lot of my relatives had and so for me it was just an assumption that I would go to university. For you, it was a it was a a real moonshot. And-
Jake (24:21)
Yeah.
Linda McIver (24:46)
-and here you are with a PhD and and changing the ecosystem. I’d I love that for you, but also holy shit, that level of that level of of the barriers, the systemic barriers that we just don’t take into account often enough.
Jake (25:06)
Then is also to recognize that privilege and everything is a spectrum, right? And and that to say that okay, there’s a lot of barriers that I’ve had to face within within that, but I also know as a white male that’s six foot four that comes with whole different levels of privileges which I haven’t been able to come across, and yet the experiences of friends-
Linda McIver (25:14)
Mm.
Jake (25:36)
-and and family and others who aren’t that and listening to their stories, you feel you feel you feel the same way. And so I think it’s a matter of also being around a diverse range of people and experiences and particularly, again, when you’re you’re in this space trying to get as many diverse voices as you can. Because again, if you’re only sampling a certain cohort of students or a certain cohort of teachers or schools, and you go, all right, well that’s great. We can now apply this to all to all schools or whatever the case may be, and then you go, actually that’s all fallen down, fallen down in a heap because my assumptions around around what-
Linda McIver (26:23)
Yeah.
Jake (26:28)
-was going to work has all gone through. Like a perfect example of that was when I was doing a masters in the science circus, we got to go out to remote communities in the territory and the workshops that we created were for we sort of test them out in schools in inner city Canberra. And so they all worked well. And then I remember the first community I went to was in Laramba, which is about two and a half hours northwest of Alice. And it was sort of the classic bridge building exercise with spag- was spaghetti and tape and whatnot and giving the kids sort of lowdown on what’s gonna happen and one little fella puts up his hand and he goes-
Linda McIver (26:49)
Mm.
Jake (27:16)
-is ‘what’s a bridge’ and of course if you like it the penny just dropped like that and I was like that makes so much sense like there’s barely any bridges in in Alice I mean poor Todd floods once in a blue moon type thing and that was a real sort of I won’t say Earth Shattering but like-
Linda McIver (27:19)
Well. Mm. Mm.
Jake (27:45)
-that I really took the step back and thought about my own own privilege in a w- in a in a in a way that I’ve never ever thought about before. and so yeah, once I th- once we got out of that situation and go, all right, we’re gonna go onto paper planes or I think that’s what we worked on instead and the kids loved that. Then you go, okay, well, next time when we are going into a community space like that, well, what actually interests them? What are they what are they into? How can you then actually wrap around and this is what the research is showing as well with the work that we’re doing? What if there’s a local context in which pe- kids can actually see themselves in and it’s relatable, it doesn’t and-
Linda McIver (28:17)
Yeah. Yeah.
Jake (28:34)
We’ve both had this discussion around like i if it’s interests if it interests them, fantastic. And that’s and that’s a boon within itself. But if you can text if you have that local context in which they can actually see that in their own environment, then they go-
Linda McIver (28:39)
Mm.
Jake (28:49)
‘This is great’, I know how to actually appl- now apply this rather in this sort of real abstract way where kids are like, I have no idea how this relates to me. And particularly like, how does a bridge relate to someone like myself where I’ve got a little creek on this side of me and the nearest bridge from me is about four hundred clicks away. So yeah.
Linda McIver (29:05)
Yeah.
That’s that’s why I love those projects where kids are trying to solve problems in their own communities, you know, working on something that’s meaningful to them and and applying the skills that they need to to figure that problem out and and measure it and come up with solutions. And that’s you know, it’s so so scalable but also so localizable, you know, it it it fits in in any context. but even with that you still, you know, you you have to think about what the base level of skill you have in the classroom or in the in the group that you’re dealing with and, you know, what what just what kinds of conversations you’re gonna be able to have. It’s no two groups are alike. That’s one of the things I love about education. Like every time you run a course it’s a it’s a different beast with a different group of kids.
Jake (30:04)
Hundred percent. And then so when you’re on the back end of that data and trying to work out on trends or working out why this worked f in this particular way and you have these sort of real heterogeneous cohorts come in then it’s sometimes hard to work out, well, how come that landed and then I did the exact same thing yesterday and come in and that’s different again. And like yes you can look at the the broad broad distribution of results and go, all right, well that’s working, but then it also doesn’t help you in the in the then and there and be like, well why why did that not land?
Linda McIver (30:28)
Yeah.
Jake (30:51)
But that’s when we try and get some data on on that group at that point in time to go, all right, well what, what did you enjoy about it or what did you get out of it and I don’t know man, sometimes it’s just the wind the wind blowing in the wrong direction some days.
Linda McIver (30:51)
Yeah. The moon’s in the wrong phase, the planets are in the wrong alignment. Yeah, definitely. Sometimes you just can’t it’s just no there’s no knowing what happened on that day. Is that you know, you work with data all the time and you’re trying to communicate that and working with other people’s data. Is there is there one thing that that if everybody knew-
Jake (31:16)
That’s right. Mercury’s in retrograde. We’re all done for.
Linda McIver (31:38)
-this particular thing about data, your life would be easier. Is there one thing you wish everybody knew?
Jake (31:45)
That’s amazing. That’s a great question.
Data is only as good as what’s happening on the ground. So the data is a reflection w- within the work that we’re doing of the great work that the people in the workshops are delivering, delivering or the young kids at their virtual STEM visits to different industry locations or whatever whatever the case may be. But then it’s a matter of understanding well, how was that data set up in the first place? And what type of picture is that data trying trying to paint and what’s actually missing from that particularly with with data you’re-
Linda McIver (32:18)
Mm.
Jake (32:30)
-getting you might have say a third of the jigsaw puzzle there and all the pieces that are from all of that jigsaw puzzle and then you’ve got like sort of blotches and you try and work out well, okay, well I’m trying to s- sort of fill in the gaps with with the data set that I have.
And in some cases it’s a good representation of of what’s going on. And sometimes it’s it’s not as clear. And that’s because where the our instruments, whether it might be survey tools or or interviews or case studies, like unfortunately what would be nice is that I could be a little fly on on the classroom, in the classroom there and just observing what’s going on, but we don’t actually have the resources or I guess the ethics.
Linda McIver (33:20)
Yeah, that’s what I was about to say.
Jake (33:23)
You know, I think our ethics board would look at that and go, no, no, that’s not that’s not right. And then all that information and sorry, all that time to then get through all of that information to then go, all right, well, is this is this the full is this now the full picture or not? And then also, you know, who designed the data systems that you have or are are using and-
Linda McIver (33:32)
Yeah.
Jake (33:49)
-what is their worldview and what biases are ingrained within those data structures again? And there’s so much complexity around data and even to the point of what you think might be a basic count or or something like that. It’s very contextual.
So Laura said that data is like a conversation. And it’s only as good as that conversation. and so yeah, again, it’s a matter of like what types of questions are you asking? How are you answering the how are you getting people to then answer those conver conversations? Are you getting them to just give a little-
Linda McIver (34:31)
Mm.
Jake (34:34)
-box tick to say, all right, yep, that’s done. And in most cases a box tick tells you nothing about the complexity of a of an intervention or their experiences and their world view. But then at the same time you don’t have the resources or no one wants to sit there for an hour and be like, so tell me everything that you did and how you felt in every single moment in that workshop no one’s gonna no one’s gonna do that.
Linda McIver (35:04)
But also you wouldn’t get an accurate reporting from people anyway, ’cause you know, it’s already been edited and processed in their own brains, and you get back something that’s not actually what happened, which is yeah, a really interesting aspect of working with with data about people. I loved your point about data design too, because that is something that I think we really don’t understand well enough, and I keep coming back to it the idea that we actually do-
Jake (35:07)
No. It. That’s right.
Linda McIver (35:32)
-design the data that we collect. So in the d- the structure of the questions or the definition of, you know, what constitutes in or out, or what, you know, what the what the different categories are, all of those things completely change the story that the data tells. And we forget that they’re there and we forget that they have those, you know, the the built-in worldview of the people who designed that.
Jake (35:42)
Mm. E- exactly.
And then we’re trying to get people to you try and put boxes on a spectrum as well and you go, all right, well, do you fit into here and here and here? And then especially with design, you work with work with a team and you go, all right, I reckon this is exactly what you need. And then you come back and eat humble pie because you go, shivers, they’re the groups of individuals actually lie in between these boxes or the b- or in comp in completely different part of that spectrum. And again, that’s a better it’s a matter of like if-
Linda McIver (36:05)
Hm. Yeah. Mm.
Jake (36:31)
-you are working with with teams, actually understanding on the ground what their needs and wants are and what they’re actually trying to get out of this out of this information. So I think in terms of what you wish everyone knew about data is that it is complex.
Linda McIver (36:48)
I love that. In a nutshell, it’s complicated.
Jake (36:51)
Yeah.
Linda McIver (36:53)
I think that, you know, you you mentioned something about, you know, not being able to film the entire hour and I think the it sums up the complexity when you think that even if you did film it, it would be something that was off camera or, you know, that you couldn’t quite see from the angle of where the camera was pointing that it turns out you need to know. And I’ve lost count of the number of studies I’ve done, surveys that I’ve done where I’ve got all of this rich data back, incredible numbers of questions and really great lens on the thing and-
Jake (37:09)
Yeah.
Linda McIver (37:23)
-there’s one thing that I should have asked that I only find out when I’m deep in the analysis and I go, man, I really need to know this thing that I didn’t that I didn’t ask And you you don’t know until you get to the end.
Jake (37:34)
That’s that’s the joys of, that’s the great joy of what we’re doing though, is that it is dynamic and it science and data is a human endeavor. And so you do learn through what you’ve what you’ve done great and then also what you opportunities to to learn and do things do things better. And it’s always it’s always like that. And I think some systems can be quite rigid in terms of well no, it needs to be perfect a hundred percent of the time and we need to have data that’s a hundred percent reliable.
I’m gonna tell you right now, there’s no number that doesn’t accurately reflect that picture, regardless of what number you you look at, whether it’s survey results, well who they actually surveyed, by who. Even something as as fundamental as physics, you know, you’re always gonna get a a a distribution of particles, you’re always gonna get a probability of where an electron’s going to be. You don’t get a definitive answer on on anything. And so we expect these really complex systems and approaches where you’re only getting a little slither of that information where again, you know, you might have your view down-
Linda McIver (38:36)
Mm.
Jake (38:58)
-in one direction and actually all the juicy information was on on the other side and yet you have to distill down all of that to one to one number and go, that’s the whole story. Now that’s not to say that everything you read is completely fabricated and and wrong and and misleading.
Linda McIver (39:09)
Yeah. Yeah.
Jake (39:20)
There’s always nuance in what you’re looking at. And then it’s up to us as data stewards and as data scientists to give transparency on what people are seeing, how that number has been derived, what who that what that number sorry what people that number actually represents in terms of the equity lens that I’ve been talking about.
Linda McIver (39:47)
Mm.
Jake (39:48)
And what are people trying to I guess sell you with with with that number?
Linda McIver (39:54)
I love that that essential point which is that the real world is always messier than the textbook. You know, I found out recently that, that not everybody has the same number of arteries to their kidneys. And I was like, See, that’s what yep, that’s that’s the face I made. I was like, you know, I was having a scan and they were like, Now we’ll look for the accessory arteries and I like, ‘The what now?’ Like-
Jake (40:15)
Wow.
Linda McIver (40:24)
-apparently, you know, even bodies, which we we we see all of these textbook pictures of, you know, and and the the heart is on the left, not necessarily, as it turns out. There are this number of kidney arteries to the kidneys, not necessarily, as it turns out. Like it it’s it we learn science and and we learn about the real world in this sort of simple textbook fashion. Yeah, and then we hit-
Jake (40:34)
No. Very objective way.
Linda McIver (40:53)
-the real world and it’s nothing like the textbook and it’s just mind blowing. It’s one of my one of my big goals is to get education more like that, you know, that that we’re not teaching these these sanitized versions of the real world. We’re actually teaching about the real world.
Jake (40:59)
One of, yeah, well one of my favorite before I again coming over to que- to CSIRO I was working at Questcon and one of the coolest exhibits I saw was that they had painted a colour on one one part of on on a wall and then next to it there was a digital screen and some and you had to fine-tune the digital screen for the color on the digital screen to match up with the color on the wall. And because everyone has a random distribution of of of cones in their eyes, people always, even though there is definitively that same color is still there, people are gonna get slightly different answers to to-
Linda McIver (41:32)
Ooh. That’s nice. Mm, mm. Love that.
Jake (41:54)
-to that to that color. And so that given something as objective as a as a hex code that you have to then print out this color and people then interpreting that is completely subjective. And that’s again the the world that we live in in the complex in the complexity. And I think it’s a matter of being able to yeah have have education in in this way of that-
Linda McIver (41:59)
Yeah. Yeah. That’s magnificent.
Jake (42:22)
-there is a lot of complexity, but within that complexity there’s a lot of exciting and and and beautiful things that are happening and things that we can contextualize but in in in nuanced ways.
Linda McIver (42:34)
Yeah. You can’t put kids, you know, the same kid in two different test tubes and only vary one one one thing, one factor, but but it’s much more interesting this way. It’s much more interesting to look at real people and and figure out what what really does make a difference. I have completely lost my place in the questions. It’s been such a cool conversation.
Jake (42:41)
Nice. You’re okay. I think we’re at the data mistakes.
Linda McIver (43:05)
Yes we are. What are the worst data mistakes that you’ve seen?
Jake (43:13)
I mean, there’s definitely mistakes that I’ve done in the past. and I guess one of the ones I have in my mind is, we were working on a bunch of telescopes about half an hour outside of Toowomba. And we’ve been working on this discovering planets around this one particular star called AU MIC, and the first discovery well it was leading to a nature paper, which it was, and so I was like, my goodness, this is amazing. And then I was setting up the telescopes for a long night of just observing that one particular star. So you get set up at about five o’clock in the afternoon and don’t finish till about six o’clock in the morning and staring at that one little one little dot the entire night and start going through through the data and start having a look at it. And you go oh my goodness. I reckon we found an additional planet around this star. And it was a really great discovery in terms of-
Linda McIver (43:56)
Oof. wow.
Jake (44:12)
this pl- the star is such a young star, but yet it’s it’s the planet that we found isn’t where it’s supposed to be. And so it’s a real sort of game changer in terms of well in terms of how we understand how planets form and evolve. They it should be around where Pluto is, but it was closer to a star than Mercury is to the Sun.
And so we see this big dip and we go through and we go, my goodness, is that a second planet? And we go through all the data and have a look at it. And then it just looked textbook, textbook definition of what a planet around a star would look like. And then it turns out it was it was moonlight. And spent-
Linda McIver (44:57)
So Kat Ross had basically the same story, only in her story it was the sun. That there was this this this artifact that they couldn’t figure out and eventually figured out it was the sun. I I just I love these stories because the fact that you can be open about it and go, you know, sometimes you don’t see the thing that is right in front of you or really obvious or much closer than another star. Yeah.
Jake (45:22)
Literally, literally right right in front of you. And you know, Kat, Kat’s a r- really dear friend of mine and on sort of a personal note she really got me on my journey of my neurodivergent journey, which was which was wonderful. So thanks Kat on that on that front.
But yeah, you know, these things these things happen. You hear about those sort of happy accidents in terms of all these other discoveries, but then there’s also not so happy accidents or mistakes that you that you see. So that was in within myself. But I think in terms of mistakes in general, I don’t think people are in in general, people aren’t malicious. And I think I remember in twenty-one-
Linda McIver (45:59)
Yep.
Jake (46:17)
-my my mental health was a little bit how ya going and remember being in Brisbane, I was having a bit of a panic attack. I ring up one of my mates, saying, ‘hey look, can I come over and hang out with you for a bit?’ And Manda was like, ‘yeah, sure’. So come on over and we were hanging out for a bit and took me to a back porch and got this table and she just chucked down a paper and some and some pastels and she’s like just draw. Just just just draw. And so I was doing that and ah you know a few weeks go by and sort of continue drawing that thing and I look at it and go, that’s great. And then I looked at what my mate was doing and it was mine looked like a like a five year old’s painting that a a parent-
Linda McIver (46:48)
Nice.
Jake (47:14)
you know, puts on the d- discouragingly or puts on the fridge to go, aw yeah, that’s a great that’s a great painting. And I like, my god, how come mine doesn’t look like yours? And she’s like, ‘You’re a sausage mate. Like I’ve been doing this for years. I’ve been doing this for like se I’ve been painting and and and been really involved in my art for years. You’re like, this is-
Linda McIver (47:22)
Ha ha ha.
Jake (47:38)
-what the first thing you’ve done in almost a decade’, I’m yeah about that. It’s like, well why are you so hard on yourself? And I think again, if you’ve had the privilege of going through an education and in in data science and uplifting yourself and understand- and understanding of how things are, you’re obviously gonna spot things that people maybe, not even careless, or they don’t actually know and understand people don’t know what they don’t know type thing. And so I think it’s up to yourself as someone who has been privileged enough to go and get that education or to go and do that research or those learnings to then go to your peers and not go, what are you doing, you silly sausage, but actually go, wow, that’s actually really cool. Can we have a chat about that or-
Linda McIver (48:28)
Ha ha.
Jake (48:35)
-what do you think about this and what w what could make it make it better? And actually bring people along for that journey and yeah, and just have that sort of strength based approach and understand that everyone’s background is completely different and their skill levels are different and instead of sort of putting people down, lift lift others up and if you are seeing some mistakes in in the data, whether it might be sort of a lot of it might is probably human error anyway, someone’s put in a nine instead of a six or or whatnot, which is, that can happen to anyone type thing.
And so I think most people are just trying to do do their best and if you’re in a position where you can build up people’s skills and capabilities so they’re not making those those mistakes, whether it’s big or small, then that’s what you should be that you should be doing in in your role and if you have the capa- have the capacity to do so then then why why wouldn’t you do it?
Linda McIver (49:43)
I love that. That’s beautiful. That’s that’s such a such a heartfelt teacher moment. That’s, that’s you’re a a natural born teacher. That’s beautiful. So do have you ever it sort of almost flies in the face of what you just said, but have you ever seen data deliberately misused? Are there is there something where you say, that was beyond the pale?
Jake (49:56)
Thanks. Um, I mean I’m I’m I mean you do and whether i- in most cases you have and again got the literacy to understand why people are doing that and in particular it’s most likely on as a advert or in socials or or whatnot where people are trying to sell you sell you something and again watching for the signs or looking at well one who was that intended for what’s the actual motive behind this data or these numbers, is it just the fact of trying to create a diversion or or to split people apart rather than sort of bring bring people people together, ah who’s actually paying for for that study or paying for for those numbers to be, to be ah to be presented in that way and so I think a lot of-
Linda McIver (51:11)
Yeah.
Jake (51:22)
-that again is just matter of why, what are people what are people’s motivations behind those numbers and for you to recognize that. But then it also comes again with that privilege of that education and background and understanding. And in particular with AI where things online are becoming more and more real where someone like myself who’s come from this background does data for a living and has been slipped up a couple of times and goes, that looks that looks believable. I’ll share that with a mate of mine and like, nah, that’s not that’s not right at all. Like that’s AI. I’m like, how on earth is that like ev- I’ve been tripped up on that and I’m pr- pretty, pretty careful. So if again, if I’m having difficulty then ah-
Linda McIver (52:06)
Yeah.
Jake (52:12)
-in that then how is the average punter doing? And I think that’s really, really troubling. But it more I think it’s more again the understanding and nuance of what definitions are and and mean and and sort of in the evaluation space we use sort of outputs-
Linda McIver (52:18)
It is.
Jake (52:34)
-to maybe be recognized as outcomes or or as impact. And, you know, we often mistake measurement for meaning. You know, we might count participation or track outputs or the number of public or n number of papers we’ve published and we go, all right, well that’s impact. And it’s like, well or is it? I, I like that’s more of an output than than an than an outcome. Like what what have those public what have those findings actually been able to achieve from you publishing that rather than just going, well that’s a box tick, we’ve got this number out there. So I think it’s a matter of again, in sort of the data and evaluation space, how can we-
Linda McIver (52:59)
Yeah. Yeah. Yeah.
Jake (53:21)
-um….get people to know the difference and nuance within those definitions and for when we say impact this is what it means or you know even in the public space where people say, well I have a theory and you know a theory in the sort of public space means something completely different in the scientific concept and yet we use that those words interchangeably, and so it’s a matter of sort of how-
Linda McIver (53:43)
Yeah.
Jake (53:51)
-how can we how can we make sure that when we are talking about this that there’s a concrete definition behind it that yeah the other person behind me then then understands? But it’s also, again, the complexity of and the nature of of what we do. And the work that really excites me that we’ve been doing at the CSIRO is working with the University of Technology Sydney’s HTI team around looking at sort of longitudinal studies and actually understanding, well, people come from these sort of real complex backgrounds of you know different school types, different attitudes towards STEM, different socioeconomic status, different sense of belonging in STEM and whatnot. And so if you have all these factors together, what is actually the most important in terms of actually changing the dial of um people wanting to pursue a career in STEM, and so using sort of these really complex Bayesian analysis models to then go, well, this is what the in the general case, this is how people sort of fit, and you know, lo and behold, your parents’ background and economic background is sort of the fundamental part of that, and you can actually see in these beautiful graphs that-
Linda McIver (54:57)
Mm.
Jake (55:22)
-different levers need to be used for different cohorts. So for boys your confid- their confidence is first where rather than the girls their attitude pre- precedes confidence and so it’s then you know using that information to then go all right well these are the different interventions that we need to then be creating curating and being delivering to these cohorts if we actually want to move the dial and go sort of going back to that whole, you know, well a one size fits all type approach where no one size there’s no such thing as the the average person.
Linda McIver (56:06)
Yeah. Yes.
Jake (56:13)
And I think that’s more so in data where we think about the average or or the mean and we go, all right, well, if it works in that centre, we’re we’re fine. where as I think the the work that we’re doing at CSIRO is really, really exciting in terms of actually understanding well if we want to go into regional schools and we want to be working with young women, and we wanna be able to show them what careers look like and want them to pursue a STEM career, how we do it, this is actually the evidence based way to go.
Linda McIver (56:46)
That’s beautiful. I love that. So that’s what excites you about your work in general. What excites you about data?
Jake (56:56)
My goodness, it’s just the endless stories. It’s just the endless stories that data can actually tell you. And I love solving problems. I’ve always been that kid who always asks why.
And particularly around you know having a research background and now work in evaluation, there’s a fantastic evaluation, extraordinaire, E. Jane Davison, who has a fantastic quote around that ‘research can tell us what so, but only evaluation can tell us so what’. And so being able to utilize data that can actually tell you that the the the ‘what so’, but then also the ‘so what’ to then be able to give that information to to the people on the ground to actually affect change and work on changing the implementation of the work that they’re doing to then proceed for better outcomes for young Australians to pursue a career in STEM is really, really amazing and I’m really grateful to be in the position that I’m in to be able to do this as again as a living and being able to grab that information and not just see the data as just lines and rows in a in a data table. It’s a matter of like every that sort of that sonder around the data that every line there is an individual throughout their own journey. And this is just a small snapshot of them and how can we then use that information and the data that they’ve given us to actually empower them further to create the change and be the people that they they’re they’re destined to be rather than sort of capping them because of the of the inherent systematic barriers that are that are affecting them.
So that’s what I think really excites me about the work, about data, is just being able to find out someone’s story even if even if it’s just a little snapshot. And then again, how can we then use that to empower themselves or empower others that are coming through through those STEM workshops or industry visits or teacher teacher workshops to then empower them further.
Linda McIver (59:25)
I love that. It is and we talked about this before that it is such a privilege to be able to work in a space where you you know what you do matters and it’s meaningful and to to apply your skills in that way. It’s been a fantastic conversation. Thank you so much. It’s been wonderful to have you here. Doctor Jake Clark, thank you.
Jake (59:42)
Thank you, Linda. Thank you so much for having me.
Linda McIver (59:48)
You’ve been listening to Make Me Data Literate, a podcast from the Australian Data Science Education Institute. Check out the rest of our work to build critical thinking, data literacy, and AI literacy at ADSEI.org.
See you next time.
