A super interesting episode with Professor Darren Mansfield on sleep, data, and sleep tracking devices.
“imagine if you go and run a one hour documentary on something that’s inconclusive. Here’s all the evidence for you is the evidence against and we can’t really don’t know. That’s not great television.”
” the truth is not always as clear cut and there are many layers of grey and there are many sides to a story”
“So you can see very rapidly that sleep apnea and for that matter other sleep disorders is a very gray area. And we have to be, I think, somewhat tailored in our approach to people when we’re deciding treating any of their sleep disorders, because we don’t have enough really high quality evidence around what the longer term benefits are.”
“So there are situations where data might be that, you know, dichotomous. But probably most of the time it isn’t. Most of the time it’s a guide of variable degrees of strength and reliability. Sometimes it’s a very good guide. Sometimes it’s a rough guide. And sometimes data is completely useless.”
Transcript
Linda: Hello and welcome back to another episode of Make Me Data Literate. We’ve been planning this one for a while and managed to schedule it eventually through two busy people. And I’m really looking forward to it because sleep is a bit of an obsession of mine as it is, I think, for a lot of women my age in particular. And so I think this is going to be a really interesting conversation. So welcome Dr. Darren Mansfield.
Darren: Good morning, Linda.
Linda: Thank you so much for coming. This is great. So tell us who are you and what do you do?
Darren: Okay, so I’m a medical doctor. I’ve trained in respiratory medicine and in sleep disorders. And that combination came historically out of sleep apnea. Sleep apnea is a respiratory breathing disorder during sleep. And there’s some physicians like myself that have taken a broader interest in sleep in its entirety. And that includes sleep quality and other sleep disorders that have nothing to do with respiratory medicine like insomnia or an excessive sleepiness, narcolepsy and parasomnias and so forth. So I, 25 years ago, took over a practice at the Epworth Sleep Center by a pioneering GP that took an interest in sleep medicine as far back in the 1960s. Almost the only person doing it at the time. So I took over his practice when he retired and continued that practice ever since.
When I’m not there, I’m at Monash Health. I’m the director of Monash Lung and Sleep Department at Monash Health and obviously in that role, I oversee the sleep service here too.
Linda: That’s really interesting because as you know, I saw you many years ago having seen another sleep doctor who was like, well, your sleep apnea is not terrible. So your sleep is fine. And I was like, my sleep is not fine. I’m dying here. And you were the first doctor I saw who actually went, well, sleep apnea is not the only issue. Maybe there are other things we could look at. And I think that it’s interesting that the origins being in respiratory medicine and then actually being able to go, well, breathing isn’t the only issue here.
Darren: Yeah. And to some extent, it creates a problem. If you think about training pathways through respiratory medicine and taking firstly an interest in sleep apnea, there’s a percentage of respiratory sleep doctors that stop there. They don’t extend their training or their interest into all the other sleep disorders and a relatively small percentage of people actually do that. And that may create a bit of a workforce shortage. If you think of it like that, there’s probably not enough physicians coming through the training pathways that are getting sufficient training in all the sleep disorders or for that matter showing a career interest. Even if they do get a bit of training.
Linda: Yeah. And it really highlights that issue of the fact that we bring our own experience and training and biases to everything that we do. And so if you’re studying a particular thing or if you’re collecting data about a particular thing and you’re looking at, well, how many people have sleep problems? If you define sleep problems as sleep apnea, you get a very different result to if you’re looking at how many people have sleep problems and you define sleep problems as anyone who sleeps poorly. It’s interesting to see how the research that we do is shaped by the interest that we have and the background that we have.
Darren: I think that that’s very true. Yeah. So and this term “sleep disturbance” is sort of a bit of an encompassing term really. And it can mean somebody with a conventional respiratory breathing disorder during sleep, like snoring or sleep apnea will produce sleep disturbance. But then there’s a whole range of other non respiratory sleep disorders that can give similar symptoms or be under that umbrella term sleep disturbance. So we are incumbent to think a bit more broadly than just sleep apnea.
And I still think there’s a little bit of a gap in the workforce, the medical workforce in covering that area, despite the fact that there is in the last 20 years a much bigger awareness of sleep disorders in general in the community, even at government level. And amongst GPs, we still do not have enough experts working in these areas.
Linda: So it sounds like you’ve sort of navigated your way into a career that wasn’t part of your formal training. How did you what was missing from your formal education and how did you fill that gap. Did you just kind of fill it as you went or was there other study you did?
Darren: Yeah, very much that it’s you fill it as you go, which sounds ah…<laughs> But that was some time ago. So when I trained in respiratory medicine and there wasn’t an obligation to do specific sleep medicine training at all in the pathway. You could go and do a year of additional work, which I did, which gave you a sleep qualification. And I did that within my sleep PhD. So I was able to use my research time and double up and it gave me my sleep qualification time as well. But respiratory physicians at that stage were coming out not necessarily with any training in sleep disorders, other than having some skills around managing sleep apnoea, which I think that they were pretty good then.
Now, things have changed. So the training pathways now in respiratory medicine, the vast majority of people will do one year of sleep training within their respiratory medicine training. And it depends which centers you train at, of course, but – and some provide more exposure to the non respiratory sleep disorders than others – But it is still fair to say that they’re a little underdone in the non respiratory sleep disorders in most instances. And there isn’t necessarily a large number of people when they finish their training that decide that this is the career pathway that they wish to go to. You know, they trained as respiratory medicine doctors and and to expect them to do a, you know, a right hand turn into non respiratory sleep disorders. You can imagine only a small number take that pathway.
Linda: Yeah, that’s a real problem. Isn’t it?
Darren: Yeah, it’s a problem. And so and it’s a problem that’s more specific to Australia. North America has managed this and Canada have managed this better. And I see quite a large number of their psychiatry workforce have actually taken interest in sleep disorders and there’s training pathways for them to manage insomnia and excessive sleepiness and circadian or body clock disorders. So psychiatrists, even though they may not be dealing with mental health issues specifically, when they’re managing sleep disorders, although there can be an overlap, they are often trained and skilled in managing those problems.
Linda: So I wouldn’t necessarily help us here, though, based on the desperate shortage of psychiatrists that I’m hearing reported from friends who are needing to be diagnosed with ADHD and things like that and trying to find someone to manage their health.
Darren: Yes, pandemics, pandemics don’t do mental health much good. And then and then combine that with the upsurge in awareness around conditions like ADHD. And of course we have a shortage, don’t we?
Linda: Yes. Yeah, it’s a big problem. So you did a PhD in sleep. Did you have to wrangle data as part of that PhD?
Darren: I certainly did. It was all data. So so my PhD was on sleep. And we were looking at some of the physiological effects on the heart of having sleep apnea. So imagine trying to breathe and your throat is blocked in the middle of the night and you’re ventilating in a futile way because your throat blocks, you’re taking these respiratory efforts. Oxygen levels are falling. The autonomic nervous system will respond to that in a sort of fight-flight fashion. So that’s what we call the sympathetic nervous system will engage and put up blood pressure and heart rate. And so my PhD was looking at what are all the cardiac effects of these episodes occurring repeatedly across the night.
So we obviously had to collect a lot of data and a lot of information. And that was, you know, that was a very data driven PhD because it was mostly around physiology.
Linda: That’s super cool.
Darren: And it’s interesting that there’s, I think, quite a separation between that type of research and how we manage many of our sleep disorders, which is a much stronger focus on perceptions of sleep quality and perceptions of waking refreshed or otherwise, daytime energy levels. And they’re our goals to improve those symptoms and those perceptions. And that’s interesting. And that’s interesting because, you know, data is not our goal necessarily. Data may be used as a guide or an aid, but data is not our goal. It’s these perceptions of improved sleep quality and how people fall and function during the day. That’s what we’re trying to achieve.
Linda: That’s one of the risks, isn’t it? When you rely on data and, you know, I’m a, I’m a data nerd. I’m obsessed with data and I love to have evidence for things, but also sometimes you don’t have the data that you need. So if you’re measuring, for example, the level of sleep apnea, in my case, it was very low. And my first sleep physician said, well, you have… your sleep apnea is very low. So you’re fine. It’s like, well, no, you just don’t have data on the ways in which I’m not fine. That bit of data you have is inadequate. And so when you look at, if you look purely at the data and say, this patient is fine because their blood tests are normal and their, you know, the thing that I have measured is not a problem. Then you’re missing the overall issue of “this patient’s quality of life is really low”. And there’s still a problem there. We just haven’t measured it.
Darren: Yeah. And now sleep apnea is a good example. So let’s just expand on that for a minute. And so when, if somebody comes to me with, and we suspect sleep apnea. there may be snoring, There may be witnessed episodes whereby they’re stopping breathing. They may feel their sleep is fragmented at night because they will wake repeatedly because of choking episodes. And yet they may not actually be all that aware of the choking episodes themselves. Often they’ll wake, unrefreshed in the morning, dry on the throat, sometimes with a headache. And there are variable degrees of daytime fatigue and sleepiness, which impact on both quality of life and performance and productivity. And there’s a safety issue to consider as well for people that are driving or operating machinery. So again, when I’m taking a history around sleep apnea, I’m interested in the symptoms and interested in how they feel in the morning and interested in how they’re performing during the day.
Now we’ll go on and do a sleep test and a sleep test will give us some data. And in that context, the data will help to support or help to reinforce the likelihood that the symptoms that are in front of us are explained by sleep apnea. Okay. That might mean if they’ve got really bad sleep apnea measured by data, it supports the case pretty strongly that if we go and treat this, they’re going to feel better.
Okay. Now let’s say we do a sleep test and get some data and there’s not much sleep apnea there on the test. Now that’s interesting because we’ve just heard a story. There’s snoring, there’s choking episodes, they’re waking, you know. So are we going to then say, well, the data says there’s not much, so we’re going to just dismiss the possibility of sleep apnea. And this is where it’s interesting is because, as it turns out, that the measures of sleep apnea that we use don’t have a particularly strong correlation with the symptoms. So it turns out we have some people sometimes who have a test which says sleep apnea is mild and yet the history we’ve just taken says, gee, this sounds an awful lot like sleep apnea is causing this and if we treat it, there’s a pretty high chance that we’re going to get some improvement.
So and then we have the reverse situation where it is some people come because look, I’m here because I’ve been told to turn up. My wife, my partner has sent me and this I’m snoring and it’s bothering them and I don’t really think I’ve got a problem. I’m told and we do a test on them and go, well, as it turns out, you’ve got a lot of sleep apnea and they would say, but I feel fine. So we see this swing both ways where sometimes we find a lot of sleep apnea on the test. So the data indicates a significant problem in the absence of a lot of symptoms and then the reverse. So this is a good example of where I think data is a guide, but the data must not be considered in isolation as to what we choose to do next.
Linda: Oh man, I want that framed in every doctor’s office.
Darren: Yeah, well, we like to think that that is, you know, what most practitioners do when we and of course we know that’s not what everybody does every time. And it is hard when you have data that indicates a diagnosis. For example, and you have hard objective evidence. That’s what data offers us. It’s hard evidence. It’s evidence that anybody who looks at this case subsequent to you will look at the data because they’re not necessarily… We might have somebody who’s not very symptomatic with their sleep apnea, but the data says it’s there. And they may say I don’t want treatment. Okay. And because I don’t have very many symptoms and we’ll have to have discussion around how we manage that. But some may not go on to treatment based on few symptoms.
Now, what happens if they go and crash their car? Who’s going to look at this case and say should this person have been treated or not? Well, they’re going to look at the data. They’re going to look at the tests that showed the condition. And you can see now what practitioners may have to do. They say, well, I’ve got a history that I’ve taken that’s very much between me and this person. And nobody else has been in a position to witness this discussion and evaluate the symptoms. But what’s left behind is a test result. And there’s significant sleep apnea there. And there is an opportunity for others to come after you and make their own judgment as to what you should have done, often in retrospect. Because something’s happened. You can see people start to actually practice a little defensively so that the data, interestingly, has a little bit more influence than it might otherwise have had. And people can be swayed by data to just say, well, the condition’s there, we probably need to treat it regardless of your symptoms. And so there can be a bit of that that comes into people’s practice and it comes into my practice even thoughI’m aware of it, but I’m also mindful of, OK, well, we still need to consider this, even though this person is not giving me a lot of symptoms, but there’s at least a reasonable amount of sleep apnea there on the tests. We may have to still consider treating it for ensuring that we’ve covered the basics.
But there is another consideration too, is that there may be other longer term health consequences of untreated sleep apnea that we have to consider that may not be symptom based.
Linda: Yeah, I was going to ask about that. Is there any correlation between damage to, for example, the cardiovascular system and symptoms?
Darren: Yeah, yeah. And certainly there is this very strong relationships between if you have untreated sleep apnea then you are more likely to have problems with blood pressure, diabetes and cardiovascular disease and stroke. What is interesting is that we have not yet shown that if you treat the sleep apnea, does that cardiovascular risk reduce? And we’ve not been able to show that in studies to date, which is interesting. So when we decide, look, we must treat your condition, we have to be mindful that, you know, we’re certainly hoping to treat symptoms if they’re there, but we may not necessarily be able to prove to them or anybody else that there is going to be a reduction in cardiovascular risk that comes from this.
Linda: So correlation isn’t causation necessarily?
Darren: Well, it’s interesting and it might very well be that the studies that have looked at treating sleep apnea are just not good enough or not big enough. And maybe there is a signal, but we’ve just not been able to demonstrate it. So we have to be very mindful that just because the study hasn’t shown it doesn’t mean it’s not there. It just may have not found the signal. So we have to factor all of these things in when we’re offering treatment to somebody. We have to say, well, look, you know, it is heavily swayed towards assessing the symptoms. The data can to a degree help us in being sure that we’re on the right track. But we still may entertain treatment if the data says the sleep apnea is mild if the symptoms are significant. And we still might consider treating it if it’s the other way around thinking, well, there may well be some longer term health benefits, even if they’re not exactly proven. So you can see very rapidly that sleep apnea and for that matter other sleep disorders is a very gray area. And we have to be, I think, somewhat tailored in our approach to people when we’re deciding treating any of their sleep disorders, because we don’t have enough really high quality evidence around what the longer term benefits are.
But obviously we do, you know, we are across, or more across, what the symptom improvements that we might be able to obtain. It’s not just tiredness and it’s not just fatigue. It’s not quality of life and productivity. All of those things wear down on people’s mental health. So I think people are happier and moods are improved and anxiety levels are diminished and resilience is built if you have energy. So a mental health side to things, even in people that don’t have mental health disorders, but they can kind of still have some mental health symptoms.
Linda: Yeah. That’s so interesting. I had no idea it was such a gray area. I mean, I know where biology and physiology are concerned, it often is because they’re incredibly complex systems and we don’t fully understand them yet. But it’s interesting because it’s not reported that way. If you look up a page on sleep apnea from one of the big institutions that we think is nice and reliable, you’ll get something that’s pretty black and white. So it’s really interesting to hear how complicated it is underneath.
Darren: Yeah, and it’s complicated and doing a sleep test will give you data. It will say, well, you think the test says, well, it’s been picked up. There is sleep apnea there. Well, there’s not, but then it’s not really a yes or no. It’s all a severity issue. And to make it more complicated, there is variability from one night to the next. So if you do the same test three nights in a row, you get slightly different results each time. And we may see as we talk a little bit more shortly about some of the opportunities to use data in simplified sleep monitoring. Because as many of your listeners will be aware who’ve had a sleep test. It’s a cumbersome way to diagnose the condition.
Linda: Yes.
Darren: So it takes time as it takes a night out of your life, many people come into a facility for it, or at least have to come into a center to be wired up to then go home. And it’s a single night. It’s a single snapshot. So it would be better if we could capture two, three or four nights in a row with the data and would the results be more helpful.
Linda: Yeah, I did notice when I had my sleep test that they trust you up like a chicken and then say now sleep normally. I’m not sure that’s going to happen really.
Darren: I think that’s everybody’s apprehension about a sleep test is, you know, the tools we’re using to measure sleep interfere with the very thing we’re measuring. And that’s that. And we get by with that for conditions like sleep apnea. But if we’re just wanting to assess somebody’s sleep quality and that’s deep sleep, like how much REM sleep or dreaming sleep like it, you know, too many wires on will will genuinely affect that.
Linda: Well, that leads beautifully into smartwatches and fitness trackers. And I guess is that how you got into it in the first place that that sort of that progression from I wish we could look at this, you know, on more on a longer term basis and when people are sleeping normally.
Darren: I, I, in part, yes, that was certainly there. And when I started in this area, there weren’t really a lot of smartwatches around and fitness trackers. They’ve sort of evolved with time. And again, there’s there’s a lot of good things to be said for people getting interested in their sleep and their health. And so I think we always start there, you know, just, and if we have a sleep app or a fitness app, is that going to help somebody to take more interest and be more engaged? And in particular, we’re talking sleep here. So if we’re going to use an app to measure their sleep, hopefully that increases people’s level of interest. And that’s, you know, that can’t be a bad thing.
There is a flip side to this. People can get a little bit too caught up in their data and too worried that their data says there’s an issue with their sleep. And that can create an anxiety. And, and we caution people around this to a degree.
Just remember all of the data, particularly in the area of sleep health and sleep medicine. I mean, almost all instances, the data is an aid or it’s a guide. Isn’t the be all, the end all. As I said before, you know, I’m far more interested in people’s perception of their sleep quality. And when somebody says I go to bed, I fall asleep reasonably promptly. I might wake up briefly once or twice across the night and I’m up in the morning and feeling reasonably good. Then I am mostly reassured we don’t have a big problem here.
Now, if they get a sleep app that says there’s all sorts of things going on in their sleep, but it’s not reflected. Let’s say we’re not talking to sleep apnea anymore, but just a general sleep disturbance. I’m going to keep coming back to the symptoms. Now, it can be more helpful the other way around. So if somebody is saying they’re very unhappy with their sleep quality. And you have some data that shows there are some concerns with the sleep in various ways. Then that is perhaps a little bit more helpful that you’ve got a test that might be pointing to what the problems might actually be. Okay. But when we have a test that says it’s not quite right, but there are no symptoms and some perfectly fine. I may not be taking a whole lot of notice in the test.
Linda: How accurate are they if you know that my watch has told me that I’m sleeping deeply when I’m up and making breakfast.
Darren: They do several things. They take measurements of movement, measurements of heart rate, and heart rate variability. And now all of the different sleep apps have got their own secret proprietary algorithm as to how they then use that to interpret sleep. So we can’t necessarily say we all know how they’ve interpreted this. We can only say, well, how does it stack up against a proper sleep test? Now, most of them are not bad at picking sleep onset – when people get off to sleep, they are generally pretty motionless and it will pick that and it will often interpret that as sleep. Clearly, if you wake up and start moving around and the heart rate goes up, etc., then it will usually pick that they’ve woken up. And they are not bad at this. Clearly, you can lie awake very still and you will fool it. But by and large, they’re reasonably reliable.
They’re generally not very good at picking the different stages of sleep. So if we’re trying to determine light sleep, from deep sleep, from dreaming sleep, and most of the devices will display all of those things now. But we would say in many instances they’re still not very good at that yet. Even with the best technology that we have in a sleep laboratory, we have some instances where we’re all having a discussion as to whether this is actually REM sleep or not. Picking some of these sleep stages like REM sleep can be quite hard in some cases, not all. Sometimes it’s quite straightforward. but we do have instances where it gets tricky. And that’s with all the monitoring that we have which includes EEGs or brain wave monitoring, direct monitoring of the brain signals. Sometimes we still have some challenges.
So these apps are not measuring any of that. It’s really usually heart rate. Now the heart rate is not just the rate itself but the way the heart rate can vary from one moment to the next is also something that they include in their measures. And that’s something to do with that autonomic nervous system. Which does do different things in wake versus sleep. So they are using that as well. That’s why there is at least some reliability with these devices. But again, I’m a little dismissive of the different stages of sleep and I’m more interested in these devices giving us a bit of a guide as to whether they are actually in bed and asleep or at the very least lying very still but awake.
And we do find that these devices can really help us because again, although I think people’s perceptions of their sleep is all important, people aren’t always that reliable about reporting their routines. So people who may say they go to bed at such and such a time and up but whatever time the next day and if we were to put one of these devices and monitor it, we actually discover well guess what you’re going to bed at 1 in the morning, 3 nights a week and that’s not what you told me.
So we can make some discoveries and get some insights into people’s routines that the history did not provide us. So I think that there is a lot to be said for helping to reinforce the history around routines.
Linda: I have pretty strong feelings about the way the devices very confidently report sleep stage even though they are not actually accurately measuring sleep stage. Do you have any thoughts about that?
Darren: Yeah and that gets back to sort of it may decide that you’re in deep sleep when you may not be. It may decide that you’re dreaming sleep or REM sleep or rapid eye movement sleep as we call it and you may not be. So the stages we think, and we are speaking generally and some of them might be getting a bit better than others, but in general we don’t think that those sleep stages have been perfected with these devices yet.
Linda: Yeah, that’s a problem. It’s so interesting. I learnt a lot and I’ve read a lot on this so I wasn’t expecting to necessarily learn a lot but just have an interesting conversation.
Darren: It’s worth pointing out that we have wearables like a watch. Increasingly we have things called nearables and a nearable is a monitoring device that may not be on you but it’s near to you and we’re seeing a lot more under the mattress devices now. It’s a device that works a little bit like a mat under the mattress that can monitor your body position in bed, your movement in bed. It can usually pick up your heart rate even from there and use all of that data to interpret whether you are awake or asleep and even it will pick up your breathing patterns and make some assessment as to whether there might be some sleep apnea there.
That’s a new opportunity to look at something like a nearable as opposed to a wearable and they can stay under the mattress and they can measure night after night after night and as many nights as you wish to measure.
Linda: Without 16 electrodes attached to you.
Darren: yeah. That’s a new opportunity and we and others are studying where that may fit in. Certainly we think if you have a diagnosis of sleep apnea and we would like to consider treating it or for that matter maybe we may choose not to treat it. You can use these devices in the home setting as a bit of a guide as to whether the sleep apnea is, on multiple nights, sufficiently well controlled or for that matter where it gets worse with time if people put on weight or whatever they may be. Certainly these things are there’s roles for these things when they but again they have to be very carefully selected because we’ve already said data is a guide and a guide that’s used in isolation may turn to prove to be a bit misleading.
Linda: Yes, in any context in education we could talk about naplan being misused as well.
Darren: Yes, yes, yes, not an expert on the naplan but it’s the same example of when you take a test result of any sort in isolation what information is it giving us and it might be a good guide. In some cases it can be an excellent guide, in some cases it’s a rough guide and you need to know the tool that you’re using to get data as to where it sits on that spectrum.
Linda: Yes, absolutely. That’s a nice segue into my next question. I got a little distracted from the questions there. Is there anything that you wish everyone knew about data is there one thing where you think if everyone understood this, you know, things would, things would be better.
Darren: Yeah, I think it might be emphasizing the point, you know, I think understanding your tool is the most important thing about data. You know, it’s, it’s can be, it can be very reinforcing of your premise that you have, I believe I have this diagnosis and the data supports it and that has a reinforcing role. When it works in opposition, then it casts doubt on your original diagnosis. So I suspect this is the diagnosis but the data says something different. Doesn’t mean the diagnosis doesn’t exist. It just means that there is, you know, there is a higher degree of doubt and you may have to go to additional lengths to either prove or disprove.
So you can see that it helps to reinforce or possibly challenges your original thinking and that’s how data needs to be used in most instances. And the extent to which it reinforces or challenges is, is, is knowing the tool that that you’re using quite well. So can data outright prove or absolutely disprove. And there are some situations where data does exactly that where the data itself is everything and it is all by itself the proof or the disproof. And it’s quite categorical. So there are situations where data might be that, you know, dichotomous. But probably most of the time it isn’t. Most of the time it’s a guide of variable degrees of strength and reliability. Sometimes it’s a very good guide. Sometimes it’s a rough guide. And sometimes data is completely useless.
That should be framed as well. I’ve noticed that all the way through you’ve been saying the data might support. It doesn’t prove you know, you keep using the word support.
Darren Yeah and often that’s the case. Now, of course, there are situations where just as I said, you know, where, you know, the level of support is so strong that it’s almost independently proof. So, you know, and there are examples of that. You know, you might say, well, what data could absolutely prove or disprove? Well, you know, we were able to prove that you entered this building because your swipe card was used. And we could absolutely say that you were there. Even though you claim you weren’t, you know, there’s an example of data that would provide a high level of certainty. Well, unless somebody else used my swipe card. So so you can see, yeah, well, you know, so there are examples. that’s a non medical example. But you can see that there are if you think about it there are many situations where collecting data can provide pretty high levels of proof for certain things. And it’s knowing the tool. And this is difficult for people in the general public because there are … obviously people advertise monitoring devices and data measuring devices. You might choose to buy one, but you know, not necessarily going to be the expert in knowing how good this tool is.
Linda: Yeah.
Darren: And that’s hard for people. And I still think that’s where, you know, seeking out some expertise as to best how I would interpret the results that are in front of me. And when it comes back to sleep, in particular, if you’re sleeping well, and if you think you’re sleeping well and you wake up feeling reasonably good and you’re reasonably energetic during the day. But your data says something different. Don’t be alarmed. You may wish to seek out an expert and get some verification of how you may sort of reconcile these two different things. And that’s a good thing to do. But you would certainly hope that the expert that you speak to doesn’t hang their hat on the data either and then create unnecessary worry.
Linda: Yes.
Darren: And we see it in healthcare all the time. You know, we do breast cancer screening and breast cancer screening can obviously detect an early breast cancer. And that’s what it’s all about. It’s a form of data. It’s information. It can also produce the shadows, which are just a bit of scar tissue, a bit of fibrous tissue that might look an awful lot like an early breast cancer and put somebody through a whole lot of unnecessary anxiety. So we know that these tools, you know, as helpful as they may be, you know, they have their issues, they have their false positives, create unnecessary worry. And they have false negatives. In other words, there could still be a tiny cancer there and it misses it.
So, you know, we use data everywhere and understanding its reliability and in what way it’s going to help you be more or less certain about a certain issue is understanding the tool a lot where you can or seeking out others that can help you.
Linda: I like that. The first question I always get students to ask when they look at a data set is what’s wrong with this data. And I guess you take it one step back and say what’s wrong with the tool that collected the data.
Darren: Well, it can be both. Yeah. So the tool may actually collect the data or the information inaccurately at that level. And you need to know the accuracy of the tool that you’re using for collecting data. Even if it’s accurate, then there’s the interpretation of the data. Well, what does this result actually mean? Yeah. When it says it was collected accurately, but it says that my sleep’s discontinuous. But I feel perfectly fine. And I’m not aware of discontinuous sleep. What might that mean for me?
And we probably would mostly say as a sleep practitioner and say I’m far more interested in your perception of your sleep and your symptoms and your performance and quality of life. More so than a measure of discontinuous sleep.
Linda: What are the worst data mistakes that you’ve seen?
Darren: Ah, this is a common one, I think unnecessary treatment. It is a common one. Now, it may not have disastrous results, but it’s really common. And let’s go back to sleep apnea again, where somebody has a sleep test and it shows a little bit of sleep apnea. It’s mild and it’s there. But somebody’s really not terribly troubled by it. And people have sleep tests for different reasons. And sometimes people don’t always have a lot of symptoms and not necessarily troubled very much. But nonetheless, for one reason or another, they still had a sleep test and then saw some sleep apnea then. So now to then take that person and say, we are going to treat this because we saw it there on the test result is actually quite a common practice. And people get put on the CPAP machines and they hate them, they don’t feel any better and it costs them money and it’s a big battle. And what’s more is they then go and tell their 10 best friends what a hideous treatment that it is and which affects everybody else’s perception.
And that’s where the damage might be because you may eventually, this person stops their treatment and I might have argued and they never needed it in the first place. What damage has been done apart from time, money, inconvenience and discomfort. It’s the 10 other people they told. What a shocking treatment it is, because they were wrongly selected. So we can build perceptions in the community that extend beyond the individual if we’re treating them wrongly. So I think that would be something that I see very frequently. I would see that every week. Somebody comes to me for a further opinion. So I’ve been told I must have my sleep apnea treated but I hate this CPAP machine. I don’t feel any different, and for that matter I don’t feel like I had a problem in the first place but here I am.
And you look at it and go, well, it’s actually pretty mild. And it’s more of an example of people treating it because it was there rather than treating it because they had a clear goal in mind as to what they were hoping to get out of treatment. If you tried, if you found a number on a page that says you have a severity of sleep apnea and we are trying to make that number look better. And that’s our goal. We’ve kind of, we’ve sort of missed something somewhere in our approach to medical practice. And so this is where data takes over. And I still think that in medical practice across the board we need to be very goal directed in our approach. And it’s usually not making a number look better on a page. If we think a bit more like that we’ll see a little bit. So over treatment I think is certainly in my practice that’s where I see unnecessary treatment or excessive treatment driven by data used too much in isolation.
Linda: I love that. That’s such a great example. Have you ever seen data deliberately misused?
Darren: Yeah, yeah. And there are many, many examples of people using data where they are trying to peddle an argument. And we see this a lot where this data, this is where there is a departure from true science. Okay. So let’s just take a step back for a moment and think what science aims to do. It is to develop evidence to support or build a hypothesis and the more evidence that you collect over time, the more it influences the hypothesis until one day it’s either accepted as truth or proven. Or accepted as very likely or accepted as the very best prevailing hypothesis. But subsequent data can come along which puts doubt on that. And a good scientist would then instantly reevaluate their hypothesis. They would go back and say, “We have new information. We now have doubt and we have to rethink.”
And a good scientist will readily rethink their position based on new information.
Many people in the community are not scientists and many people here in the news go about this the other way. They have an idea in their head for whatever reason. It could be an ideological position. It could be for personal gain. It could be for whatever belief system people have. And they will in those situations stick to their belief system almost at all costs. And they will use data that supports their position or use data misinterpreted to support their position. Or they will downplay and ignore data that doesn’t support their position. Or find some reason why that data is wrong or incorrect or whatever it might be. So it’s not a scientific position where data is to actually form your view. In this instance, data is used selectively to reinforce your view. And we see that all the time. And guess what we even see it in scientists?
Linda: yes we do!
Darren: You would like to think the scientific process is impervious to people’s selected belief systems. Even scientists are human sometimes.
Linda: Sometimes? Only sometimes?
Darren: So data is misused all the time. For whatever perspective or viewpoint somebody wishes to stick to for some other reasons. And they don’t like, necessarily, inconvenient data coming to mess with that view. But with climate change when the inconvenient truth term was thrown around is because that was a time when new evidence and new data was actually forcing people to have to rethink their views on things when perhaps they didn’t want to. And that was very inconvenient.
Linda: Yeah. And uncomfortable.
Darren: Yeah. All the time. And we see it in media. We see it in the general public. All the time. We see it in politicians. We see it… everywhere we look. We see data being misused.
Linda: Yeah. What’s the first question you ask when you see graphs in the media given you have some understanding of data? do you look at them differently?
Darren: Yeah. Yeah. Yeah. So it’s it’s and you know the media. The media will be there often the media will be you know will have a central story around something that it wishes to run. So you know it will you know it will create a narrative around that if you’re going to run a one hour documentary on something it’s often starting by a preconception. You know I’ve got an idea. We’re going to run this story and the stronger or compelling that story is. The better it’s received and the better the audience can shape their views and identify with the subject.
If you imagine if you go and run a one hour documentary on something that’s inconclusive. Here’s all the evidence for you is the evidence against and we can’t really don’t know. That’s not great television.
So every graph I see on the television. I ask myself where did the data come from? How was it collected? What was the sample size – a graph that shows you know people are doing X, but but how many people were collected that showed this? you know it might be a it might be an opinion poll it might be a you know election poll. It’s swinging to the left or swinging to the right but how many people were sampled and where were they sampled from and how representative was that sample of the whole of the Australian community.
And so those questions sort of run through my mind all the time, and it’s almost like a reflex from me because I’m kind of trained to think that way.
Linda: Yeah,
Darren: nothing is accepted at face value particularly in the media for the reasons that I’ve indicated because media has trouble getting viewers for a wishy washy story that’s got two sides to it. That’s not a very great subject. So, so we will see a graph that shows compelling evidence of X, but we kind of know that there could be out there graphs that show the opposite that were never shown. we sort of wonder how representative that particular graph or set of data was of the broader issue and was it, you know, carefully selected and therefore not representative was it poorly collected, all this sort of stuff, so all of this comes through my mind when I look at information, particularly in the public domain because, you know, because that is a context in which journalists, for example, are motivated to tell a story, a clear message.
Linda: Yeah.
Darren: Not necessarily because the truth is not always as clear cut and there are many layers of gray and there are many sides to a story and we see it with social issues we see with conflicts.
Linda: Grey and complicated is a much harder story to tell.
Darren: Yeah. Conflict is much easier if you can just take a side isn’t it. Yeah. You know, like it’s, you know, the goodies and the baddies or whatever and let’s take a side.
Linda: What excites you about data?
Darren: I actually think it’s the exercise. It’s, you know, and it’s, it’s really, it’s much of what we’ve talked about already is, is data is, is, it’s a good intellectual exercise, isn’t it? And to say, look at a piece of data and go, okay, let’s just for a moment think, okay, how is it collected? How reliably or accurately was it collected? What signal is it offering? And how does that impact on, you know, other prevailing data? And how does that help form my view? And that’s a bit of an exercise, which can actually be quite, quite a lot of, you know, an enjoyable challenge to go through. I like to use data in that way and to, you know, to test the rigor of the data. And, you know, in my line of work, because you are, you’re very much taught to think that way, you can do it rather quickly. And you can do it almost reflexively, you know, it’s almost a reflex to, to do that. But it’s still an enjoyable process. And it’s, you know, we often get into our meetings in our department. Data is presented and the discussion time is all around that. You know, how good was that data? How, how well was it collected? How reliably? how does it fit with data that shows something very different or the opposite? Is it better? You know, does it supercede all the previous data because it’s better? And, and that’s a whole sort of exercise that we, we all find ourselves going through, you know, every week.
Linda: Thank you so much. It’s been such a great conversation. So interesting to see how the conversations travel with the same questions, but in totally different directions. Thanks so much for your time.
Darre: Pleasure indeed. Thank you.
