Stream: Covid-19 Response
Topic: Metrics
Michael Donnelly (Mar 25 2020 at 12:59):
What metrics have people been using so far for COVID-19? Epic's current set is:
Current status
- Open ICU Beds
- Open Acute Beds
- Total Number of Ventilators
-
Number of Ventilators Available
Stats from the last week -
Communicable disease screening compliance
- COVID-19 tests ordered
- COVID-19 tests with a presumptive positive result, by age group
- COVID-19 tests without a final result yet
- Percentage of all visits that have a presumptive positive test result, by age group
Michael Donnelly (Mar 25 2020 at 12:59):
We've been adding an additional set each week, so I'll have more some time soon.
David Pyke (Mar 25 2020 at 13:12):
That's a great set! Is there a way to track features (negative/positive pressure, etc.)?
Michael Donnelly (Mar 25 2020 at 13:16):
Not yet. We're working on the set for phase 4 now (above is phases 1 and 2), so I'll bring any suggestions to our clinicians.
Jose Costa Teixeira (Mar 25 2020 at 13:26):
do you use Measure / MeasureReports to define and capture those ?
Jose Costa Teixeira (Mar 25 2020 at 13:27):
that would be a nice IG - a set of Measures defining "open ICU Beds"
Jose Costa Teixeira (Mar 25 2020 at 13:27):
etc
Jose Costa Teixeira (Mar 25 2020 at 13:29):
given the social impact of the current situation, I don't expect to have consistent criteria (see the work that UN WHO has done on this matter, see how different countries are counting "infected", "deaths".
Michael Donnelly (Mar 25 2020 at 13:32):
None of this is in FHIR at this point. :(
Jose Costa Teixeira (Mar 25 2020 at 13:34):
Understood. Would be a challenge for the CQI guys?
@Floyd Eisenberg do you see anyone potentially interested?
Floyd Eisenberg (Mar 25 2020 at 14:05):
Jose Costa Teixeira said:
Understood. Would be a challenge for the CQI guys?
Floyd Eisenberg do you see anyone potentially interested?
I would say many are potentially interested with respect to resources - "open ICU beds" (noting that definition of ICU bed needs to be clear to assure the right resource is actually available), ventilators, IV pumps, etc. And dynamically differentiating "in use" Vs "available. And David's comment about "negative pressure rooms" or "positive pressure rooms" may also be helpful. I don't know or any specific measures/reports currently in development or use but I see them as valuable. The "measure" would need to be focused on facilities (not patients) for which there is reference in the IGs. What resource would you consider to express the required data? Looking at Epic's set above, I might consider Questionnaire/QuestionnaireResponse with each as an observation if using FHIR for exchange - and consider location resource for hospital and hospital settings (note US Core and QI-Core may limit location for use as reference to a single patient). NOTE - all of this information has been identified as essential to share for public health and pandemic planning for a number of years but I am not aware of a pervasive systematic collection capability that can be "turned on" when needed at times like these.
Jose Costa Teixeira (Mar 25 2020 at 14:08):
I mean if there is interest in defining consistent metrics (using the Measure resource), so that when people say "we have less infected people than you" we have some idea whether these are comparable figures - tested positive, symptomatic, presumed positive...
Floyd Eisenberg (Mar 25 2020 at 14:12):
Jose Costa Teixeira said:
I mean if there is interest in defining consistent metrics (using the Measure resource), so that when people say "we have less infected people than you" we have some idea whether these are comparable figures - tested positive, symptomatic, presumed positive...
Yes - such a measure is feasible - proportion measure or continuous variable. Is there interest in pulling together a small team to express such a measure/measures?
Abbie Watson (Mar 25 2020 at 14:54):
I've seen the following population health measures in literature, which will variously be measured at the county, state, and Health Service Area levels:
- totals / percentage of population with condition X (fever, cough, suspected COVID19, pneumonia, etc)
- totals / percentage of population with COVID19
- totals / percentage of population self-isolating (home health encounters?)
- totals / percentage of population in quarantine (home health encounters?)
- totals / percentage deaths in population
- totals / percentage recovered from a positive case of COVID19
- totals / percentage estimated to have COVID19 immunity
- totals / percentage of homes or buildings under quarantine
- totals / percentage of homes 'cleared' by EMT/Firefighters/Military
Parts of the world are entering a new phase of the pandemic, where entire buildings are being lost to the virus and abandoned, and Military and other emergency personnel are 'clearing' neighborhoods and doing building-by-building searches for the deceased. They're going to resort to chalking and spray painting homes and buildings as they clear them, like they did in rescue efforts after Katrina and other hurricanes.
I imagine #DeathOnFHIR will also have some input on metrics to report.
Abbie Watson (Mar 25 2020 at 15:12):
Additional supply chain metrics might include:
- device inventory count of masks
- device inventory count of gloves
- device inventory count of face shields
- device inventory count of gowns
- medication inventory count of Albuterol canisters
- medication inventory count of Hydroxychloroquine tablets
- medication inventory count of Azithromycin IV bags
Lloyd McKenzie (Mar 25 2020 at 15:12):
Also inventory of testing supplies (swabs, reagents, etc.)
Lloyd McKenzie (Mar 25 2020 at 15:13):
Keith's blog post suggested differentiating between available physical beds and available staffed beds - as the numbers might not be the same
Jose Costa Teixeira (Mar 25 2020 at 15:14):
are we still talking about defining the metrics using measure (and using measureReport)? IMO that is great for more "official" indicators. Like what happened in China when the WHO wanted to define consistent metrics for infected people.
Josh Mandel (Mar 25 2020 at 15:15):
Active area of discussion (see https://github.com/AudaciousInquiry/saner-ig/issues/11, captured from chat here yesterday)
Lloyd McKenzie (Mar 25 2020 at 15:15):
Also, when we talk about # of ventilators, is that # of physical devices, or number of patient slots (given that those numbers are now starting to diverge)
David Pyke (Mar 25 2020 at 15:18):
I'v eadded that as an issue to consider
Michael Donnelly (Mar 25 2020 at 15:18):
Patient slots is the more useful number.
Jose Costa Teixeira (Mar 25 2020 at 15:20):
I suggest as an outcome that we actually define these. there will be lots of metrics and lots of interesting perspectives. the key is whether they are calculated the same way
Michael Donnelly (Mar 25 2020 at 15:21):
I strongly agree.
Jose Costa Teixeira (Mar 25 2020 at 15:21):
I believe the way that they count available hospital beds in Spain is NOT the same as they did a few weeks ago
John Moehrke (Mar 25 2020 at 15:22):
yup, seen some hospital layouts that clearly defined a region of the hall that is now a 'bed' that would not have been considered before
John Moehrke (Mar 25 2020 at 15:25):
both seem useful but need to be defined unambiguously. Knowing if you have equipment to equip a 'bed' is information. Knowing you have a bed a patient can go into is another.
Abbie Watson (Mar 25 2020 at 15:26):
The notion of a 'ward' is probably applicable, and patients per ward. Although I've never personally seen an EHR configured to have wards, Locations generally support it.
Jose Costa Teixeira (Mar 25 2020 at 15:26):
so do we create Measure resources for standard metrics? I guess the WHO should have some guidance? perhaps people from Emergency Response?
Michael Donnelly (Mar 25 2020 at 15:26):
I think the real thing people are most likely to need is "how many patients are you able to receive?"
David Pyke (Mar 25 2020 at 15:26):
And what service level can be supported for received patients
Jose Costa Teixeira (Mar 25 2020 at 15:27):
Well, "how many patients can we receive" - the use of Measure is to avoid that physicians will have one answer to that, and politicians will have another answer
Abbie Watson (Mar 25 2020 at 15:27):
Agreed. And keeping in mind that the Javits center in NYC is being provisioned into a field hospital, as will other venues like schools gymnasiums.
John Moehrke (Mar 25 2020 at 15:28):
provisioning a new location is common in emergency.
Josh Mandel (Mar 25 2020 at 15:28):
I agree that "how many patients are you able to receive?" is basically the critical "understand your use case and real world needs" question -- but it's unclear to me that we're going to get there with any kind of automated reporting. Which means human-in-the-loop to indicate capacity, preferences, judgement calls... which basically means data entry. Keith's blog post gets at this a bit, but I feel like I'm missing the point here. What are we automating?
Jose Costa Teixeira (Mar 25 2020 at 15:28):
Abigail Watson said:
the Javits center in NYC is being provisioned into a field hospital, as will other venues like schools gymnasiums.
(already? That is saddening...)
Michael Donnelly (Mar 25 2020 at 15:29):
I believe Epic is able to answer that, @Josh Mandel
Josh Mandel (Mar 25 2020 at 15:29):
Can you say more, re: what are we automating?
Abbie Watson (Mar 25 2020 at 15:31):
Paper is a type of communication medium, and benefits from interoperability. I would rather the paper forms they're filling out track medical doctor's notions of beds than a politician's who is focused on the economy. There might be data entry, but at least there won't be a terminology mismatch.
Jose Costa Teixeira (Mar 25 2020 at 15:31):
My point was not to look for more indicators or discuss the merits of each.
My suggestion was towards "using a FHIR resource to define how these indicators are defined and calculated"
Jose Costa Teixeira (Mar 25 2020 at 15:34):
AFAIK, the Measure resource is exactly to capture a definition - whether it is a good or bad indicator, that is another question.
examples: https://manual.jointcommission.org/releases/TJC2017B2/MIF0130.html
Lloyd McKenzie (Mar 25 2020 at 15:34):
@Jose Costa Teixeira That's only useful if those filling in the forms have a way of rendering the information from such a resource. Formal definition with a resource really only makes sense if you have systems that can consume that and automate the data extraction - and I doubt we're there in terms of penetration of these resources or bandwidth/capacity to look at them.
Lloyd McKenzie (Mar 25 2020 at 15:35):
We're going to care more about the MeasureReport instances passing around the measure values - gathered however is possible by whoever isn't needed to do more urgent stuff.
Michael Donnelly (Mar 25 2020 at 15:35):
Lloyd McKenzie said:
Formal definition with a resource really only makes sense if you have systems that can consume that and automate the data extraction - and I doubt we're there in terms of penetration of these resources or bandwidth/capacity to look at them.
Exactly right. We're focused at this point on clearly defining metrics in ways humans can understand and agree on.
Lloyd McKenzie (Mar 25 2020 at 15:35):
The definitions will likely be on a website somewhere. We can propose standardization, but I think the real definitions are going to need to be approved by folks like WHO/CDC
Josh Mandel (Mar 25 2020 at 15:35):
there might be data entry, but at least there won't be a terminology mismatch.
I wholeheartedly agree @Abigail Watson.
Jose Costa Teixeira (Mar 25 2020 at 15:36):
Agree we may not have penetration to make it operational. But I would hope that these formal definitions can be used for man-made calculations
Josh Mandel (Mar 25 2020 at 15:37):
We're going to care more about the MeasureReport instances passing around the measure values - gathered however is possible by whoever isn't needed to do more urgent stuff.
@Lloyd McKenzie if that's true, I don't see the point. Just stand up a portal with a form and let people type numbers into it. You don't need layers of FHIR stuff in the mix.
Jose Costa Teixeira (Mar 25 2020 at 15:38):
Michael Donnelly said:
Exactly right. We're focused at this point on clearly defining metrics in ways humans can understand and agree on.
All metrics are understandable by humans. and their definitions too, not sure what is missing.
Jose Costa Teixeira (Mar 25 2020 at 15:38):
Jose Costa Teixeira (Mar 25 2020 at 15:39):
this was my point. That was when the WHO changed how to measure "# of cases"
Lloyd McKenzie (Mar 25 2020 at 15:40):
HL7 stuff comes in if there are any metrics we can extract from/infer from existing systems and there's a need to transmit those elsewhere. Other than that, I agree that a phone/tablet/computer-friendly form with the data captured available in raw form for manipulation by public health folks (including rendering subsets into various dashboards) is all that's needed/appropriate.
Abbie Watson (Mar 25 2020 at 15:42):
Just stand up a portal with a form and let people type numbers into it. You don't need layers of FHIR stuff in the mix.
Eh, I'm a pessimist, and think this may be going on for 18 to 24 months. In that time, we may very well operationalize all of this. If a website and form is going to be stood up to type numbers into, why not use FHIR to figure out the fields they input, rather than rolling a one-off and then having to re-implement. Plus, ONC made FHIR R4 the law of the land a few weeks ago.
Josh Mandel (Mar 25 2020 at 15:43):
But in the web form example, what data is flowing? I think you're just saying that you want to be able to expose the collected data using FHIR, which seems fine. What's not clear to me is what the source system should be doing here, other than providing a link.
Josh Mandel (Mar 25 2020 at 15:44):
From the perspective of automation, flowing real-time data about bed usage seems like the best use case I know about. It doesn't directly tell you anything about capacity, but it seems highly feasible and automatable.
Josh Mandel (Mar 25 2020 at 15:44):
My concern is that we are going to get stuck in an uncanny valley where we standardize things that can't be automated, instead of focusing on what's practical.
Abbie Watson (Mar 25 2020 at 15:49):
This seems to me a lot like MACRA reimbursement reporting to me. The issue isn't necessarily automation, per se. It's making sure we're measuring apples-to-apples and oranges-to-oranges. It's a matter of giving people confidence in the numbers and preventing undue panic and thrashing.
Josh Mandel (Mar 25 2020 at 15:50):
Good point. But I think there are two issues:
-
You've got many vendors who independently are producing MACRA reporting tools within their own systems, so it's useful to standardize the flow. Here we are not going to see that, or at least I don't expect it in the short-term.
-
I deeply suspect that anybody who wanted to go in and audit those MACRA reports would find discrepancies across the board. It's very hard to trust these results when they are basically a gateway to reimbursement ;)
Josh Mandel (Mar 25 2020 at 15:51):
I'm not saying it makes the technology choices wrong, but you have to evaluate the whole system.
Josh Mandel (Mar 25 2020 at 15:52):
I think we would tend to do better passing along more raw data and allowing different conclusions to be evaluated and compared, rather than trying to pass along tidy reports -- this is a general kind of personal perspective, not a ground truth.
Abbie Watson (Mar 25 2020 at 15:54):
Well, it's become a pressing concern, but it doesn't necessarily need to all be solved today. Has certainly given me some things to think on though. Gonna sleep on this one.
Jose Costa Teixeira (Mar 25 2020 at 15:58):
Josh Mandel said:
I think we would tend to do better passing along more raw data and allowing different conclusions to be evaluated and compared, rather than trying to pass along tidy reports -- this is a general kind of personal perspective, not a ground truth.
i also believe more in raw data than tidy reports but here is value and huge impact of making them comparable (not necessarily tidy).
In this aspect I must say that e.g. Finance regulators are quite advanced. They dictate exactly what needs to be reported.
Jose Costa Teixeira (Mar 25 2020 at 16:03):
back to metrics: Number of ventilators available = total # ventilators - (ventilators currently off site) - (# currently broken) - (currently in use) ?
Lloyd McKenzie (Mar 25 2020 at 16:09):
I would think that total # ventilators would exclude those that are broken. And I think we should go with total # ventilator slots vs. those in use rather than counting devices.
Lloyd McKenzie (Mar 25 2020 at 16:10):
A separate metric for ventilators needing repair might be useful to allow coordination of swapping parts and getting machines back online might be appropriate
Abbie Watson (Mar 25 2020 at 16:14):
- Device inventory count of ventilator tubes
Tubes are one use per patient. Ventilators themselves get reused. Hence the 3D printing efforts.
John Moehrke (Mar 25 2020 at 16:14):
@Keith Boone have you reached out to @Derek Ritz ? He has good connections at the WHO.
Abbie Watson (Mar 25 2020 at 16:18):
I did my gradschool capstone project with Universal Hospital Supply (formerly ABC Oxygen Tent Rental Co) who are sitting on warehouses full of broken and obsolete ventilators and beds. I'm sure they're busy refurbishing and bringing older inventory back on line. I can reach out and ask for specifics on what's happening in that regard.
David Pyke (Mar 25 2020 at 16:19):
How many devices, even if broken is useful data if they're available for parts. they'd just have to be marked as such
Jose Costa Teixeira (Mar 25 2020 at 16:22):
This seems to confirm that all metrics have a perspective.
Abbie Watson (Mar 25 2020 at 16:23):
Which contributes to trust issues which leads to fear, uncertainty, doubt, and panic. See: fake news. There's a human/societal angle to this conversation that the framing around automation misses, I think.
David Pyke (Mar 25 2020 at 16:24):
That will always be a problem. Our job isn't to worry about societal functions at this point. Let's focus on tech
Jose Costa Teixeira (Mar 25 2020 at 16:26):
Another metric: start counting AdverseEvents where suspectedEntity is hydroxychloroquine...
Abbie Watson (Mar 25 2020 at 16:28):
One doctor I am in contact with brought up that hydroxychloroquine is a common medication with lupus and related autoimmune conditions, and that's a patient cohort to monitor for COVID19 as a control group to determine hydroxychloroquine efficacy. I've been working on FHIR queries in that regard.
Jose Costa Teixeira (Mar 25 2020 at 16:31):
Abigail Watson said:
There's a human/societal angle to this conversation that the framing around automation misses, I think.
True. We can automate everything and make lots of KPIs but how can you compare infected rate between countries if it is not clear / consistent what "infected" means?
Rob Hausam (Mar 25 2020 at 16:32):
@Abigail Watson Yes, it will be interesting to see what (if anything) comes from that. It's definitely something to track.
Jose Costa Teixeira (Mar 25 2020 at 16:36):
Several institutions have put considerable effort in standardizing those metrics and providing guidance. I belive there are some FHIR IGs on similar guidance.
Maybe that is not urgent to standardize and write down metric definition.
Jose Costa Teixeira (Mar 25 2020 at 16:36):
(I do believe that if we ask "how do we count the number of infected patients" to 10 experts, we will have at least 11 opinions. Which IS a problem)
Floyd Eisenberg (Mar 25 2020 at 16:49):
Jose Costa Teixeira said:
Several institutions have put considerable effort in standardizing those metrics and providing guidance. I belive there are some FHIR IGs on similar guidance.
Maybe that is not urgent to standardize and write down metric definition.
I think we can try to define things like resources - hospital beds, ICU, ventilators (with the caveat Lloyd mentioned - so may need some further delineation). But for defining "infection" requires a lot more public health input to define an infected patient - presumptive, definitive, etc. and that may chance depending on the test.
Jose Costa Teixeira (Mar 25 2020 at 16:51):
Yes, these definitions are fluid and have a lof of non-technical considerations as @Abigail Watson mentions.
Rob Hausam (Mar 25 2020 at 17:07):
I think probably all of the experts will say that to have a reasonable answer for "how do we count the number of infected patients" we need to do more testing.
Abbie Watson (Mar 25 2020 at 17:10):
Thinking through the rollout of drive-through testing sites at CVS, Wallgreen, and Walmart parking lots:
- count of testing sites (per county/state)
- count of citizens tested per day
- inventory count of test kits
- avg inventory count of tests kits / site
- avg tests performed / testing site
John Moehrke (Mar 25 2020 at 17:11):
my thinking over in the Test Results thread is that we have a fully connected nationwide personal health sharing network. (Sequoia, CommonWell, etc). Consistent test results publication should be encouraged. Once we have consistent publication, there are already defined ways to mine statistics off of that system using the IHE Multi-Patient Query (MPQ) profile. Leverage infrastructure we have, don't create new infrastructure.
Abbie Watson (Mar 25 2020 at 17:14):
I'm not sure the National Guard or police organizations or firefighters are connected with Sequoia, CommonWell, etc. Hopefully they'll work under the direction of the personal health sharing networks, but I'm not sure we can assume that. They may use their (para)military infrastructure. They're deploying the naval hospitals, after all.
Abbie Watson (Mar 25 2020 at 17:15):
Actually... you've got some great insight to that. What's your opinion on the military operations and how they'll be reporting?
Jose Costa Teixeira (Mar 25 2020 at 17:22):
Rob Hausam said:
I think probably all of the experts will say that to have a reasonable answer for "how do we count the number of infected patients" we need to do more testing.
i forgot to add: how, in the current situation, do we count/report number of infected? There are no tests, diagnosis are not being very specific nowadays. Around here it is: it seems like a URTI. Stay home, regardless of what would be the outcome of the test that we don't have anyway.
Jose Costa Teixeira (Mar 25 2020 at 17:22):
some are counting confirmed positive, some are confirming presumed positive...
Michael Donnelly (Mar 25 2020 at 17:23):
True.
Jose Costa Teixeira (Mar 25 2020 at 17:24):
this is not downplaying or putting obstacles in the work for reporting. it needs to move forward full speed. just that a clarification of the metrics is important (or will eventually be).
Jose Costa Teixeira (Mar 25 2020 at 17:25):
testing sites= HealthService?
Michael Donnelly (Mar 26 2020 at 16:43):
Jose Costa Teixeira said:
I suggest as an outcome that we actually define these. there will be lots of metrics and lots of interesting perspectives. the key is whether they are calculated the same way
Michael Donnelly (Mar 26 2020 at 16:46):
Lloyd McKenzie said:
I would think that total # ventilators would exclude those that are broken.
That's how Epic does it.
Jose Costa Teixeira (Mar 26 2020 at 16:48):
as someone mentioned, those that are broken are quickly getting unbroken, so is it relevant to mark them as "operational" vs "not operational" vs "gone forever"?
Michael Donnelly (Mar 26 2020 at 16:50):
We don't yet have a metric for ventilators that aren't operational. The total only includes working ones.
Jose Costa Teixeira (Mar 26 2020 at 16:51):
ok
Jose Costa Teixeira (May 13 2020 at 13:04):
Back to this - we are looking at a possible new resource for Inventory - the use case is when someone (or a system) tells "there are 30 of these in my shelf"
Jose Costa Teixeira (May 13 2020 at 13:10):
what we may do on the longer term does not impact the current design which is urgent.
But would be interesting to take the conclusions of this group to shape that new design.
Jose Costa Teixeira (May 13 2020 at 13:12):
can someone point me to the inventory cases that we are covering now? (absolutely nor urgent)
Last updated: Apr 12 2022 at 19:14 UTC
