Stream: genomics
Topic: Sanger confirmation/testing
Jamie Jones (Jun 17 2019 at 16:10):
Re: Inclusion of Sanger confirmation information (GF#19829), we'd like to open up discussion on whether confirmations of NGS results should be separate Observations (likely) and if so, are they additional Variant profiles (duplicating many components), or something simpler?
Kevin Power (Jun 17 2019 at 16:22):
Two things:
1. I lean towards a simple 'Confirmation Status' profile that points at what the confirmation testing was about (perhaps via Observation.focus?), we would need a good Observation.code, Observation.method could indicate 'Sanger', and the ConfirmationStatus.value contains the status (perhaps there is a good value set out there for this?).
2. Perhaps this is a good question for O&O? Seems other types of Observations might have a similar requirement and maybe there is already a pattern?
Jamie Jones (Jun 17 2019 at 20:04):
I wonder if this could be a use of the RegionStudied profile... saying the specific variant regions were completely studied and verified (method as Sanger). Is that asking too much? Thoughts, @Bob Dolin ?
Bob Dolin (Jun 17 2019 at 21:10):
I was kinda thinking of another option, where we primarily represent each test, and then have optional associations that indicate concordance. So for instance, imagine a patient that had a variant detected and reported based solely on Sanger sequencing.
@James Jones I like where you're going with RegionStudied :-). A very interesting recent article (https://jmd.amjpathol.org/article/S1525-1578(18)30291-5/pdf) suggests that we can algorithmically tier variants into high confidence true positives (may not need confirmation); candidate true positives (likely should have confirmation); and high confidence false positives (wouldn't be reported). Just thinking out loud here, but if we had a 'high confidence subregion' and a 'middle confidence subregion' component, I'd be able to represent high and middle confidence subregions in the RegionStudied profile, and then use that information by a decision support appplication to alter the strength of the recommendation.
Jamie Jones (Jun 17 2019 at 21:12):
Other groups have certainly asked for more structure in modeling individual tests inside a larger report...
Kevin Power (Jun 17 2019 at 21:49):
I suppose we could use Region Studied, but it seems pretty heavy handed to me. Plus, I think in most cases when they do confirmation, the labs are basically saying "this variant was confirmed by Sanger", not necessarily "I confirmed everything in this region"? But maybe I am off base?
Bob Dolin (Jun 18 2019 at 20:52):
Hi @Kevin Power I agree that using RegionStudied is probably too much to do prior to publication. And I understand that labs often report 'confirmed by Sanger', so I don't mind adding some type of flag if it's of value to someone, above and beyond what's in the narrative report. But I also feel like introducing the flag gives us yet another way of representing analytic validity. How would you differentiate, say, one variant that was 'mid-level confidence' and therefore subjected to sanger sequencing vs. another variant that was 'high confidence' and therefore not confirmed with Sanger?
Kevin Power (Jun 18 2019 at 21:15):
Well, your comments make me wonder - does this solve a computational problem now? Meaning, do we think the 'Sanger Confirmation' needs to be a computable value for the first release, or is simply stating it in the report sufficient? Tagging @Mullai Murugan since she logged the tracker.
Jamie Jones (Jun 24 2019 at 14:15):
The more I think about it, the more uncomfortable I am with us not giving Variant a uniform way to convey confidence. I almost feel it should be required to enter into an EMR...
Jamie Jones (Jun 24 2019 at 14:16):
Unless there is already one suggested somewhere I'm not seeing
Jamie Jones (Jun 24 2019 at 14:24):
Maybe this is something for a future "daughter IG" and not the universal one, but I think we need to set aside a component apart from method for comments like this. Thoughts??
Jamie Jones (Jun 24 2019 at 14:25):
Observation.note could also function here if we provide guidance saying so but I envision this being coded in the near future.
Kevin Power (Jun 24 2019 at 14:35):
Confidence is something we need to spend some time on, but the biggest problem is what I haven't seen - I don't know how or if this is delivered today, and what the senders and receivers will expect. This is likely because the labs only deliver things they are 'confident' in today. But in tomorrow's world, the lines will get blurred. We just need a solid proposal to start from, and if it causes us to rework / break some things from our first release, that is OK.
Kevin Power (Jun 24 2019 at 14:36):
And if we think the Sanger Confirmation is all tied up in this confidence, then perhaps we should defer it?
Jamie Jones (Jun 24 2019 at 20:06):
On call today we discussed a Variant component for "confidence" with example values of (Confirmed/High/Medium/Low/Unknown), considering results in the paper https://jmd.amjpathol.org/article/S1525-1578(18)30291-5/fulltext. High indicating high confidence true positives (don't need confirmation), Medium indicating candidate true positives (in need of confirmation), and Low indicating high confidence false positives (included for completeness).
Not 100% sold on H/M/L as labs must specify algorithm, but would love to have the group consider if this concept is inside the 80% or the 20%
Kevin Power (Jun 24 2019 at 23:24):
Seems like a pretty good middle ground to me. I am not sure about the value set though.
I suppose we could consider a Confidence Method component to go along with it in case they wanted to indicate how they determined it.
Mullai Murugan (Jun 29 2019 at 13:53):
To provide more context - for clinical reports, every variant reported includes a confirmation status, it does not have to be Sanger. For example, we validate CNVs using MLPA and NGS with Sanger. Would @Kevin Power's recommendations in the second thread be a good start? As @Bob Dolin had suggested, the notion of confidence should also be helpful though I am not sure of the value set. Another point to note, if it is a research report, we don't need confirmation. Would the DescribedVariant be an appropriate Observation to include this info?
Kevin Power (Jun 30 2019 at 15:08):
@Mullai Murugan thanks for the input. Which recommendation of mine are you referring to? Having a separate profile for the confirmation? And do you not want to combine the confirmation and confidence concepts?
Regarding your comments about a research report not having a confirmation, any approach we take should be optional.
If we include the confirmation as a component, we would certainly want to include it in the Variant profile.
Mullai Murugan (Jul 01 2019 at 00:18):
@Kevin Power Here is your recommendation 1. I lean towards a simple 'Confirmation Status' profile that points at what the confirmation testing was about (perhaps via Observation.focus?), we would need a good Observation.code, Observation.method could indicate 'Sanger', and the ConfirmationStatus.value contains the status (perhaps there is a good value set out there for this?).
Perhaps we could also additionally include the confidence element?
Bob Dolin (Jul 01 2019 at 14:28):
One reason I was thinking the confidence element would be helpful is, for example, a case where two variants are identified. One is deemed high confidence and therefore no confirmatory testing is performed, whereas the other is intermediate confidence and therefore has confirmatory testing. Without the confidence element, all we'd see is that one was confirmed and one wasn't. As for value set for confidence, I'm suggesting we start with something very simple, based on https://jmd.amjpathol.org/article/S1525-1578(18)30291-5/fulltext: 'high confidence (definition: likely TP); low confidence (definition: likely FP); intermediate confidence (definition: probably TP but may need confirmation). This would be just an example value set, to let us get more experience with it.
Jamie Jones (Jul 01 2019 at 14:54):
I agree the confirmation and confidence concepts are different and think both should be included, particularly since our IG currently gives no guidance in how to represent any of that info elsewhere and it could solve computable needs right now
Jamie Jones (Jul 01 2019 at 14:55):
I was originally thinking we could just add "confirmed" as another answer in the confidence value set...
Kevin Power (Jul 01 2019 at 14:57):
I can't make the FHIR subgroup today, but we just need a proposal to review. I will admit to being a bit nervous about introducing a new(ish) concept this late in the game, but if we can identify a straightforward proposal, and soon, I don't see why we couldn't vote on it.
Jamie Jones (Jul 01 2019 at 21:27):
Proposal tweaked on call today:
To keep it as simple as possible to cover the use cases of specifically stating a variant is confirmed and/or in a high confidence area deemed not needing confirmation, we could add a single "Confirmation" component to variant. Example Answer list would contain "Sanger", "MLPA", "not needed" (High confidence true positive), "needed" (likely needs confirmation), and "unknown". May want to link with others from our current (extensible) observation.method list: https://r.details.loinc.org/AnswerList/LL4048-6.html
Jamie Jones (Jul 01 2019 at 21:34):
It may be more natural to split it into 2 components, 1 for confirmationMethod and one for confirmationStatus. Thoughts?
Kevin Power (Jul 01 2019 at 22:30):
Interestingly, that answer list just has 'sequencing' that includes Sanger - so certainly would need to be extensible. But otherwise, I think a component called Confirmation Method sounds good.
Regarding Confirmation Status, I guess we just need to concretely define what the answer list would be?
Did the FHIR subgroup still want to recommend components of Variant for these, or a separate profile that references Variant?
Kevin Power (Jul 01 2019 at 22:37):
@Mullai Murugan You seemed to like the separate profile approach. I can see the simplicity of just having new component(s) on Variant. In short, I am torn between our options.
Patrick Werner (Jul 02 2019 at 09:48):
I agree: having confirmation method and status as separate components seems to be cleaner.But these two items are clearly linked together which is not possible with one component.
Patrick Werner (Jul 02 2019 at 09:49):
We could solve this with an invariant.
Patrick Werner (Jul 02 2019 at 09:51):
Or we could say that a variant conformation should be its own profile on Observation.
Kevin Power (Jul 02 2019 at 14:20):
This is on the agenda for today's meeting, so we can review our options.
Mullai Murugan (Jul 11 2019 at 16:34):
@Kevin Power @Patrick Werner any consensus on this?
Kevin Power (Jul 11 2019 at 21:57):
I am afraid we don't have a consensus yet. :frown: I hope this one is a topic next Monday / Tuesday. But of course, we always welcome additional input, ideas, or proposals
Patrick Werner (Jul 17 2019 at 14:07):
we discussed this during the last FHIR call:
https://docs.google.com/document/d/1FGCQRtxJKyHhnC1uB_t4sJZ9yXbLMGOqPXHPr5tSLLQ/edit#bookmark=id.o5puo7wbprdm
Patrick Werner (Jul 17 2019 at 14:08):
and had three options:
- 1 component
- 2 components
- its own observation profile
Mullai Murugan (Sep 05 2019 at 00:12):
@Patrick Werner @Kevin Power thanks for discussing this. I am currently including this as a note element in Variant. If CG roadmap includes treating variant as more of a KB item, then including confirmation as a component in variant probably voids future proofing. I am now leaning more towards an Observation profile.
Re Clem's comment, while we report only on confirmed variants for clinical reports, we can report without confirmation for research reports.
Bob Dolin (Jul 09 2020 at 23:42):
Interesting article I just read, describing the Medical Genome Initiative (https://genomemedicine.biomedcentral.com/articles/10.1186/s13073-020-00748-z). From the article "The themes of analytical validity, clinical utility, data infrastructure, and test interpretation and reporting emerged as areas where this guidance was most needed". I wonder if they might be a good group to talk to, if we try to tackle a representation of analytic validity.
Bret H (Jul 27 2020 at 13:04):
confidence in the a call is a quality statement about the methodology. 1) Why is would a Clinical Lab report a low confidence variant? 2) Is this low confidence in the call that there is a variant? OR low confidence that the sequencing method actually detected nucleotide over the relevant positions for the variant? Keep it simple. A component is appropriate to me , confidence is a qualifier - NOT an independent observation.
Bob Dolin (Dec 14 2020 at 19:43):
Wondering if we can revisit this. We have a scenario where we'll be receiving a raw VCF file plus a separate file showing those variants that have past some type of confirmatory testing. We are exploring the use of both files in a decision support pipeline, but need to differentiate those that are / are not confirmed. I'd like to propose something relatively simple for starters, that we add a 'variant-confirmation-status' component, with HIGH/INTERMEDIATE/LOW values:
variant-confirmation-status
- HIGH: High confidence true positive variant call. Suitable for clinical reporting.
- INTERMEDIATE: Candidate true positive variant call. Unable to confirm without additional testing.
- LOW: High confidence false positive variant call.
Kevin Power (Dec 15 2020 at 17:01):
@Bob Dolin - Any publicly available value set we could try to leverage? I am betting the answer is no, but just wondering if we should look.
Bob Dolin (Dec 15 2020 at 17:08):
@Kevin Power Not that I know of. The HIGH/INTERMEDIATE/LOW was inspired by this article [https://www.jmdjournal.org/article/S1525-1578(18)30291-5/pdf], which includes this picture: image.png
Kevin Power (Dec 15 2020 at 17:50):
If the group can review that article and feels the options are good, then that is good enough for me. When I ask about a value set, I don't necessarily mean "a pre-existing, codified list of values" (though that is always great). But maybe better stated question would be "is there a list of answers from a reputable source?" Often times a list proposed in an article from a well regarded source is perfectly fine for input.
Jamie Jones (Dec 15 2020 at 17:58):
Can/should we marry this with guidance on how to declare a variant already confirmed?
Bob Dolin (Dec 15 2020 at 17:59):
@Jamie Jones not sure I'm following. Can you expand on that a bit?
Jamie Jones (Dec 15 2020 at 18:00):
Would we suggest labs to use this component to mark confirmation status where they did orthogonal testing? Could a fourth value of "confirmed" work?
Bob Dolin (Dec 15 2020 at 18:01):
I was thinking that if a lab did orthogonal testing, they would use a value of HIGH.
Kevin Power (Dec 15 2020 at 19:12):
I do wonder if we need to represent 'variant-confirmation-method' in some way?
Bob Dolin (Dec 15 2020 at 22:28):
@Kevin Power I think I'm kinda neutral on the need to represent variant-confirmation-method in that I just need to know that the lab considers it confirmed. That said, I don't mind adding it if others need it.
Kevin Power (Dec 15 2020 at 22:51):
@Bob Dolin -- So, we have this in the backlog: https://jira.hl7.org/browse/FHIR-19829
It came from @Mullai Murugan so perhaps we need to get her take on this for the value set, and if method is really required?
Jamie Jones (Dec 15 2020 at 22:54):
In my mind they are 2 separate requests, one to support listing confirmation test method and one to flag unconfirmed results as high confidence. Mullai's use cases could be approximate by using 'high' for confidence level, but that list would really be built from the perspective of a computational analysis rather than a separate test.
Mullai Murugan (Dec 16 2020 at 22:02):
Our use case does two things - indicate the confirmation method and result of the confirmation. The primary intent is to communicate that the variant was confirmed.
Regarding the other use case mentioned by @Jamie Jones - flag unconfirmed results as high confidence- we typically don't do that in a final report, only confirmed variants are resulted in a final report. However, a preliminary report might list a variant that is not confirmed yet. The preliminary report will eventually be overridden by a final report.
Bret H (Mar 03 2021 at 14:19):
Is anything other than Sanger confirmation/testing used routinely? And can it be assumed that we don't need to give the primer set? Also, wouldn't confirmation testing be part of the 'package' of the method of the lab? How deep do you need to have the report go? The simplicity of High/Intermediate/Low as reported by the lab is easy to use. It requires trusting the lab's determination. In practice, how is the method description of the confirmation testing used? Does a person use it in decision making?
Mullai Murugan (Apr 02 2021 at 20:03):
@Bret H , I will add this item back to the agenda for the Apr 06th meeting so that we can get more feedback. Our lab primarily uses Sanger for SNPs and MLPA for CNVs. Yes, confirmation testing is part of the package. The report is rather light - all we indicate is "Confirmed by Sanger sequencing" for the variant. We do not use High/Intermediate/Low; I see that the confidence status is optional (variant-confidence-status (0..1)), with our current pipeline, we will very likely not include the confidence status field.
Bob Dolin (Jan 15 2022 at 00:13):
@Kevin Power @Jamie Jones I might be spacing out. Jira 19829 shows that we added a new component variant-confidence-status, and I could have sworn it was there, but now I don't see it in the latest build.
Kevin Power (Jan 15 2022 at 00:28):
Well it would seem it didn’t get applied for some reason. I will try to knock it out
Bob Dolin (Jan 15 2022 at 00:54):
Thanks Kevin
Jamie Jones (Jan 15 2022 at 00:57):
Yes it should be in
Kevin Power (Jan 15 2022 at 15:37):
This has been applied:
HL7/genomics-reporting: master rebuilt
Commit: FHIR-19829
Add new CS and VS for variant confidence status
Add component slice to Variant profile
Add documentation to Variant Reporting
Add examples
(also includes a minor QA fix to remove errors of duplicate resource names where title text between parens is ignored)
(Kevin Power) :thumbs_up:
Kevin Power (Jan 15 2022 at 15:38):
I did apply it directly to master, but welcome everyone to review and provide some feedback. I will admit I am not really crazy about my attempt of documentation on the Variant Reporting page.
Arthur Hermann (Jan 17 2022 at 21:10):
Kevin Power said:
I did apply it directly to master, but welcome everyone to review and provide some feedback. I will admit I am not really crazy about my attempt of documentation on the Variant Reporting page.
Kevin - I looked on the new Variant Reporting page.... below is a suggested "improved version" (hopefully). I just edited for clarity and ease of reading... please feel free to ignore if you don't agree with these edits..
Variant confidence status provides a way to indicate the reporting organization’s confidence that the variant is truly positive. Noting the confidence level may be important in the overall interpretation of the variant and related implications. Confidence is noted as High, Intermediate or Low. The following example shows how to express a high variant confidence:
It is important to note that variant confidence status is not a required component. If the reporting organization’s confidence levels are not structured in a way that it can be reported using this standard coding system, implementers must determine other ways to ensure that confidence in a variant call is understood. For example, a laboratory might only send structured variants when the confidence is 'high' that they are truly positive.
Kevin Power (Jan 18 2022 at 14:15):
Thanks @Arthur Hermann - I made these changes and you can see them here:
http://build.fhir.org/ig/HL7/genomics-reporting/branches/master/sequencing.html#variant-confidence
Last updated: Apr 12 2022 at 19:14 UTC
