Stream: implementers
Topic: Image Annotations
Grahame Grieve (Jun 06 2018 at 21:51):
https://stackoverflow.com/questions/50727179/fhir-imagingstudy-annotation - @Brad Genereaux @Elliot Silver
Elliot Silver (Jun 06 2018 at 21:52):
The answer currently there is mine. Lloyd alerted me. Is there anything you want me to add?
Grahame Grieve (Jun 06 2018 at 21:52):
lol I hadn't read that far - sorry
Elliot Silver (Jun 06 2018 at 21:52):
Np.
Grahame Grieve (Jun 06 2018 at 21:53):
Don't know whether this is something that would arise with media
Elliot Silver (Jun 06 2018 at 21:57):
We had a short discussion at the WGM about linking Media and ImagingStudy via Provenance, and will come back to it on an upcoming call. It’s not the same use case but there’s probably some overlap. More details would help.
Brian Bialecki (Dec 12 2019 at 19:08):
Hello, ACR is interested in linking physical annotations to observations, thus providing a concrete link to the markup made on an image and the observation itself. For example a bi-ruler on a liver noted as a lesion with the observed characteristics, could be linked with the derived from and imaging study in observation, but to list the actual annotation itself, such as in DICOM TID1500 AIM does not seem currently possible. This is important when lesions with different characteristics are close to each other, as well as to properly annotate and report results from machines. We are looking to post a sample for review and wondering if there is anyone else who has looked at this in the past?
Brian Bialecki (Dec 12 2019 at 19:13):
A sample of what we have been looking to post in the SIIM FHIR server is here:
https://github.com/ACRCode/dsi-standards-examples/blob/master/CDE%20%2B%20AIM/FHIRpneumothorax.json
Elliot Silver (Dec 12 2019 at 19:24):
Current Imaging Integration leadership @Jonathan Whitby and Chris Lindop should be able to provide the latest details, but I would use an extension. There are plans to bring back a method to reference specific instances and frames, but I don't think that's granular enough for what you want. You're talking about an observation being "derived from" a specific finding in a DICOM SR. If you are using v4, I would define an extension to that lets you identify the specific SR series/instance, and finding within the instance. This extension could be applied to any ImagingStudy reference (hence, why you don't need to include the SR's study in the extension).
Grahame Grieve (Dec 12 2019 at 19:26):
that example is very non-conformant, which makes it hard to figure out which bit you are asking about
Grahame Grieve (Dec 12 2019 at 19:28):
how much sophistication do you want for identifying a region? rectangular? or full polygon?
Brian Bialecki (Dec 12 2019 at 19:28):
Thank you Elliot, not just the DICOM SR, or GSPS which I have seen in a previous string, but if 2 lines were drawn on the same image instance, a way to discern, which line was drawn for two potentially different observations, for example 2 lesions on a mammogram, where 1 was recommended for biospy as a BI-RADS 4B and one was to be followed as BI-RADS 3
Elliot Silver (Dec 12 2019 at 19:28):
I'm not thrilled with the approach you've taken creating an ImagingStudy with only the endpoint and the single instance information, but I realize we (the working group) didn't give you much choice--I think this is was an error on our part. Please submit a Jira ticket so that the work group is sure to consider your use case. I would prefer to see an ImagingStudy resource with information about all instances, and then an extension on the reference to that resource, calling out the specific content you're interested in.
Elliot Silver (Dec 12 2019 at 19:31):
@Grahame Grieve , Brian is trying to tie a finding or annotation in a DICOM instance to an Observation created from that finding. The question is how, from FHIR, to reference something within a DICOM instance.
Brian Bialecki (Dec 12 2019 at 19:31):
@Grahame Grieve I want to say that a line 5.7mm was drawn on the image because I am calling that a lesion, and then reference the 5.7mm annotation line
Brian Bialecki (Dec 12 2019 at 19:32):
@Elliot Silver Sorry to ask, but I am new here and where can I input the JIRA ticket
Grahame Grieve (Dec 12 2019 at 19:32):
what's 'a line 5.7mm'? that doesn' t sound very identifying to me
Elliot Silver (Dec 12 2019 at 19:33):
I can't recall if individual annotations within a GSPS get UIDs, but if so, you could use that. If you are talking about burned-in markings, yeech, well, I suppose some sort of coordinates would be the best you could do.
Elliot Silver (Dec 12 2019 at 19:34):
@Brian Bialecki https://jira.hl7.org
Brian Bialecki (Dec 12 2019 at 19:41):
Thank you @Elliot Silver for the Jira link. @Grahame Grieve the 5.7 mm line is an example. Lets say I wanted to tell you in my example I found a Pneumothorax, and the one of the items I should be observing was pleural separation, so I measured that on an image and reported it as an observation. I want to be able to tell you the exact image I derived that observation from and exactly the end points of the line I drew to determine it.The thought is that this provides a clear link when annotating a dataset and allows for the observer to be external and the radiologist to agree or disagree with the use of provanence and versioning
Elliot Silver (Dec 12 2019 at 19:42):
You might be able to identify the line in a GSPS using its Tracking UID or Graphic Instance ID. I'd need to delve deeper.
Grahame Grieve (Dec 12 2019 at 19:45):
what's a GSPS?
Elliot Silver (Dec 12 2019 at 19:45):
Also, take a look at Provenance. This is the general mechanism for "where did this info come from". The "derived-from" attribute in Observation has a narrower scope. Provenance won't have a way to reach into a DICOM instance--you'll still need that extension--but it will allow you to record more details about the derivation (when, where, etc.)
Grahame Grieve (Dec 12 2019 at 19:46):
I know that in both my own implementation of this, and in CDA, we defined a region of interest on the image, and then we referenced that from the narrative. The region of interest had an id, and also identified an area of the image by rectangle. I'm not sure what 'a line' is
Elliot Silver (Dec 12 2019 at 19:47):
GSPS: Grey Scale Presentation State. A DICOM instance that has direction on how to display other instances (images) including annotations (text and graphics), brightness/contrast, zoom, etc.
Grahame Grieve (Dec 12 2019 at 19:47):
so the core question is, this time, how do you refer to an object inside a DICOM instance as the source for an observation?
Brian Bialecki (Dec 12 2019 at 19:48):
Yes, This is true, but if I draw two arks on the same image or in the same study, they would be contained in the same GSPS and I could not link the individual mark to a specific observation
Brian Bialecki (Dec 12 2019 at 19:50):
The core is, just as AIM in DICOM was added to provide context to the mark itself, how in FHIR do I link an observation to the mark/annotation on a specific image, or ROI within a study, not just point to a GSPS, but the specific area
Elliot Silver (Dec 12 2019 at 19:53):
@Grahame Grieve -- yes. @Brian Bialecki How would we do it if this was all DICOM? I think the identifiers I mention above are what's used. Whatever identifiers we use in DICOM (or AIM -- the Annotation and Image Markup standard) to identify the markings, we should be able to record those in FHIR. And, for now, solving the issue on the FHIR-side would mean creating an appropriate extension.
Elliot Silver (Dec 12 2019 at 19:54):
@Brad Genereaux @Kevin O'Donnell @Teri Sippel Schmidt Thoughts?
Brian Bialecki (Dec 12 2019 at 19:55):
AIM is the way in DICOM and I took this information from TID1500 and discussions with Daniel Rubin. The example, and yes, I unfortunately know it is not the best, but I was hoping to show what I was attempting to accomplish as an example to begin a discussion
Grahame Grieve (Dec 12 2019 at 19:59):
if I draw two arks on the same image or in the same study, they would be contained in the same GSPS and I could not link the individual mark to a specific observation
Do they get identifiers within the GSPS?
Gino Canessa (Dec 12 2019 at 19:59):
Is GSPS or AIM common enough now to exclude other methods (e.g., burn-ins, overlays (60xx,3000), Graphic Annotation Module, etc.)?
Brian Bialecki (Dec 12 2019 at 20:19):
AIM has recently been added to DICOM TID 1500 and this was a buzz at this year's RSNA. This does address the example of maultiple marks in the same GSPS. I think that currently Key Images and GSPS can be handled if they are 1-1, but this is not the case, and I am looking to get a more concrete way to link annotations and observations, which would hopefully encompass all these methods.
Gino Canessa (Dec 12 2019 at 20:57):
That's what I get for not going this year. That said, a 4-day Thanksgiving holiday was incredible.
Catching up, TID 1500 appears to map AIM into DICOM SR (which is yet another format from what we've been discussing). 2019e-3.21 specifies the mapping is only for one direction, so presumably the idea is that PACS systems can ingest AIM documents and convert them into DICOM for use.
But I feel like we've lost sight of the question (which Grahame nailed earlier)..
Generically, how do you link something inside a DICOM instance to an Observation? In particular, Observation.derivedFrom can goes to ImagingStudy, not a specific instance and not a specific tag/UID within an instance. Is that the question (leaving aside that AIM is one of many annotation formats in use today)?
Or is this a more specific request about handling image annotations (and findings per the original message in the thread) in particular?
Brian Bialecki (Dec 12 2019 at 21:10):
I am looking to link something like a ruler measurement, which is a line defined by two endpoints in DICOM on a specific image instance to the observation it was made in reference to. So If I see a non-spiculated mass in the right breast on the CC view that is 5.8mm in diameter. I want to link the to the line itself.
Grahame Grieve (Dec 12 2019 at 21:13):
well, @Elliot Silver is right that it's going to be an extension. its not obvious, though, how you actually refer to the content, if the annotations aren't identified internally?
Vassil Peytchev (Dec 12 2019 at 21:19):
From what I am reading, they have internal IDs
Grahame Grieve (Dec 12 2019 at 21:19):
and they object has an OID? so urn:oid:[oid]#id ?
Elliot Silver (Dec 12 2019 at 21:19):
If the annotations within the instance don't have individual identifiers (I think they do, but am not sure), then you can come up with a way of identifying them yourself, for example, by their position in the graphical element sequence. It's not pretty, but since DICOM requires a new instance UID if you modify the clinical information in the instance, like reordering the elements in a sequence, you should be safe.
Elliot Silver (Dec 12 2019 at 21:27):
Vassil, yeah, that's my reading too. I just want to confirm my understanding.
Graham, no need for the #id, I think it would just be an identifying OID for the marking. Actually, you'd probably want to use the DICOM UID identifier scheme, rather than a plain OID, and have a type that said whether it was text or graphics or a region of interest or... (I believe each are listed within a separate part of the annotation instance, so knowing where to look would be helpful.)
Gino, I suspect that while GSPS isn't universal and is somewhat young still (15 years?), it is a reasonable restriction to put on a solution looking to use FHIR. I also suspect that the use case is AI related. Nothing wrong with pulling vendors kicking and screaming into the 21st century, now that we're 20 years in.
Grahame Grieve (Dec 12 2019 at 21:31):
I think it would just be an identifying OID for the marking
but you have to know which object to look inside of? or is it reasonable to expect inner UIDs to be indexed?
Elliot Silver (Dec 12 2019 at 21:55):
Yes, you'd have to communicate the DICOM SOP instance (i.e, the GSPS "file") UID that the marking was in. I had assumed there would be a separate element in the extension to communicate that. No, I don't think you can look up the marking UID anywhere either external to the file, or internal to it.
Josh Mandel (Dec 12 2019 at 21:57):
Bigger-picture FHIR modeling question here: we know how to write a reference to an element within a resource, like Questionnaire/123/#some-id. But we don't actually have a way to profile the use of this (like, defining a profile with a Reference to an element within a resource (or a particular type of backbone element within a particular type of resource). Like, to say in a profile "This slot should be a reference to an Item backbone element within a specific Questionnaire. Has this come up before?
Grahame Grieve (Dec 12 2019 at 22:22):
yes you can do that.
Lloyd McKenzie (Dec 12 2019 at 22:31):
How?
Grahame Grieve (Dec 12 2019 at 22:35):
http://hl7.org/fhir/extension-elementdefinition-profile-element.html
Gino Canessa (Dec 12 2019 at 22:38):
Elliot, yes I think GSPS is reasonable - but GSPS isn't AIM.
GSPS objects contain information about the state of the image view (e.g., window/level) and pixel data which can be used as overlays. The pixel data is not separated into unique annotations within an instance. They are essentially burn-ins that aren't burned in.
Edit: Overlays can be graphics objects (e.g., a line, etc.), but both types are supported and I cannot speak to the prevalence of either method. That said, they do not have UIDs that I can find (only a label in 0070,0080).
Edit: GAM is in GSPS (had to look up since it was driving me crazy). This is for things like annotating a displayed MPR - which aren't tied to a single instance. I have no idea how that would get linked in (series level?)
Brian Bialecki (Dec 16 2019 at 18:38):
@Grahame Grieve @Elliot Silver I see there is a draft of ImagingReference that is posted as item 17183 in gforge, and I was wondering if I could take a look at this and try and fit it into the ACR use case
Grahame Grieve (Dec 16 2019 at 19:43):
Grahame Grieve (Dec 16 2019 at 19:45):
that doesn't seem to actually be going anywhere. you're welcome to assume that it's going to be part of R5 but it's pretty old... @Elliot Silver what's the plan here?
Brian Bialecki (Dec 16 2019 at 19:53):
I was on a call with WG20 today and it pears this has some traction again
Elliot Silver (Dec 16 2019 at 22:07):
It did stall out for a bit. When I last looked it was something they were planning to get into R5, but I'm not sure how quickly its getting there. Last I looked it needed a resource proposal. If @Brian Bialecki was on the call today, you've probably got the best info.
Brian Bialecki (Jan 22 2020 at 13:35):
Based on DICOM WG 20 ImagingReference, I created the ImagingReference resource which can be found here: https://github.com/ACRCode/dsi-standards-examples/tree/master/CDE%20%2B%20AIM We are attempting to place this on the public SIIM FHIR server to get feedback on this along with examples that have been cretaed. Although the server accepts it as a StructureDefinition and we can see it, when we try and upload resources using it we get the following error: "Unknown resource type 'ImagingReference' - Server knows how to handle: [Appo..." Is there somewhere this also needs to be referenced to be listed?
Brian Bialecki (Jan 22 2020 at 15:26):
I have been looking around, do you think I would need to add a Domain Resource definition to get ImagingReference to be available
http://hl7.org/fhir/domainresource.html
Jose Costa Teixeira (Jan 22 2020 at 17:06):
Imaging reference is not a FHIR resource so the server is right.
Jose Costa Teixeira (Jan 22 2020 at 17:07):
If you want to get feedback : Do you want to publish the specification or the implementation? The specification would come first and this is for the imaging committee to approve and put in the build (you can run the build locally as you are working on it)
Jose Costa Teixeira (Jan 22 2020 at 17:08):
Before that - what is an ImagingReference? What is the use you have for it?
Brian Bialecki (Jan 22 2020 at 17:10):
@Jose Costa Teixeira we realize that is is not a resource, but we are trying to add it so that we can solicit feedback to some work we are doing as we work on adding this as part of the next ballot period
ImagingReference is used so that you can point to a specific image and markup(annotation) in reference to an observation
Brian Bialecki (Jan 22 2020 at 17:12):
We have defined the StructureDefinition as a resource and it is accepted but we do not know how then to get the server to use it, and thought maybe we needed the addition of domain resource to make that happen
Lloyd McKenzie (Jan 22 2020 at 17:16):
Some test servers will support custom resources but most don't. You'll have to reach out to the maintainer of the server
Brian Bialecki (Jan 22 2020 at 17:28):
Just a question, if they do, which the maintainer says they do, to get it on the list of known resources, do we need to create a DomainResource definition as well as the StructureDefinition?
Lloyd McKenzie (Jan 22 2020 at 18:18):
DomainResource is an abstract type - you can't create an instance of it. I'd expect servers to require a StructureDefinition defining the resource and potentially some SearchParameters
Jose Costa Teixeira (Jan 22 2020 at 18:19):
If I'm not mistaken, the documentation for the server should include the instructions on how to add custom resources
John Moehrke (Jan 22 2020 at 18:25):
There is a resourceProposal but we have not yet prototyped it as a FMM 0 resource. Given this proposal, it seems that II might be ready to give me the details so that I can prototype this as an FMM 0 resource. Right @Brad Genereaux ?
Brian Bialecki (Jan 23 2020 at 15:34):
@Brad Genereaux @John Moehrke I have started to protype this with DICOM WG20. Johnathan Whitby was going to use what I have on Github as a start, which is why we are looking ot post it on the SIIM server as well, which it is located here:
http://hackathon.siim.org/fhir-overview/fhir/StructureDefinition/ImagingReference
Brian Bialecki (Jan 23 2020 at 15:38):
We also updated observation to add this to basedOn
"base": {
"path": "Observation.basedOn",
"min": 0,
"max": "*"
},
"type": [
{
"code": "Reference",
"targetProfile": [
"http://hackathon.siim.org/fhir-overview/fhir/StructureDefinition/CarePlan",
"http://hackathon.siim.org/fhir-overview/fhir/StructureDefinition/DeviceRequest",
"http://hackathon.siim.org/fhir-overview/fhir/StructureDefinition/ImmunizationRecommendation",
"http://hackathon.siim.org/fhir-overview/fhir/StructureDefinition/MedicationRequest",
"http://hackathon.siim.org/fhir-overview/fhir/StructureDefinition/NutritionOrder",
"http://hackathon.siim.org/fhir-overview/fhir/StructureDefinition/ServiceRequest",
"http://hackathon.siim.org/fhir-overview/fhir/StructureDefinition/ImagingReference"
]
}
],
Brad Genereaux (Jan 27 2020 at 16:28):
tagging @Jonathan Whitby (should also tag Chris Lindop) - yes - this relates to the proposed resource that we batted about back in 2018 - the ImageReference. (in answer to @John Moehrke question)
Martin Bellehumeur (Visage Imaging) (Mar 11 2020 at 15:38):
@Brian Bialecki I’m working on a PACS application that generates FHIR observation resources that are consumed by a DiagnosticReport resource app. I’m interested in testing implementation of the ImagingReference resource. If you have additional examples created, I would appreciate checking them out.
Brian Bialecki (Mar 11 2020 at 16:17):
After work with II and the DICOM WGs it was decided to pursue this as an extension of DocumentReference. I have placed the imagingContext extension on the SIIM public FHIR server and have been able to get an example resource to properly parse. I plan to now create the full circle of AIM/DICOM SR and FHIR Diagnostic Report to show these all together and mapped.
Brad Genereaux (Oct 27 2020 at 04:45):
Hello - sorry for reviving an old thread, but I think this topic is the closest to a question I'm thinking through.
I'm thinking about annotations, and specifically, on 3D representations outside of DICOM (represented in NIFTI, think of it as a three dimensional matrix file format). I could have both a NIFTI 3D representation of an imaging study, as well as a 3D representation of a mask (0 = not an area of interest, 1 = area of interest). If I represent these as FHIR objects - presumably Media - is there any obvious way to relate them? (basedOn doesn't seem to work unless I extend Media).
Jose Costa Teixeira (Oct 27 2020 at 06:19):
Hi Brad :)
Jose Costa Teixeira (Oct 27 2020 at 06:22):
I think it makes sense, but I defer to colleagues more into imaging for a better answer. Just adding one possibly related topic: In some procedures we also capture other media (waveforms) and it is important not only to relate them, but also to "synchronize" them. For example in wavelenghts, we say that the zone of interest starts at 1'02" in one channel but in the other channel (which was recording earlier), the same event starts at 2"54'.
Jose Costa Teixeira (Oct 27 2020 at 06:23):
I don't know if you need only to link the media, or to establish the common points - any offset for alignment.
Lloyd McKenzie (Oct 27 2020 at 13:11):
@Elliot Silver
Elliot Silver (Oct 27 2020 at 16:02):
@Brad Genereaux, are you looking at R4 or is this a more abstract question? Media is gone from R5.
In R5, my answer would be to put each representation in an Observation (specifically, Observation.valueAttachment) and link them with Observation.derivedFrom. I remember we had a way to put content in a DocumentReference, and point the Observation to that, but I can't recall the details of that right now.
In R4, I don't see an obvious way to do this. (How did we miss "derivedFrom" in Media?) I think there is an extension to add an attachment as an Observation value, but can't find it immediately.
Last updated: Apr 12 2022 at 19:14 UTC
