Stream: implementers
Topic: AuditEvent
Ardon Toonstra (Aug 18 2016 at 12:59):
Hi there!
I have two questions about making a profile on the AuditEvent resource. The use case entitles that every action within applications that contain patient information needs to be logged. The two questions:
1. There is a need to capture the ‘role’ of the participant, which contains in this case a concatenated value of an ID, position and organization. Is it possible to use the AuditEvent.participant.role element for this? If I look at, for example, the extensible valueset beloning to this element I get the idea that this element is meant for something else. The element, however, would cover our need.
2. For the AuditEvent.object we want to capture the patient, the accessed information and some metadata properties of the information that is accessed (or example the lastmodified date). What would be the best way to model the metadata of the accessed information in our profile?
Thanks in advance for any suggestions/ideas!
Grahame Grieve (Aug 18 2016 at 18:52):
#1 - why a concatenation of these things? but position would be the nearest in intent to role.
#2 - I think the intent is that these things go in entity.detail. but that looks less appropriate than using extensions, really. name = value pair - you might as well use an extension
John Moehrke (Aug 18 2016 at 19:51):
Ardon,
John Moehrke (Aug 18 2016 at 19:54):
#1 - yes, this can go in agent.role; it is extensible because we have an infinite set of roles. You can concatenate those values to show the relationship, or you could use the fact that your conconcatenation is a form of code. with the organization is the sysem that issued the 'position' as the value. I don't understand why you would repeat the ID, unless you don't mean the user-ID, but some role-ID?
John Moehrke (Aug 18 2016 at 19:57):
#2 - you would record one AuditEvent.entity that just describes the patient. Likely just an AuditEvent.entity.reference pointing at the Patient resource. You would have another AuditEvent.entity for the data. If you point at the versioned data (AuditEvent.entity.reference pointing to version specific); then you don't need to duplicate the lastmodified (or any other metadata about the data). This would be my recommendation.
Ardon Toonstra (Aug 19 2016 at 09:50):
Thanks for your input.
#1 is clear to me now, we will use agent.role.
#2 We have indeed one AuditEvent.entity for the patient with a reference to a Patient Recourse. Also another entity for the data, however we won’t be able to point to versioned resource of the accessed data.
We would like to use entity.detail but this seems inappropriate with its base64Binary value element. Wouldn’t it be an improvement to have a value[x] in the entity.detail?
Grahame Grieve (Aug 19 2016 at 10:55):
that's entity.detail should just be an Extension - that's just a name = value[x] pair
John Moehrke (Aug 20 2016 at 16:40):
Ardon, so you say you can't use version specific data resource references, so you want to duplicate that which would be in a version specific data resource in the audit? This seems like a bad design. If you need version specific data resources, then you should have them. Trying to use audit as an auxiliary database of metadata is very inefficient. You would need to duplicate this metadata thousands of times over, for every access/use/disclosure. Do you use Provenance resources? Can you point at a version specific Provenance resource that points at the non-versioned data resource?
Brian Postlethwaite (Jan 11 2017 at 01:45):
@John Moehrke I'm reviewing my implementation (again) of the population of the AuditEvent resources and noted that the object.type coding seems to have an odd set of values, I would have expected it to have included the resource types in that coded set (yes I could add them as it is extensible).
http://hl7.org/fhir/auditevent-definitions.html#AuditEvent.object.type
John Moehrke (Jan 11 2017 at 13:58):
@Brian Postlethwaite It is defined to pull in the resource types. Not sure why it isn't. I might need some @Grahame Grieve help. http://build.fhir.org/valueset-object-type.html
Grahame Grieve (Jan 11 2017 at 18:38):
That list comes from atna. I always thought it was crap.
John Moehrke (Jan 11 2017 at 19:48):
Grahame that is not the question... The question is that the valueset should also include all of the FHIR Resource types. But it appears not to include them...
Grahame Grieve (Jan 11 2017 at 19:51):
well, we can make it do so. editorially, we would have to :
- grab the generated code system and value set resources and copy them into the source directory
- change the binding to reference the value set resource directly
- add the resource type value set to an include in the value set
Abbie Watson (Jan 23 2017 at 16:20):
+1 for using the FHIR Resource types. That's roughly how we record events with our current HIPAA Audit Logger, and we'll happily adjust things to use the AuditEvent resource and suitable value sets.
John Moehrke (Jan 23 2017 at 17:06):
This is now logged as GF#12677
Shovan Roy (Mar 24 2020 at 22:07):
Hi All,
I've couple of queries regarding AuditEvent:
- I can see a dedicated element 'AuditEvent.entity.query' for query. what about read? what is the suggested way to store they 'id' for a read operation? put it in 'AuditEvent.entity.detail'? Any particular reason why 'AuditEvent.entity.query' doesn't have a 'string' option as well?
- What is the best place to record 'x-correlation-id' or similar type of interaction id? We would like to trace all internal events (orchestration) for a particular transaction but at this moment didn't find any such place in AuditEvent resource
@John Moehrke, @Lloyd McKenzie any suggestion?
Lloyd McKenzie (Mar 24 2020 at 23:19):
I'm going to leave the response for this one to John :)
Shovan Roy (Mar 24 2020 at 23:26):
thanks @Lloyd McKenzie I'll wait for @John Moehrke :)
John Moehrke (Mar 25 2020 at 12:24):
We have a bit of explanation http://build.fhir.org/auditevent.html#6.4.4.4
John Moehrke (Mar 25 2020 at 12:28):
- The id would go into an entity .what element. Have as many .entity. The example saves more than the id, but I think the id is sufficient as the audit analysis can always lookup by id. An augmentation of this is to add another .entity with the patient identity, if you know it. This enables privacy use-cases http://build.fhir.org/auditevent.html#patient
John Moehrke (Mar 25 2020 at 12:31):
The entity.query requires you to base64 encode, so as to protect the query parameters. Prevents query parameter abuse as a vector to attack the database. If they are always base64 encoded then one can't send in fragments of json or xml and cause database overruns. Consistency is important
John Moehrke (Mar 25 2020 at 12:34):
- Seems x-coorelation-id would be another .entity.
John Moehrke (Mar 25 2020 at 12:34):
Please submit a CP with recommendation and example
Shovan Roy (Mar 25 2020 at 20:09):
thanks @John Moehrke , this helps a lot. I'll provide a CP with detailing the scenario
Shovan Roy (Mar 25 2020 at 20:49):
one more thing @John Moehrke , in case of conditional create/update/delete, I guess
- The query string will go to the 'query' element
- The 'what' shall not be populated for any of these cases
- The payload will go inside the 'detail' as base 64 for cond create and update
Is this a correct assumption?
John Moehrke (Mar 25 2020 at 21:22):
queries are just record of the query with success/failure. The results are not recorded, as it is considered reproducable and to record it is to put sensitive information in the audit log for no good reason
John Moehrke (Mar 25 2020 at 21:22):
or did I misunderstand your question?
Shovan Roy (Mar 25 2020 at 21:34):
Sorry @John Moehrke , I would have provided a few more details about my quesiton.
The situation here is that we have received a conditional update API request (PUT [base]/[type]?[search parameters]) from an external party. We want to create an Audit for that. To capture the details as received in the request, this is what my plan is:
- Use the AuditEvent's 'query' element to record the query as received in API query string ([search parameters]). Unless the guideline is to capture only '_query' value in 'query' element.
- Use the entity.details to capture the payload received as base 64 bin.
- I still need to figure-out where to store the request headers
is my approach correct here?
John Moehrke (Mar 26 2020 at 12:06):
Thanks. that was not clear to me originally. Yes this would be situation we have not yet talked about. It is best to model these with someone who is working on the issue, as you are. Hence why we don't have an example, so good to get an example from you. I think that you should record this as a UPDATE restful operation, but you should include an .entity with the query parameter. I would recommend against capturing the details, as we try hard to not duplicate sensitive information in the audit log.
Shovan Roy (Mar 27 2020 at 02:36):
thanks @John Moehrke I'll provide an example..
Regarding capturing the details in Base 64. I was trying a model where Audit is kept totally in a separate database and works as single source of truth for every single transaction, kind of an event source but not really the event source.. But I see your point..
John Moehrke (Mar 27 2020 at 12:05):
that functionality is best supported by a technology specific solution. like a jorunaling database.
Jens Villadsen (Apr 22 2020 at 10:07):
John Moehrke said:
The entity.query requires you to base64 encode, so as to protect the query parameters. Prevents query parameter abuse as a vector to attack the database. If they are always base64 encoded then one can't send in fragments of json or xml and cause database overruns. Consistency is important
I don't quite follow - query parameters are base64 encoded because of what? Do you consider the parameters in plain text to be a security issue?
John Moehrke (Apr 22 2020 at 13:19):
so that they can survive being converted between FHIR XML, FHIR JSON, FHIR Turtle, FHIR JSON-LD, DICOM XML, etc.
John Moehrke (Apr 22 2020 at 13:20):
it is just a safe way to do that. Do you have an alternative? This model has worked for 20 years in the companion standard ATNA/RFC/DICOM
Jens Villadsen (Apr 22 2020 at 13:20):
I can't see that is an argument
Jens Villadsen (Apr 22 2020 at 13:20):
why not convert all strings to base64 then?
Jens Villadsen (Apr 22 2020 at 13:21):
why is this single string such a security risk while all the others are not?
John Moehrke (Apr 22 2020 at 13:22):
this is not primarily as a security risk, this is primarily as an encoding risk. The query parameters accross all possible query languages (AuditEvent is used for things other than FHIR) are not constrained enough to guarantee an ecoding problem
John Moehrke (Apr 22 2020 at 13:23):
please recognize that AuditEvent is intended to record ALL kinds of audit events, not just FHIR events. Thus it must have a general use way of recording a QUERY regardless of query language
John Moehrke (Apr 22 2020 at 13:30):
should this fact be emphasized in the AuditEvent explanation of why query is execute and why query base64 encodes? This does seem to be the thing that I need to bring forward often, and that oftenness is usually a good clue to a CP need.
Jens Villadsen (Apr 22 2020 at 13:35):
Well .. shouldn't the FHIR standard first and foremost make sense in FHIR itself - meaning that query should have been a string - or secondly a multiple type value - either string or base64?
Jens Villadsen (Apr 22 2020 at 13:37):
I cannot understand why this sort of use (base64) couldn't be handled in an extension or so
Jens Villadsen (Apr 22 2020 at 13:37):
it seems like a pretty exotic choice
John Moehrke (Apr 22 2020 at 13:51):
That is an interesting proposal. I would need others to weigh in on if that makes sense in all cases. I can't claim to know enough to guarantee that anything someone puts on a query or search could safely be placed into a string element. @Grahame Grieve ?
John Moehrke (Apr 22 2020 at 13:54):
And, it does still have the security problem... Given that AuditEvent is also there to record failed queries/search, so that investigation can be accomplished. Where an attacker could send in a query with carefully crafted parameters, such that they KNOW it will not succeed as a query, but they then know that string is encoded into an AuditEvent and processed... thus producing future bad behavior. It is true that base64encoding is a simple and effective way to encode everything consistently thus preserving all evidence as it was used.
Jens Villadsen (Apr 22 2020 at 13:55):
Again - that argument goes for all strings
John Moehrke (Apr 22 2020 at 13:55):
Thus I could see using a string for successful FHIR queries, but would still require failed queries to be recorded base64 encoded.
Jens Villadsen (Apr 22 2020 at 13:56):
why? - its still a string
Jens Villadsen (Apr 22 2020 at 13:56):
whether or not the query goes wrong
John Moehrke (Apr 22 2020 at 13:57):
so you are saying that the normal string escaping mechanics should be sufficient for all of these concerns?
Jens Villadsen (Apr 22 2020 at 13:58):
I'm saying that you could put garbage data in all the string fields out there
Jens Villadsen (Apr 22 2020 at 13:58):
and they are not base 64 encoded
John Moehrke (Apr 22 2020 at 14:03):
I am trying to be convinced.
Jens Villadsen (Apr 22 2020 at 14:04):
well the arguments regarding security does not hold - you said so yourself
Jens Villadsen (Apr 22 2020 at 14:05):
and if that was the argument, well doesn't that settle it?
John Moehrke (Apr 22 2020 at 14:05):
no I didn't. I said it was not primary
Jens Villadsen (Apr 22 2020 at 14:06):
ok - bring on the other arguments
John Moehrke (Apr 22 2020 at 14:06):
I already have. bring on the solution.
Jens Villadsen (Apr 22 2020 at 14:06):
use strings
Jens Villadsen (Apr 22 2020 at 14:07):
you don't need the envelope of base64 encoding it
Jens Villadsen (Apr 22 2020 at 14:09):
it brings no value as I see it
John Moehrke (Apr 22 2020 at 14:10):
I have explained the value.
Jens Villadsen (Apr 22 2020 at 14:17):
you mean the part where it is easier to convert between formats?
John Moehrke (Apr 22 2020 at 15:11):
Lets try a different approach. Can you express what problems you are encountering because of base64 encoding that would not appear with string? I ask this as there was an issue brought up that it was not clear which queries included as their context (parameters or output) the Patient id, thus the problem was that a query for all audit events involving patient X was unsuccesful. Thus we identified that in other cases of this common information model (audit event) there is a requirement that when the Patient id is known, that it is included as an .entity. Thus the overall use-case is satisfied without breaking the model.
John Moehrke (Apr 22 2020 at 15:12):
said another way... what is the problem, vs what is your solution to an unstated problem.
Jens Villadsen (Apr 22 2020 at 19:01):
I would go with "keeping it simple" -> use a plain string. If it is encoded due to interoperability with other standards, my suggestion would be to put it in a standardized extension or so instead. I can imagine that there are all other kinds of standards, legislation and restrictions in many areas that put limitations on a lot of various data types. If you were to embrace all that there would probably be no operational standard in the other end.
Jens Villadsen (Apr 22 2020 at 19:04):
The 'problem' is that the datatype 'string' seems sufficient to encapsulate what is needed by FHIR, IMHO. If base64 is needed due to needs from others standards / legacy reasons I would suggest to have that stated more clearly in the spec.
Grahame Grieve (Apr 22 2020 at 19:16):
well, in general I agree with @Jens Villadsen - you cannot ignore encoding issues with any wire format, so it cannot be necessary to base64 encode things to make the wire format safe. And security issues innate to the data apply anyway, after you've base64 decided the content.
Where base64 might be required is if the query is binary not text
Jens Villadsen (Apr 22 2020 at 19:18):
that is an angle that I have not considered - a binary query ...
Jens Villadsen (Apr 22 2020 at 19:18):
that is a query I would like to see
Jens Villadsen (Apr 22 2020 at 19:21):
@Grahame Grieve you have any example of that?
Grahame Grieve (Apr 22 2020 at 19:32):
we don't do any of that in FHIR
Jens Villadsen (Apr 22 2020 at 19:39):
Right - but since you mentioned it - I would still like to see an example use /use case
Grahame Grieve (Apr 22 2020 at 19:39):
dicom query? I'm guessing here... have to go digging through my dicom implementation to be sure
Grahame Grieve (Apr 22 2020 at 19:39):
LDAP query for sure, but I don't know that's in scope
John Moehrke (Apr 22 2020 at 20:28):
those are good factors. As I indicated above, I am not against having a query[x] that allows for the AuditEvent recorder to determine that the query they have is a string vs binary. However, this doesn't really simplify the consumption of AuditEvent as the recipient must be able to decode both string and binary. Even in a pure FHIR environment the recipient must recognize that it is possible that a AuditEvent recorder might have decided to use the binary encoding. So I think all we would accomplish is the appearance of more easy to use, while actually making it harder to use.
John Moehrke (Apr 22 2020 at 20:29):
there are many technical solutions --- what is the problem we are trying to solve? The only problem that has been expressed is that it doesn't fit the picture.
John Moehrke (Apr 22 2020 at 20:32):
note, I am not as negative on this as I appear... but don't want to change without good and clear benefit gained.
Jens Villadsen (Apr 22 2020 at 21:15):
what is the problem we are trying to solve?
having a clean model, would be my answer. Having chosen one datatype in the past does not necessarily make it the right choice. - and, if something doesn't fit the picture, aren't we then obliged to fix it? If the model is easier to understand and interpret by changing the datatype or allowing it to be either binary or not - shouldn't we then do it?
Lloyd McKenzie (Apr 23 2020 at 01:34):
If a query might be expressed as actual binary executable, that'd be a good reason to use base64encoded. If you're going to send something as a string, it ought to meet the semantics of string - i.e. be intended to be human-readable. If we think we'll need to convey queries from arbitrary standards that don't fit that semantic, then base64encoded as a choice is fine. But encoding issues should not be an issue - even if round-tripping. If they are, the solution is to fix the problem, not introduce base64encoded anywhere such issues might exist.
Jens Villadsen (Apr 23 2020 at 07:52):
So to sum up: Should I create a CP?
Jens Villadsen (Apr 23 2020 at 08:19):
@John Moehrke @Lloyd McKenzie @Grahame Grieve
John Moehrke (Apr 23 2020 at 13:00):
So we need to support binary, but it is true that we could better support when a query is known to be a string. So it seems that a query[x] of String and binary might be useful. What I think we must then do is come up with very clear rules on when String can/should be used, or clear rules on when Binary can/should/shall be used. I am understanding that all forms of FHIR query/search can be represented in String safely; right? If so, that is a nice deterministic rule that is well elaborated in a FHIR spec.
Lloyd McKenzie (Apr 23 2020 at 14:46):
I think the base criteria is whether the query value is something that's intended to be directly viewed/written by a human
Lloyd McKenzie (Apr 23 2020 at 14:46):
(The same as we would for attachment)
Lloyd McKenzie (Apr 23 2020 at 14:47):
Also, do we want base64binary or do we actually want Attachment? The latter would let you specify a mime type so you'd have a clue what to do with the binary content...
Jens Villadsen (Apr 23 2020 at 18:47):
very clear rules on when Binary can/should/shall be used.
Ain't the rule that if the query contains binary content, it must be base64 encoded?
Jens Villadsen (Apr 23 2020 at 18:48):
@Lloyd McKenzie
intended to be directly viewed/written by a human.
that is too vague. Even queries in FHIR expressed in either the URL or in the body of HTTP post is not intended for humans. It is intended for machines
Jens Villadsen (Apr 23 2020 at 18:50):
HTTP is not intended for human processing. Its intended for machine processing. Somewhat understandable by humans, yes
Lloyd McKenzie (Apr 23 2020 at 20:59):
My point was that if your query is XML or JSON, it would be appropriate to use Attachment for those rather than string.
Jens Villadsen (Apr 23 2020 at 21:00):
eg. search using HTTP POST?
Jens Villadsen (Apr 23 2020 at 21:32):
@Lloyd McKenzie
Lloyd McKenzie (Apr 23 2020 at 23:17):
Could be. Or some non-FHIR mechanism that uses a technical syntax that has some sort of mime type
John Moehrke (Apr 24 2020 at 14:55):
I think attachment is overkill. and I would not expect most query languages to have registerd a mimetype
John Moehrke (Apr 24 2020 at 14:55):
are all uses of string datatype intended for human consumption? That doesn't feel right
John Moehrke (Apr 24 2020 at 15:02):
Search using post would preserve the WHOLE post body in the AuditEvent.entity.query as base64binary.... This likely is not explained well enough so likely could get CR to explain it, and provide an example... I guess I could take the search example and create an auditEvent example off of it... Sound like a good idea?
John Moehrke (Apr 24 2020 at 15:04):
That doesn't mean that it couldn't be using a string datatype rather than base64 binary... right?
John Moehrke (Apr 24 2020 at 15:07):
so we are back to the proposal which is to update AuditEvent.entity.query to a query[x] of type string or base64binary.... with explanation that string should be used when it is known to be a string, with FHIR query and search as examples. The reader should understand that binary might have been used with strings as the recorder might not have been sure their query was string compatible.
John Moehrke (Apr 24 2020 at 15:07):
does string work with all FHIR query and search?
John Moehrke (Apr 24 2020 at 15:19):
John Moehrke said:
Search using post would preserve the WHOLE post body in the AuditEvent.entity.query as base64binary.... This likely is not explained well enough so likely could get CR to explain it, and provide an example... I guess I could take the search example and create an auditEvent example off of it... Sound like a good idea?
I created J#26093 for this post search example
Jens Villadsen (Apr 26 2020 at 13:07):
@John Moehrke The https://jira.hl7.org/browse/FHIR-26093 points some Da Vinci related stuff. Was that intended?
Jens Villadsen (Apr 26 2020 at 13:10):
If FHIR queries issued as GET can be in thee string then FHIR queries issued as POST should also be in the string as the searches are equivalent AFAIK. The use of base64 should be isolated to binary cases, IMHO
Jens Villadsen (Apr 26 2020 at 13:13):
does string work with all FHIR query and search?
Well, since it must be representable within an URL, I would expect so.
Jens Villadsen (Apr 26 2020 at 13:13):
Jens Villadsen (Apr 26 2020 at 13:13):
From https://www.hl7.org/fhir/http.html#search
John Moehrke (Apr 27 2020 at 11:11):
John Moehrke (Apr 27 2020 at 11:13):
Jens Villadsen said:
If FHIR queries issued as GET can be in thee string then FHIR queries issued as POST should also be in the string as the searches are equivalent AFAIK. The use of base64 should be isolated to binary cases, IMHO
You have somewhat convinced me... the main problem I have is that a consuming application must know how to deal with both string and binary encoded query. Thus adding string does not help consuming apps. So what does adding a string option help? I am not clear on what the benefit is, and I am clear that there is a negative impact on the apps that consume/use the AuditEvent.
Jens Villadsen (Apr 27 2020 at 11:48):
Well that depends on your approach and how you count. First of all, if you restrict the use of base64 to queries that contain binary elements, then that part would not be rendered anyway - hence, clients can ignore this (besides clients that are really dedicated to that exact use case of rendering binary data). Second, if that first assumption is made, then query strings are directly visible without having to decode the content as it is represented in a normal string. Third, if its a normal string it is more easily made available for searches in the future. Fourth, do you expect the vast majority of clients to be looking for binary encoded content - because I don't. As @Grahame Grieve says: you can hide complexity, but you cannot make it go away. Currently, you are hiding it, by base64 encoding all of the queries.
Last updated: Apr 12 2022 at 19:14 UTC
