Stream: implementers
Topic: How to keep consistency with previous data?
shoho (Apr 22 2021 at 03:26):
One day you find you are using wrong property for Resources, what should you do for previous data?
For example, about Identifier in Patient, you used SB (social beneficiary identifier). But after a few months, you find the number you regarded as SB is actually TAX (tax id number), and you want to start to use TAX instead of SB.
But previous Patient resources are stored with code SB, so I want to revise the previous data to keep consistency.
However, I don’t want to touch data itself since it is not safe to do it.
Are there any good ways to keep consistency for data?
Thank you.
Lloyd McKenzie (Apr 22 2021 at 19:39):
Inconsistencies in data is a reality:
- different users capture data differently
- systems have different rules (and capabilities) at different points in time
- different systems have different rules
- conversion engines perform conversions differently over time
- different interface standards have different expectations
- different 'versions' of the standards used to define the interface have different expectations
However, forcing searching/consuming systems to manage that inconsistency (the default) imposes significant costs on a potentially large number of client systems. So it's generally cheapest if the data source manages and hides the inconsistency from the client systems accessing it.
In general, the 'new' expectations for data are the 'right' ones. I.e. you want all legacy data to be exposed according to the 'current' rules, 'current' standards, etc. These are presumably more useful, more correct or more complete - or they wouldn't have changed. However, some clients might not be prepared to move to the 'new' behavior.
There are two typical strategies that should be employed in parallel
- systems should have a 'conversion' layer between the internally persisted data and the external interface that is capable of manipulating the data to expose it in a consistent way, regardless of how it's stored internally
- this same layer should allow for multiple external interfaces
There's no magic to this - writing conversions is hard grunt work and, unless you're talking about converting between two externally-defined standards, the work is going to be system-specific. FHIR defines ConceptMap and StructureMap that can help with this. There are lots of third-party transformation tools and integration engines that can perform this too.
Lloyd McKenzie (Apr 22 2021 at 19:43):
(Edited the above - hit enter before I'd finished re-typing from my draft response to your stack overflow post before the question got closed.)
shoho (Apr 23 2021 at 06:55):
@Lloyd McKenzie
Thank you for your detailed advice. It is very important to keep consistency since FHIR standards are still being updated and they still have flexibility to implement almost anything.
I will deal with each case to decide whether to keep frontend consistent or backend consistent.
Last updated: Apr 12 2022 at 19:14 UTC
