Stream: ibm
Topic: Persistence Tests vs E2E Tests
Lee Surprenant (Oct 30 2019 at 20:43):
one interesting thing about our layered implementation is we end up with extremely similar tests at multiple levels:
- fhir-model (unit)
- fhir-persistence (functional?)
- fhir-server-test (e2e / integration)
Lee Surprenant (Oct 30 2019 at 20:43):
I was just reviewing https://github.com/IBM/FHIR/pull/341 and this came to my mind once again
Lee Surprenant (Oct 30 2019 at 20:44):
whats our perspective on the right place for these tests?
Lee Surprenant (Oct 30 2019 at 20:44):
should we test everything we can at each layer?
Lee Surprenant (Oct 30 2019 at 20:44):
should we be more exhaustive in the unit tests and less exhaustive in the e2e tests
Lee Surprenant (Oct 30 2019 at 20:44):
should we just favor the e2e ones and do less unit testing?
Lee Surprenant (Oct 30 2019 at 20:45):
for this particular issue, i was expecting to see some tests added in fhir-persistence
Lee Surprenant (Oct 30 2019 at 20:45):
but perhaps i'm just biased because thats where i've spent a lot of time recently?
John Timm (Oct 30 2019 at 20:47):
I think it's probably a case by case thing. I would expect that changes to PL yields, at the very least, additional unit tests. Ideally, there would also be an integration test if we think that the REST layer might be impacted by a change (for example, the paging context work we did yesterday).
John Timm (Oct 30 2019 at 20:47):
integration tests are needed when we cross component boundaries
John Timm (Oct 30 2019 at 20:48):
if the change is isolated, then i think integration tests are a nice to have but not necessarily a requirement
John Timm (Oct 30 2019 at 20:48):
IMO, most of the "exhaustive" testing should be done in the units (if possible)
Lee Surprenant (Oct 30 2019 at 20:49):
what i've sort of been thinking (for changes effecting the persistence layer at least) is to have through/exhaustive unit tests
Lee Surprenant (Oct 30 2019 at 20:49):
shooting for good code coverage
Lee Surprenant (Oct 30 2019 at 20:49):
and then a smattered of end-to-end / integration tests to show things are working as expected in the cases we care most about
Lee Surprenant (Oct 30 2019 at 20:50):
but sometimes i waver on that and think "what really matters is the e2e behavior of the system, so maybe thats where we should be exhaustive"
Lee Surprenant (Oct 30 2019 at 20:50):
i go back and forth depending on the weather...
John Timm (Oct 30 2019 at 20:51):
just due to the nature of e2e tests and the setup and time required to run, etc. it's more difficult and not always practical to build something that is exhaustive
Lee Surprenant (Oct 30 2019 at 20:51):
also, our fhir-persistence tests aren't REALLY unit tests
Lee Surprenant (Oct 30 2019 at 20:51):
i think they're much closer to functional tests
Lee Surprenant (Oct 30 2019 at 20:52):
or, if they are unit tests, the unit under test is quite large
John Timm (Oct 30 2019 at 20:52):
whereas with units it is easier to be more exhaustive
John Timm (Oct 30 2019 at 20:52):
I don't think the PL tests are technically unit tests. in DSTU2 we called them "lightweight integration tests"
Lee Surprenant (Oct 30 2019 at 20:52):
yeah, that sounds about right
John Timm (Oct 30 2019 at 20:52):
they aren't e2e but they do require "integration" with a database backend
Lee Surprenant (Oct 30 2019 at 20:52):
so is that the spot to be exhaustive?
John Timm (Oct 30 2019 at 20:53):
for PL stuff, I think it's the best option
John Timm (Oct 30 2019 at 20:53):
hard to test the PL stuff without at least Derby at your disposal
Lee Surprenant (Oct 30 2019 at 20:54):
ok, thanks for the discussion. it makes me feel more confident to ask albert to add PL tests for testing this "duplicate search results" case
Last updated: Apr 12 2022 at 19:14 UTC
