Stream: bulk data
Topic: Cerner server
Michele Mottini (Sep 21 2018 at 14:33):
I am seeing duplicate Patient - first without any data and then complete:
Michele Mottini (Sep 21 2018 at 14:33):
{"resourceType":"Patient","id":"bba01432d6105631c9115683f52b0956f3a03b6c"}
{"resourceType":"Patient","id":"bba01432d6105631c9115683f52b0956f3a03b6c","name":[{"family":"Baeza","given":["Dudley*"]}],"birthDate":"2009-01-01","address":[{"line":["136 Lang Avenue"],"city":"Springfield","state":"22","postalCode":"34471"}]}
Dennis Patterson (Sep 21 2018 at 14:36):
You're absolutely right, Michele. This was found during the last connectathon and I reported it to the folks behind the data generation. I'll be focused on Backend Services support leading up to the event, but I'll be glad to start sifting through the patient files after that to clean up things like this. We'll definitely take any feedback on data issues.
Michele Mottini (Sep 21 2018 at 14:40):
OK, thanks. I'll work around it. I _think_ this should be considered an error and clients should flag it (...and try to keep going)
Dennis Patterson (Sep 21 2018 at 21:02):
For anyone hitting the Cerner server, we've taken down the previous instance and are in the process of deploying a separate server as we prep for the connectathon. Apologies for the downtime inconvenience; it's likely to be resolved early next week. One change with the separate server is that it should immediately be pointing to gzipped ndjson, for one thing. Will provide an update when I have additional information.
Dennis Patterson (Sep 24 2018 at 14:15):
The server is back up and should serve up gzipped ndjson
Karen Fairchild (Sep 24 2018 at 17:30):
@Dennis Patterson - for clarification - does Cerner also support a non-gzip ndjson file? We are expecting a ndjson file but not gzipped for the Connectathon. I recall earlier discussions on gzipped files, but if those are in the connectathon, how does one either request a gzip file or a non-gzip file when invoking the FHIR bulk data downloader?
Dennis Patterson (Sep 24 2018 at 17:56):
Based on discussion in https://chat.fhir.org/#narrow/stream/95-bulk-data/subject/Gzip.20compression.20on.20Cerner.20staging.20server, the server now currently only serves up gzipped contents. On the day of the connectathon, this will make downloads much faster. @Vladimir Ignatov does the bulk data downloader support gzipped contents? If not, could it?
Vladimir Ignatov (Sep 25 2018 at 14:09):
The SMART Bulk Data Server did not support gzip, but I found this is is a great idea so now it does. You enable that by sending "accept-encoding: gzip" header to the file download endpoint. The sample client was also updated and you can pass a "--gzip" option to set that header. It will also display download sizes before and after decompression and the total time spent.
Dennis Patterson (Sep 25 2018 at 14:11):
Fantastic. Thanks, @Vladimir Ignatov ! cc: @Karen Fairchild
Karen Fairchild (Sep 25 2018 at 15:50):
@Dennis Patterson - FYI we updated our client now to be able to accept gzip files and what had taken 20 minutes to get data from your Cerner server now only takes 3 minutes.
Dennis Patterson (Sep 26 2018 at 17:23):
Going to be out of office the rest of the day, but I'm planning to push out a token endpoint for the Cerner server in the morning (central time). If anybody has a client that they want registered, feel free to reach out to me and I'll register your jku (or jwks) so we can start trying to integrate.
Michele Mottini (Sep 26 2018 at 20:53):
Great. Our client JWKS url is https://demo.careevolution.com/CNjwtcareevolutioncom.json (I added it to the spreadhseet as well)
Dennis Patterson (Sep 27 2018 at 16:03):
Deployed the token endpoint and added it to the spreadsheet
Karen Fairchild (Sep 28 2018 at 14:37):
@Dennis Patterson - when we get your gzip files, our standard gzip utilities (python, linux, windows/7zip) don't seem to be able to recognize these as gzip files (even if we give them a gz suffix). We've tried multiple platforms and tools but not able to be recognized as gzip. How are you creating your gzip files?
Michele Mottini (Sep 28 2018 at 14:40):
Using .NET GZipStream worked for me
Michele Mottini (Sep 28 2018 at 14:42):
(from code ... did not try decompress utilities)
David Hay (Sep 28 2018 at 15:21):
@Michele Mottini is this something worth adding to the conMan server tab as well?
Michele Mottini (Sep 28 2018 at 15:24):
Sorry, what do you mean by 'this' ?
Josh Mandel (Sep 28 2018 at 15:25):
I'm not sure if we're planning on using conman (though I'm happy to; I also set up a spreadsheet)
Josh Mandel (Sep 28 2018 at 15:26):
shared spreadsheet to list participants and the scenarios they're working on
Dennis Patterson (Sep 28 2018 at 18:33):
@Karen Fairchild are you still having this issue? I'm able to retrieve an example via curl --compressed https://d3mme9o1hh15io.cloudfront.net/0000.Patient.ndjsonor use gunzip once the compressed version is retrieved e.g. curl https://d3mme9o1hh15io.cloudfront.net/0000.Patient.ndjson | gunzip
Karen Fairchild (Sep 28 2018 at 18:42):
I’m traveling to Baltimore now but will give that a try when I get to the hotel tonight. The first 2 bytes of the file didn’t seem to match up with ISO standards I looked at for fIle sIgnatures. Maybe we can discuss what standard to use for gzip files so it’s highly portable. Thanks.
Dennis Patterson (Sep 28 2018 at 18:47):
I used gzip on my Mac to compress them, but I'll be interested to hear what you find. Glad that Michele didn't have issue. Please, anybody else having issues (or not), feel free to chime in. You could also let me know what language/library you're using to try to decompress them beyond the command-line utilities you're troubleshooting with
Karen Fairchild (Sep 28 2018 at 18:54):
Python libraries
Karen Fairchild (Sep 28 2018 at 18:55):
But we also tried some command line utilities on Mac, windows, Ubuntu Linux.
Ryan Brush (Sep 28 2018 at 18:58):
FWIW, I happened to have a REPL open when I saw this and the following worked in Python. The Python requests library automatically unzipped, so I saw the content as expected.
import requests
response = requests.get('https://d3mme9o1hh15io.cloudfront.net/0000.Patient.ndjson')
response.content
Karen Fairchild (Sep 28 2018 at 20:13):
Thanks. We did some more testing.
This works
curl https://d3mme9o1hh15io.cloudfront.net/0000.Patient.ndjson > outfile.gz
Gunzip outfile.gz
And this works
curl https://d3mme9o1hh15io.cloudfront.net/0000.Patient.ndjson > outfile
Gunzip outfile
However, if we save the file has an NDJSON file directly, it seems to corrupt the file:
This does not work
curl https://d3mme9o1hh15io.cloudfront.net/0000.Patient.ndjson > outfile.ndjson
gunzip outfile.ndjson
➢ gunzip: outfile.ndjson: unknown suffix -- ignored
Karen Fairchild (Sep 28 2018 at 20:15):
Perhaps we can discuss more tomorrow at the connectathon....
Dennis Patterson (Sep 29 2018 at 01:58):
Sure, yeah gzip just expects a particular suffix. If you gunzip via python I'm curious if it'd even care. We can chat tomorrow
Michele Mottini (Oct 05 2018 at 15:30):
Apparently something broke with authentication: trying to get access token fail with invalid_request - Signature nbf has not been reached
Michele Mottini (Oct 05 2018 at 15:31):
(The nbf looks OK)
Michele Mottini (Oct 05 2018 at 15:31):
correlation ID: 4dd590ac-ee39-4301-b0fe-b77d8ac149c5
Michele Mottini (Oct 05 2018 at 15:32):
If I remove the nbf I can get an access token - but apparently an invalid one:
Michele Mottini (Oct 05 2018 at 15:32):
error="invalid_token", error_description="Invalid issuer. Expected https://fhir-ehr.stagingcerner.com/r4/a758f80e-aa74-4118-80aa-98cc75846c76/token, received https://fhir-ehr.stagingcerner.com/r4/0b8a0111-e8e6-4c26-a91c-5069cbc6b1ca/token"
Michele Mottini (Oct 05 2018 at 15:32):
request id 6a9fa245-5fa3-4128-a993-356e0a347ac2
Michele Mottini (Oct 05 2018 at 15:33):
(also: the invalid token errors above are inside the WWW-Authenticate header - should probably be in the body instead, as an OperationOutcome
Jenni Syed (Oct 05 2018 at 17:20):
I think @Dennis Patterson is traveling today, so it may take some time to troubleshoot.
Jenni Syed (Oct 05 2018 at 17:21):
The body part is an interesting question, since OAuth 2 defines this response here: https://tools.ietf.org/html/rfc6749#section-5.2
Michele Mottini (Oct 05 2018 at 17:28):
No problem - not urgent
Michele Mottini (Oct 05 2018 at 17:29):
Clarification: I get that WWW-Authenticate header in the response to the $export request, not to the token request
Dennis Patterson (Oct 08 2018 at 14:38):
@Michele Mottini Your first issue is related to clock skew because our server saw the nbf time as a future value. I have a todo to account for this, though I'll mention it's Cerner Health Conference (CHC) week so no immediate plans.
For the second issue, it looks like you're using two separate base urls for creating the token and requesting the export. (You may have been doing this before, but I did work at the connectathon to issue JWTs instead of mock tokens.) Our logs show you're using https://fhir-ehr.stagingcerner.com/r4/0b8a0111-e8e6-4c26-a91c-5069cbc6b1ca/token and https://fhir-ehr.stagingcerner.com/r4/a758f80e-aa74-4118-80aa-98cc75846c76/Patient/$export . Note that the uuid in those urls is different. Because our data is still mocked, it doesn't matter which of the two base urls you use, but they must be consistent.
Michele Mottini (Oct 08 2018 at 15:02):
Got it, thanks Dennis
Michele Mottini (Nov 08 2018 at 14:14):
@Dennis Patterson Getting the Signature nbf has not been reached problem - it was on and off but not it seems to be happening always (maybe the click change last Sunday?)
Michele Mottini (Nov 16 2018 at 04:44):
Something apparently broke: now getting
Michele Mottini (Nov 16 2018 at 04:45):
Unable to get an access token: invalid_request - <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"> <TITLE>ERROR: The request could not be satisfied</TITLE> </HEAD><BODY> <H1>502 ERROR</H1> <H2>The request could not be satisfied.</H2> <HR noshade size="1px"> The origin closed the connection. <BR clear="all"/> If you received this error while trying to use an app or access a website, please contact the provider or website owner for assistance. <BR clear="all"> If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by following steps in the CloudFront documentation (<A href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-502-bad-gateway.html">https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-502-bad-gateway.html</A>). <BR clear="all"> <HR noshade size="1px"> <PRE> Generated by cloudfront (CloudFront) Request ID: 9BAjBda3RxYJ0jXZEVAb999gQA_hgKOVhIO9N22B1Mq9XrMVv8IiqQ== </PRE> <ADDRESS> </ADDRESS> </BODY></HTML>
Dennis Patterson (Dec 05 2018 at 16:20):
@Michele Mottini apologies for the delay. We are looking at what's going on. To clarify, I assume you're using the Care Evolution client id? (We found that the cdn.rawgit.com service we were using to access the sample Bulk Data spec jwks has been deprecated, so wanted to confirm which client id)
Michele Mottini (Dec 05 2018 at 16:52):
No problem
Michele Mottini (Dec 05 2018 at 16:52):
Yes, CareEvolution - e91fff6f-4d78-4b0a-b5a6-6fa04c23a10c
Dennis Patterson (Dec 05 2018 at 20:55):
@Michele Mottini I addressed some issues that had arisen since the last Connectathon. Can you give it another shot?
Michele Mottini (Dec 05 2018 at 21:05):
No...fails trying to get the token:
Michele Mottini (Dec 05 2018 at 21:05):
invalid_request - <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"> <TITLE>ERROR: The request could not be satisfied</TITLE> </HEAD><BODY> <H1>502 ERROR</H1> <H2>The request could not be satisfied.</H2> <HR noshade size="1px"> The origin closed the connection. <BR clear="all"/> If you received this error while trying to use an app or access a website, please contact the provider or website owner for assistance. <BR clear="all"> If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by following steps in the CloudFront documentation (<A href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-502-bad-gateway.html">https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-502-bad-gateway.html</A>). <BR clear="all"> <HR noshade size="1px"> <PRE> Generated by cloudfront (CloudFront) Request ID: zgmFNGGotOawHDg4hKjBYmOjQK2AqVTAHPxiNkBd5ld5yPqVMlN0TQ== </PRE> <ADDRESS> </ADDRESS> </BODY></HTML>
Dennis Patterson (Dec 05 2018 at 21:20):
Can you verify what url you're hitting?
Michele Mottini (Dec 05 2018 at 21:24):
https://fhir-ehr.stagingcerner.com/r4/a758f80e-aa74-4118-80aa-98cc75846c76/token
Dennis Patterson (Dec 05 2018 at 21:33):
Hmm it's working for me.. I'll send you a private message and we can debug
Bradley Strecker (Jan 04 2019 at 21:47):
The Cerner Server has a new home!
- https://fhir-open.stagingcerner.com/beta/a758f80e-aa74-4118-80aa-98cc75846c76
- https://fhir-ehr.stagingcerner.com/beta/a758f80e-aa74-4118-80aa-98cc75846c76
- https://fhir-myrecord.stagingcerner.com/beta/a758f80e-aa74-4118-80aa-98cc75846c76
Michele Mottini (Jan 10 2019 at 22:49):
clients need to be registered again?
Michele Mottini (Jan 10 2019 at 22:52):
(authentication request fails with 401, empty body and WWW-Authenticate header containing Bearer realm="Cerner Millennium", error="invalid_token", error_description="The request has not been applied because it lacks valid authentication credentials for the target resource.)
Michele Mottini (Jan 10 2019 at 22:52):
(that looks like what should be in the body)
Bradley Strecker (Jan 11 2019 at 14:29):
We've moved around some of the client registration logic to make it easier to add registrations. Yours should have been carried over though. Do you happen to have an X-Request-Id response header I could used to try to track down the issue?
Michele Mottini (Jan 11 2019 at 19:56):
Here is request / response:
Michele Mottini (Jan 11 2019 at 19:56):
Michele Mottini (Jan 11 2019 at 19:56):
<LogInfo xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Time>1/10/2019 5:48:11 PM -05:00</Time>
<UserID xsi:nil="true" />
<SessionID xsi:nil="true" />
<ApplicationID>0</ApplicationID>
<Method>POST</Method>
<Uri>https://fhir-ehr.stagingcerner.com/r4/a758f80e-aa74-4118-80aa-98cc75846c76/token</Uri>
<RequestHeaders>
<Header Name="Accept">
<Value>application/json</Value>
</Header>
<Header Name="Content-Type">
<Value>application/x-www-form-urlencoded</Value>
</Header>
<Header Name="Content-Length">
<Value>943</Value>
</Header>
</RequestHeaders>
<RequestBody>grant_type=client_credentials&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer&client_assertion=eyJ0eXAiOiJKV1QiLCJrdHkiOiJSU0EiLCJhbGciOiJSUzM4NCIsImtpZCI6IjY2NjJGMkY4MUMxQTQ1QTcxNTNEREI3NUM4MjJCNjY3M0U2RTdBMjkifQ.eyJpc3MiOiJlOTFmZmY2Zi00ZDc4LTRiMGEtYjVhNi02ZmEwNGMyM2ExMGMiLCJzdWIiOiJlOTFmZmY2Zi00ZDc4LTRiMGEtYjVhNi02ZmEwNGMyM2ExMGMiLCJhdWQiOiJodHRwczovL2ZoaXItZWhyLnN0YWdpbmdjZXJuZXIuY29tL3I0L2E3NThmODBlLWFhNzQtNDExOC04MGFhLTk4Y2M3NTg0NmM3Ni90b2tlbiIsImp0aSI6ImFjYWQ1NjM4LTJkZmUtNDAwYy05MTIxLWE4M2I0NjBjNWEzZCIsImV4cCI6MTU0NzE2MDc5MH0.EpFw0pSjOzoiPli75wvMCIJwuY0MlWqpRJQArO6vTxNrN23V-yxGcm8i76azdcuD49_rbJyra1XqX3q5PeKsswgCcEZe0Rn2RYlXuPBuhmKX_SlS7W-pZOrtiErYSzKNx4cJwh2NBO3NqiP5VHXgpIfpYhpmyWRfLP9S8b9Q7F3qITCrHjaxj61SBnXKT1bSrtC41TYD_Y-7bjt9myTbqp8Dcz7W3TjVSYREoQFH2JZFKQ7VMkldDhDVpZCaZH9eF5gvObsR1tnoNEc6gOV85BhrK0Baz1Chu3F6FcUmc2Xs1CYt9DI0-Zj9FYy6JwTnZ5OHWQ9X-zCPavEhfbsbZw&scope=system%2F%2A.read</RequestBody>
<ResponseStatusCode>401</ResponseStatusCode>
<ResponseHeaders>
<Header Name="Connection">
<Value>keep-alive</Value>
</Header>
<Header Name="x-amzn-RequestId">
<Value>d36dc663-1529-11e9-a8e0-65c0231b79c6</Value>
</Header>
<Header Name="x-amzn-ErrorType">
<Value>UnauthorizedException</Value>
</Header>
<Header Name="x-amz-apigw-id">
<Value>TTzEEFcKvHcF-mg=</Value>
</Header>
<Header Name="X-Cache">
<Value>Error from cloudfront</Value>
</Header>
<Header Name="X-Amz-Cf-Id">
<Value>8Rp0T_5LWeP4jQEES39QWW5b86b8MJgYRDFL8jHTd-hvRa5cM2DFHA==</Value>
</Header>
<Header Name="Date">
<Value>Thu, 10 Jan 2019 22:48:19 GMT</Value>
</Header>
<Header Name="Via">
<Value>1.1 6acd4ebf1a0179dd8e00eb58764e453a.cloudfront.net (CloudFront)</Value>
</Header>
<Header Name="WWW-Authenticate">
<Value>Bearer realm="Cerner Millennium", error="invalid_token", error_description="The request has not been applied because it lacks valid authentication credentials for the target resource."</Value>
</Header>
</ResponseHeaders>
<ResponseBody> </ResponseBody>
</LogInfo>
Michele Mottini (Jan 11 2019 at 19:56):
Michele Mottini (Jan 11 2019 at 19:57):
(In our log format - apologies for that)
Michele Mottini (Jan 11 2019 at 21:45):
Found the problem - I updated the FHIR end point but not the token end point in our configuration - now works
Michele Mottini (Jan 11 2019 at 21:45):
apologies
Bradley Strecker (Jan 12 2019 at 14:21):
Yup, that'll do it. Explains why I wasn't able to find it in the logs :)
Michele Mottini (Jan 12 2019 at 16:05):
Already spoke about it with Bradley, but posting it for reference: it would be nice to have a smaller test data set - the current one is so big that we cannot process it in any reasonable time - just importing the patients take hours, so I can never get to the rest of clinical data
Michele Mottini (Jan 12 2019 at 16:06):
(we are inserting the data in our db, and that takes time for 350K patient)
Last updated: Apr 12 2022 at 19:14 UTC
