FHIR Chat · Search via POST · implementers

Stream: implementers

Topic: Search via POST

Lee Surprenant (Jan 05 2021 at 16:16):

https://www.hl7.org/fhir/search.html mentions that search can be invoked via a POST to [base]/[type]/_search.

However, in the swagger definitions being generated for the various implementation guides, I don't see this endpoint listed. Are servers required to support this search variant? Should I open an issue to have this endpoint added to all the generated swagger?

Lloyd McKenzie (Jan 05 2021 at 16:18):

@Grahame Grieve

Lee Surprenant (Jan 07 2021 at 14:16):

This seems important to me because if its not clear then not all servers will support it and then clients can't depend on it. Any thoughts?

Eric Haas (Jan 07 2021 at 17:18):

in US Core we specify the search requirements in the CapabilityStatements, I would expect that would drive the swagger definitions too.

Eric Haas (Jan 07 2021 at 17:19):

Are servers required to support this search variant?

short answer is no.

Robert Scanlon (Jan 07 2021 at 18:58):

It is a shall requirement in the base spec though:

Because of the way that some user agents and proxies treat GET and POST requests, in addition to the get based search method above, servers that support search SHALL also support a POST based search

Lee Surprenant (Jan 25 2021 at 12:42):

The conflict above from base spec to IG seems to highlight the need for clarification here and so I went ahead and opened https://jira.hl7.org/browse/FHIR-30577

Lee Surprenant (Jan 25 2021 at 12:43):

@Grahame Grieve is there a better spot to leave the "add the _search endpoints to the generated openapi" request?

Grahame Grieve (Feb 02 2021 at 21:35):

I don't think so

Lloyd McKenzie (Mar 22 2021 at 19:29):

It wasn't Eric's intention to indicate that US-core relaxes the base spec requirement - and support for POST to _search is mandatory. (i.e. it must be supported for any US core implementation that supports search at all).

Eric Haas (Mar 22 2021 at 20:48):

see FHIR#31585

Yunwei Wang (Mar 25 2021 at 19:41):

@Eric Haas Cannot open the reference guidance in your ticket

Add guidance on "POST ... _search" in general guidance here: file:///Users/ehaas/Documents/FHIR/US-Core-R4/output/general-guidance.html#search-syntax

Eric Haas (Mar 25 2021 at 20:46):

http://build.fhir.org/ig/HL7/US-Core/general-guidance.html#search-syntax

Eric Haas (Mar 25 2021 at 20:48):

basically going to say server reuqired to support the POST syntax per the FHIR spec but we are only going to use the GET syntax in the guide.

Ayush Shrivastava (Mar 29 2021 at 12:02):

However, in the swagger definitions being generated for the various implementation guides, I don't see this endpoint listed. Are servers required to support this search variant? Should I open an issue to have this endpoint added to all the generated swagger?

Can anyone please point me to the swagger definition of endpoints

Mona O (Apr 13 2021 at 17:08):

Is an implementer conformant if only the Search via POST is supported or are all implementers required to support the GET method for searches?

Michele Mottini (Apr 13 2021 at 17:26):

You must support GET. I don't know if that's actually written somewhere but it is what most / all clients would expect

Mona O (Apr 13 2021 at 18:38):

Michele Mottini said:

You must support GET. I don't know if that's actually written somewhere but it is what most / all clients would expect

Are there any exceptions to this rule? Due to PHI in the URL our security dept. does not allow us to support GET.

Eric Haas (Apr 13 2021 at 18:40):

here is the text: http://build.fhir.org/http.html#search

.... in addition to the get based search method above, servers that support search SHALL also support a POST based search:
(emphasis mine)

Jose Costa Teixeira (Apr 14 2021 at 06:16):

I did not read this as "all servers shall support GET search". in fact, we discussed excluding GET search.

Michele Mottini (Apr 14 2021 at 13:32):

'...in addition.... SHALL also....' seems pretty clearly requiring GET to me

Mona O (Apr 14 2021 at 15:30):

Am I'm hearing that companies are forced to follow conformance over their own security policies? Are there no exceptions?

Paul Church (Apr 14 2021 at 16:31):

Forced by whom? Your legal counsel can advise you of the risks.

In practice, most clients use GET and every server I've ever seen supports GET.

Jose Costa Teixeira (Apr 14 2021 at 20:01):

We decided that all searches should be by POST. does this mean we are not FHIR conformant?

Jose Costa Teixeira (Apr 14 2021 at 20:02):

"servers shall also support" does not mean "implementations must also enable". I may be wrong here

Jose Costa Teixeira (Apr 14 2021 at 20:11):

is there a reason why all implementations would need to support GET search?

Daniel Venton (Apr 14 2021 at 20:15):

Servers supporting the usage of GET does not require or even recommend the usage of the GET method. Choice of which method to use is up to the client application. If the client chooses to use the GET method then they are the ones placing the potentially restricted information in the URL.

Every resource in USCore says, "SHALL support searching using the combination of the patient and intent search parameters:
including support for composite OR search on intent (e.g.intent={system|}[code],{system|}[code],...)
GET [base]/MedicationRequest?patient=[reference]&intent=order,plan".

Some people will read that and say, must support GET. Other's will read that and say that the usage of GET in the example doesn't mandate the usage of the GET method or the requirement to support the GET method. Some client applications won't even attempt to use the POST because they "know" all servers SHALL support GET.

Jose Costa Teixeira (Apr 14 2021 at 20:15):

Michele Mottini said:

'...in addition.... SHALL also....' seems pretty clearly requiring GET to me

I think the sentence is not that unequivocal. A implies B does not mean A is true, or that B implies A, right?

Tony ThaDee (Apr 14 2021 at 20:17):

hey, if I'm using HAPI or some FHIR server to send data to our EMR, do EMRs like Athena provide a different means of passing data to/from them (perhaps with more access or benefits ) ASIDE from their usual API? I guess I'm wondering if a FHIR server would exchange data with an EMR via their API just like any other program or if there's another way.

Daniel Venton (Apr 14 2021 at 20:23):

Also, the prohibition against using the GET method and requiring the usage of only the POST method, doesn't prevent the client app from putting the values on the URL, in exactly the same manner as they would using the GET method. HAPI will even conveniently merge the two sets of parameters together for you (URL + post body).

Tony ThaDee (Apr 14 2021 at 20:23):

I know the FHIR server wouldn't do the data exchange, but I mean to ask, do EMRs provide teams/projects that utilize certain technologies that utilize something like a FHIR server greater or special access that anyone here is aware of.

Daniel Venton (Apr 14 2021 at 20:24):

@Tony ThaDee not to hijack this thread, but every implementation is going to be different. Some will be more features using FHIR others will be more features using custom API.

Tony ThaDee (Apr 14 2021 at 20:27):

didn't mean to distract from the thread.

Jose Costa Teixeira (Apr 14 2021 at 20:27):

getting back to the topic:

'...in addition.... SHALL also....' seems pretty clearly requiring GET to me

I think the sentence is not that unequivocal. A implies B does not mean A is true, or that B implies A, right?

Paul Church (Apr 14 2021 at 20:34):

This wordsmithing seems confusing to me. The definition of search is a GET operation. The spec doesn't say you SHALL support it - you don't have to support all API methods, just declare what you do support in your capability statement. But if you support search, this is the definition. If you don't support GET, you don't support search. POST is the additional thing, which the spec goes out of its way to say is required if you support search.

Look at any other operation, like create: "The create interaction creates a new resource in a server-assigned location [...] The create interaction is performed by an HTTP POST command" - the word SHALL does not appear here, nor does it need to.

Jose Costa Teixeira (Apr 14 2021 at 20:36):

is the definition of Search a GET operation? I understood that GET is one way to implement search

Oliver Egger (Apr 14 2021 at 20:38):

I think https://www.hl7.org/fhir/http.html#search is pretty clear?

1. The interaction can be performed by several different HTTP commands.
2. Because of the way that some user agents and proxies treat GET and POST requests, in addition to the get based search method above, servers that support search SHALL also support a POST based search:

Jose Costa Teixeira (Apr 14 2021 at 20:38):

This interaction searches a set of resources based on some filter criteria. The interaction can be performed by several different HTTP commands."
I don't think this unambiguously mean this is a GET

Jose Costa Teixeira (Apr 14 2021 at 20:39):

Right @Oliver Egger. This reads indeed that servers shall support both.

Daniel Venton (Apr 14 2021 at 20:40):

I read it the same same way, SHALL support GET and POST.

-This allows the client application to determine if the usage of a GET method is problematic for them.
-No matter which method is used/allowed, the http logs have to be treated as protected. This probably tells all clients that they SHOULD use only post, but they don't have to.

Jose Costa Teixeira (Apr 14 2021 at 20:42):

Jose Costa Teixeira said:

I did not read this as "all servers shall support GET search". in fact, we discussed excluding GET search.

This sentence is not correct.

What I should have said is that

I didn't read this as "all implementations shall support GET search"

Jose Costa Teixeira (Apr 14 2021 at 20:43):

Daniel Venton said:

I read it the same same way, SHALL support GET and POST.

-This allows the client application to determine if the usage of a GET method is problematic for them.
-No matter which method is used/allowed, the http logs have to be treated as protected. This probably tells all clients that they SHOULD use only post, but they don't have to.

Right. And I'd say that some implementations may disable GET. If there is a reason that an IG cannot exclude GET search, I'd like to see it so that I can transmit to the implementers

Jens Villadsen (Apr 14 2021 at 20:43):

For this very reason, I've built FHIR servers that disallowed certain searches with GET searches depending on the parameters whereas the search can always safely be performed using POST

Paul Church (Apr 14 2021 at 20:46):

I agree that no client is under an obligation to use GET, and an IG can say "don't use GET". I was just arguing that servers are obligated to support both.

Daniel Venton (Apr 14 2021 at 20:49):

Jens Villadsen said:

For this very reason, I've built FHIR servers that disallowed certain searches with GET searches depending on the parameters whereas the search can always safely be performed using POST

If you reject the GET method, for any reason, then the damage has already been done. That URL with the PHI/PII has already been written to your logs and the log of every router between you and the client. Perhaps this will train clients to not use the GET method, but I suspect your capability statement doesn't say, 'POST parameters allowed X, GET parameters allowed X-n"

Paul Church (Apr 14 2021 at 20:52):

This concern seems a bit archaic to me - we know how to do TLS now. The URL is not getting written to the log of any router between you and the client.

Jose Costa Teixeira (Apr 14 2021 at 21:07):

"the damage has already ben done" - not a lot.
The client can try once but it will not do all searches using GET. So I hope that a client, even if it forgot to undergo the testing phase to see what is allowed, can try one search but after that search it will realize that it doesn't work. Stubborn clients may give away a few searches, but even then I don't think the first search a client would do would contain sensitive parameters.

Paul Church (Apr 14 2021 at 21:09):

I encourage my customers to do GET searches. Seriously. We designed for the presence of sensitive data in the query parameters and there's no "damage".

Lloyd McKenzie (Apr 16 2021 at 16:42):

The intention was very much that servers would be allowed to NOT support GET - because there are environments that have architectural requirements (well-founded or not) that prohibit using GET. We don't want to prevent those systems from being FHIR conformant. The only mechanism you can count on ALL conformant systems supporting is POST.

Lloyd McKenzie (Apr 16 2021 at 16:42):

That said, all systems SHOULD support GET.

Daniel Venton (Apr 16 2021 at 17:42):

Then the IG should explicitly state SHALL support search via POST method, SHOULD support search via GET method. The way it's currently worded SHALL also support POST implies that there is another SHALL. It doesn't read right if it's "MAY support GET and SHALL also support POST." If you don't support GET then you aren't also doing anything.

Lloyd McKenzie (Apr 16 2021 at 17:46):

If you feel the current wording needs clarification, please submit a change request.

John Moehrke (Apr 16 2021 at 17:57):

some will only support GET, some will only support POST, some will support both... right? I don't think the core should declare either as manditory.

Lloyd McKenzie (Apr 16 2021 at 17:57):

The core spec does declare POST as mandatory.

Lloyd McKenzie (Apr 16 2021 at 17:57):

And I don't think we can relax that at this point...

John Moehrke (Apr 16 2021 at 18:03):

Lloyd McKenzie said:

The intention was very much that servers would be allowed to NOT support GET - because there are environments that have architectural requirements (well-founded or not) that prohibit using GET. We don't want to prevent those systems from being FHIR conformant. The only mechanism you can count on ALL conformant systems supporting is POST.

I was racting to this statement... I don't think you can "count" on "ALL conformant systems supporting is POST"

Lloyd McKenzie (Apr 16 2021 at 18:03):

If it doesn't support POST, it's not a conformant system.

John Moehrke (Apr 16 2021 at 18:06):

ah, that is news to me. where is that stated?

John Moehrke (Apr 16 2021 at 18:17):

search page seems to speak to GET and POST equally http://hl7.org/fhir/search.html

John Moehrke (Apr 16 2021 at 18:20):

well, if equal is (POST = 2 mentions) vs (GET =124 mentions)

Paul Church (Apr 16 2021 at 18:21):

It baffles me that the intent would be for GET to be optional. The entire search spec is written around GET.

John Moehrke (Apr 16 2021 at 18:22):

I find on the http page is a hint about POST being mandatory - http://build.fhir.org/http.html#search

John Moehrke (Apr 16 2021 at 18:22):

Because of the way that some user agents and proxies treat GET and POST requests, in addition to the get based search method above, servers that support search SHALL also support a POST based search:

Lloyd McKenzie (Apr 16 2021 at 18:23):

I don't see how an explicit SHALL statement is a 'hint'?

John Moehrke (Apr 16 2021 at 18:23):

does this mean that an Implementation Guide can not forbid POST and mandate GET?

Lloyd McKenzie (Apr 16 2021 at 18:23):

We don't want GET to be optional, but in some environments, it's banned. It would be non-conformant to ban POST.

John Moehrke (Apr 16 2021 at 18:23):

I say hint.. because it is passive voice, and at the very end of a oddly phrased statement

John Moehrke (Apr 16 2021 at 18:25):

Lloyd McKenzie said:

We don't want GET to be optional, but in some environments, it's banned. It would be non-conformant to ban POST.

I think this is wrong. FHIR core should not have taken a position.

John Moehrke (Apr 16 2021 at 18:28):

not all uses of FHIR are PHI (Personal health information). Many people know how to properly configure their web-servers to not log GET parameters.

Vassil Peytchev (Apr 16 2021 at 18:54):

POST search is not used just for security/privacy reasons. The practical limits on a query string are orders of magnitude lower than the limits on a POST body. It makes good sense to have POST support required for servers, while GET is strongly recommended. "some servers will have this, other servers will have that" doesn't cut it in this case.

Cooper Thompson (Apr 16 2021 at 19:06):

(deleted)

Cooper Thompson (Apr 16 2021 at 19:09):

John Moehrke said:

not all uses of FHIR are PHI (Personal health information). Many people know how to properly configure their web-servers to not log GET parameters.

"Many" is very relative... Be careful about bias here. People in your circle probably do know about this, but you're a security guy in a security-conscious group of experts.

John Moehrke (Apr 16 2021 at 19:30):

I am well aware of my credentials. But failure to configure your server is poor security.

John Moehrke (Apr 16 2021 at 19:32):

and those that can't protect logs, should not be given the responsibility to protect the data.... it is such a rediculous fallacy that a server can have PHI but can't protect http server log files.

John Moehrke (Apr 16 2021 at 19:34):

the size limits of GET vs POST are far more useful discussion.. . I don't think this is a reason to mandate POST, but it is a consideration that should be voiced.

John Moehrke (Apr 16 2021 at 19:35):

my biggest complain is that I am being told that it is non-conformant to FHIR for me to have an implementation guide that focuses on GET and does not include POST.

Vassil Peytchev (Apr 16 2021 at 19:39):

focuses on GET and does not include POST

or

focuses on GET and prohibits POST

?

There is a difference. If the latter, what would be the reason to prohibit POST searches?

John Moehrke (Apr 16 2021 at 19:42):

I agree that I was tactful in my words... but I don't think there is a difference. as absence of a statement is often seen as a strong implication.

John Moehrke (Apr 16 2021 at 19:43):

plus, I was about to add a shall-not to an implementation guide... as asked for by that community... but now need to reverse direction and force that community to do POST searches that they don't want to be forced to do.

Vassil Peytchev (Apr 16 2021 at 19:45):

No one is forcing them to do POST searches. The server must support POST searches. Why would an implementation guide promote incomplete server implementations?

John Moehrke (Apr 16 2021 at 19:48):

that is a confusing message... I am hearing that any search must be supported by way of POST search, not just by GET... right? Thus I must tell the servers that they must implement POST search, not just GET search... right?

John Moehrke (Apr 16 2021 at 19:49):

I understand that this does not mean clients must be able to query by POST... but it does mean that servers must implement POST search, even when they will never be asked to do it in real-life.

Vassil Peytchev (Apr 16 2021 at 19:56):

Yes, servers must support POST search - this is a basic property of a FHIR server.

A FHIR implementation guide that describes some type of use of the FHIR RESTful API expects that there will be FHIR servers to support that API. We don't want to promote the illusion that the implementation of a FHIR server is trivial, and does not require rigorous software engineering. The existence of reference implementations and various products already satisfies that requirement by default.

If someone needs to create a FHIR server from scratch, the ability to support search via POST doesn't seem to be such an onerous requirements compared to the overall requirements a FHIR server needs to meet.

John Moehrke (Apr 16 2021 at 20:07):

It just seems odd to me that all the documentation is on GET and I am confident that all the testing is using GET... yet we are comfortable that POST is what we mandate.

Lloyd McKenzie (Apr 16 2021 at 20:10):

POST is the common denominator. We can mandate support for that everywhere (and have). We can't do the same for GET. And whether anyone feels we ought to or not, we did, and given that the requirement is normative, it's not something we can reasonably revisit - nor have I heard a strong reason why we should try.

Vassil Peytchev (Apr 16 2021 at 20:14):

John does make a good point though - do certification testing actually test this? I will ask in the Inferno stream...

Oliver Egger (Apr 16 2021 at 20:15):

According to our experience in real life it is also a POST for clients due to organisational regulations. We had this discussion already for IHE MHD implementations in a project (MHD implies GET for search queries). Since IHE MHD is in public comment phase added just a comment to clarify, thanks for that interesting discussion here!

Robert Scanlon (Apr 16 2021 at 20:23):

John does make a good point though - do certification testing actually test this? I will ask in the Inferno stream...

No. We are well aware of this though it will in by the next release. Anecdotally, if an IG doesn't (re)state that POST is required, this requirement will be missed. FHIR is a big spec, and much of it is optional depending on use case.

John Moehrke (Apr 16 2021 at 20:43):

Oliver Egger said:

According to our experience in real life it is also a POST for clients due to organisational regulations. We had this discussion already for IHE MHD implementations in a project (MHD implies GET for search queries). Since IHE MHD is in public comment phase added just a comment to clarify, thanks for that interesting discussion here!

yup, I noticed... drat

John Moehrke (Apr 16 2021 at 20:45):

I am still worried that we are driving this use of POST for search purely academically... adding testing is not changing that academic point... what we need to understand is.. clients support for POST search..

John Moehrke (Apr 16 2021 at 20:49):

I have had many discussions about the security fallacy... not helpful that it continues. (I am not saying it is 100% false, but rather that it is a configuration failure by incompetent security. Either protect your logs, or configure your auditing properly. I prefer you protect your logs).

Lloyd McKenzie (Apr 16 2021 at 20:51):

Right. But that's only one of the reasons. The length limits of URLs is a more valid reason

Eric Haas (Apr 16 2021 at 22:38):

Robert Scanlon said:

John does make a good point though - do certification testing actually test this? I will ask in the Inferno stream...

No. We are well aware of this though it will in by the next release. Anecdotally, if an IG doesn't (re)state that POST is required, this requirement will be missed. FHIR is a big spec, and much of it is optional depending on use case.

See this update for the next version of US Core: http://build.fhir.org/ig/HL7/US-Core/general-guidance.html#search-syntax. Which I now realized misinterprets the shoddy conformance language in the FHIR spec. So based on this conversation it will have to say SHALL POST and SHOULD GET (we will need to clarify the CapabilityStatement narrative too).

Eric Haas (Apr 16 2021 at 23:03):

Lloyd McKenzie said:

If you feel the current wording needs clarification, please submit a change request.

FHIR#31912

Robert Scanlon (Apr 17 2021 at 15:12):

@Eric Haas said:

So based on this conversation it will have to say SHALL POST and SHOULD GET (we will need to clarify the CapabilityStatement narrative too).

This feels more like a substantive change to US Core requirements from v3.1.1 to v3.2.0, and not just a clarification. I know this wasn't necessarily your intent, but the language in the search syntax section, and the hundreds of GET-based search examples throughout the guide (without POST being mentioned anywhere), very strongly implies that search by GET is a requirement. I absolutely do not think impacts to test tooling should drive your decision here, but to illustrate one downstream effect of this change, we will have to alter all of our tests to no longer ever rely on search by GET working, because from our perspective it is no longer required. I suspect most clients would need to do the same.

Lloyd McKenzie (Apr 17 2021 at 17:08):

US Core is free to mandate support for GET too. The key thing is that it can't loosen the mandate to support POST that's already in the core spec.

Robert Scanlon (Apr 17 2021 at 19:04):

Or mandate support for search by GET for only the specific searches required by the US Core CapabilityStatement, but make no statement about other searches that may be supported by the API. Then you can objectively look at the finite set of searches and ask the question: "do these searches realistically need more characters than what you can fit in a GET request?" and maybe "do the search parameters themselves constitute PHI?"

Eric Haas (Apr 17 2021 at 20:00):

@Robert Scanlon my concern was that in order to comply with the base spec we would need to add this language and I agree what you proposed is a better solution. But I have concerns that implementers are going to push back and say that they only support POST. But I imagine that pushback will be experienced by your team first.

Robert Scanlon (Apr 19 2021 at 12:45):

@Eric Haas -- understood, adding constraints on top of base FHIR (in this case, search by GET is required, not optional) needs to be done intentionally and with an understanding of why the flexibility in the base spec isn't needed for the use cases that an IG applies to. And if you don't feel like you have that yet, then you shouldn't have that constraint. I do think that you can make a good case that this constraint is appropriate, but it may take some work to properly validate it, and if may feel safer to stick with what base FHIR says.

Robert Scanlon (Apr 19 2021 at 13:01):

The problem is that I believe US Core very strongly implies that it already added the constraint "GET is required, on top of whatever other requirements come from the base spec" , and at this point stating that it doesn't will incur some costs. But that is just my opinion, and "very strongly implies" doesn't mean anything from a conformance perspective, and even if it did you are still free to change it in v3.2.0.

Robert Scanlon (Apr 19 2021 at 13:13):

If you state that GET isn't a requirement, I do think that you should rewrite US Core so that search requirements within the Capability Statement are presented in terms of POST for clarity's sake. It's a lot of changes, and a pain, because the POST syntax isn't nearly as concise. But that's why we like GET, it is concise. Also, do you need to add a constraint on top of the flexibility in search by post so that servers can't do something strange, like support POST but decide to ignore the body and only obey the querystring parameters?

Daniel Venton (Apr 19 2021 at 13:14):

John Moehrke said:

I have had many discussions about the security fallacy... not helpful that it continues. (I am not saying it is 100% false, but rather that it is a configuration failure by incompetent security. Either protect your logs, or configure your auditing properly. I prefer you protect your logs).

I'm sure that your system has correct controls on your logs. I'm sure mine does. But what about those I'm not control of, ie every router and sniffer between the client and me, treating their logs as might contain PHI. Do you think Starbucks treats their logs as protected?

John Moehrke (Apr 19 2021 at 13:15):

Daniel Venton said:

John Moehrke said:

I have had many discussions about the security fallacy... not helpful that it continues. (I am not saying it is 100% false, but rather that it is a configuration failure by incompetent security. Either protect your logs, or configure your auditing properly. I prefer you protect your logs).

I'm sure that your system has correct controls on your logs. I'm sure mine does. But what about those I'm not control of, ie every router and sniffer between the client and me, treating their logs as might contain PHI. Do you think Starbucks treats their logs as protected?

so you are communicating PHI without TLS?

Daniel Venton (Apr 19 2021 at 13:16):

Does the URL parameters get encrypted now? I thought the url was only in plain text. If the URL is encrypted what is everybody's problem with using GET in the 1st place?

John Moehrke (Apr 19 2021 at 13:17):

see what I mean by a security fallacy... The URL has always been encrypted using https (TLS). ALWAYS.

Daniel Venton (Apr 19 2021 at 13:17):

I learned something today, thanks!

Cooper Thompson (Apr 19 2021 at 13:50):

The whole TLS encryption point isn't a boolean "security: yes" thing. You'll normally have TLS between the client and wherever you terminate the TLS connection, but you might terminate TLS near your front-door, and have non-TLS connections for some hops behind the front-end reverse proxy and the back-end FHIR server. Depending on network architecture and what hops are involved with which parties, you may still have some areas that are not using TLS. POST can add one layer to a defense-in-depth security posture.

Yunwei Wang (Apr 19 2021 at 14:13):

Operation has similar GET format, such as ValueSet/$validate-code?url=xxx&code=yyy&system=zzz Will this SHALL POST SHOULD GET thing apply to operation too?

John Moehrke (Apr 19 2021 at 14:59):

Cooper Thompson said:

The whole TLS encryption point isn't a boolean "security: yes" thing. You'll normally have TLS between the client and wherever you terminate the TLS connection, but you might terminate TLS near your front-door, and have non-TLS connections for some hops behind the front-end reverse proxy and the back-end FHIR server. Depending on network architecture and what hops are involved with which parties, you may still have some areas that are not using TLS. POST can add one layer to a defense-in-depth security posture.

I don't disagree... but my point still stands. You have chosen to not encrypt and thus you don't seem to care about the PHI in the body either. So there is no difference here. GET vs POST are identical.

Brendan Keeler (Apr 20 2021 at 13:48):

I'm not sure why we're relitigating something that is decidedly not FHIR specific
https://security.stackexchange.com/questions/233795/are-url-parameters-of-get-and-post-requests-over-https-secure
https://stackoverflow.com/questions/3063530/whats-the-difference-between-get-and-post-encryption
https://stackoverflow.com/questions/499591/are-https-urls-encrypted

HTTPS protects GET and POST equally against man in the middle. Only the FQDN is really exposed (due to unencrypted SNI pre TLS 1.3).

Server logs have unencrypted URLs is the main point of contention. We need to think whether that's a real risk

ryan moehrke (Apr 20 2021 at 17:32):

on this whole topic, is there a way to (in the capability statement) declare support for search as POST-only vs GET and POST?
would that have to go in interaction.documentation?

Eric Haas (Apr 20 2021 at 17:57):

I will most likely put in documentation here for each SP defined in IG:

searchParam:
            definition: >-
              http://hl7.org/fhir/us/core/SearchParameter/us-core-allergyintolerance-clinical-status
            documentation:   #<<<<<< put it here <<<<<<<<

Eric Haas (Apr 20 2021 at 18:03):

if you declare across the board then here: CapabilityStatement.rest.documentation

Lloyd McKenzie (Apr 20 2021 at 22:15):

@ryan moehrke Not computably - at least not right now. As documentation, it could show up a few places.

John Moehrke (Apr 21 2021 at 14:48):

I have submitted a request to reevaluate this with a request to relax. To at the very least allow an Implementation Guide to constrain out POST. Better to eliminate this misguided requirement and replace it with a design consideration. https://jira.hl7.org/browse/FHIR-31943

John Moehrke (Apr 21 2021 at 15:51):

add to the fallacy that POST is better than GET.... that POST allows exactly the same encoding of the parameters on the URL... Thus without further refinement to forbid URL parameters, there exactly the same concern with POST as GET.

This has exactly the same semantics as the equivalent GET command. Note that in the POST variant, parameters may appear in both the URL and the body. Parameters have the same meaning in either place. Since parameters can repeat, putting them in both places is the same as repeating them (which is valid for some parameters and not for others).

Vassil Peytchev (Apr 21 2021 at 16:04):

Once again, POST is required to work around URL size limitations. You cannot predict or force a limit on the URL size, since the base is variable. All FHIR servers support search over POST. There is no value in IGs being able to constrain out POST searches.

John Moehrke (Apr 21 2021 at 16:12):

@Vassil Peytchev that only works if you require POST search parameters in the body... our spec allows them on the URL.

John Moehrke (Apr 21 2021 at 16:13):

I am not at all opposed to making a strong statement about considerations for size.. that is a very legitimate need.

Lloyd McKenzie (Apr 21 2021 at 16:35):

Once a SHALL statement is normative, I don't know that we have a mechanism that would allow that to be relaxed - and we'd certainly need a compelling reason to try to do that. Right now, all I've heard is "some people might maybe have issues with it" rather than "this is a serious issue here, here and here".

John Moehrke (Apr 21 2021 at 16:51):

relaxing a requirement is not a breaking change. right? (Especially one that few knew about)
I have stated my case in the jira ticket FHIR-31943
I have offered many potential paths.
Others are encouraged to weight in

Jame Dang (Apr 28 2021 at 05:52):

@Lloyd McKenzie : If I understand well, it is not possible to search with extension in HAPI now (I'm sorry if wrong topic)?

James Agnew (Apr 28 2021 at 12:50):

HAPI does support search by POST if that's what you mean.

Lloyd McKenzie (Apr 28 2021 at 14:23):

You would have to define a new SearchParameter instance to allow search by the desired extension. However, so long as you do that and register it with the HAPI server, HAPI will support the search.

John Moehrke (May 17 2021 at 20:29):

with a POST based search. what is the Content-Type header to hold? Is application/x-www-form-urlencoded? This is defined as the encoding, but is not clear it is defined to be the content-type header value

Yunwei Wang (May 29 2021 at 23:30):

Sorry to bring this topic back. The resolution for FHIR-18551 is not aligned with what we discussed here.. Quote the resolution

In section 3.1.1.2, right in front of the paragraph starting "The server determines which of the set of resources...", will add a new paragraph stating:

Conformant FHIR servers SHALL support both the GET and POST-based mechanism for all FHIR queries they support.

Since this is an old ticket which was resolved two years ago, I am wondering if the resolution is still correct based on the discussion here that POST search is mandatory and GET search is optional.
@Lloyd McKenzie

Lloyd McKenzie (May 30 2021 at 01:33):

I'm a bit confused about how we thought we could make GET mandatory given that R4 was already normative?

Yunwei Wang (May 30 2021 at 02:17):

I think, the ticket was based on the assumption that GET was already mandatory by FHIR spec. The resolution was to add POST as mandatory also. It is the same misunderstanding at the beginning of this thread.

Yunwei Wang (May 30 2021 at 02:20):

(deleted)

Lloyd McKenzie (May 30 2021 at 02:38):

I don't think we can make GET mandatory - too many organizations have policies (sensible or not) that prohibit it.

Yunwei Wang (May 30 2021 at 02:50):

That is my question about this resolution. Should we reopen it?

Lloyd McKenzie (May 30 2021 at 04:09):

I think so. Can you add it to our agenda?

Yunwei Wang (May 30 2021 at 14:08):

done

John Moehrke (Jun 01 2021 at 12:55):

Neither should be mandatory in FHIR core.

Lloyd McKenzie (Jun 01 2021 at 13:53):

POST is mandatory. It's a SHALL that's normative.

Kate Dech (Aug 19 2021 at 23:31):

(Sorry-NotSorry for resurrecting this thread!)

When you use POST for a search, and you return a bundle that must have a the self-link (to communicate back the parameters used in the search)...can you expose all of the query parameters as a URL even though it was received as body using a POST command? Are users expecting that a self-link is executable?
Thanks

@Mona O , @Bapi Behera

Lloyd McKenzie (Aug 20 2021 at 03:33):

That's actually a really good question. I'm assuming the intent is that the 'self' link is always expressed as a GET, regardless of how the search is invoked. However, we should definitely discuss that and, whatever the answer is, make it explicit in the specification. @Kate Dech, can you submit a change request on this issue?

Josh Mandel (Aug 20 2021 at 03:49):

(agreed, good question! Similar question applies to next/prev links, especially to the extent that they're repeating params rather than totally opaque structures.)

Paul Church (Aug 20 2021 at 03:58):

I feel like that's the only option available, put the query params in the URL. We had the same consideration when trying to define a gRPC FHIR API where the query goes in its own field - the client is going to have to decipher the self/next/etc. links and move the query params somewhere else to formulate its next request.

Paul Church (Aug 20 2021 at 04:00):

I think we should clarify that explicitly, but also consider whether there's some different path we could be considering. It seems like a burden on the client to expect that they can decipher the URL when we haven't defined exactly what it contains.

Lloyd McKenzie (Aug 20 2021 at 04:36):

With next/prev, we also need to clarify whether - if the next/prev are expressed as GET, is the client allowed to invoke them with POST - and must the server accept that.

Paul Church (Aug 20 2021 at 04:40):

is allowed to invoke them as a POST with the query parameters, or as a POST with the parameters in the body, or both?

Grahame Grieve (Aug 20 2021 at 05:03):

no need to decipher, simply to convert to a POST. Though for most servers, there's no reason not to do a get (unfortunately, though, that's not something that can be relied on)

Vassil Peytchev (Aug 20 2021 at 12:30):

Given that one of the reasons for POST search is the size limit on URLs, it seems that we should at least mention that the URLs in the response Bundle may have to be converted to POST with parameters in the body. Unless, as Paul mentioned, we can come up with an alternative approach.

John Moehrke (Aug 20 2021 at 12:34):

sigh... and all this because of a fallacy about POST being more secure than GET.

Vassil Peytchev (Aug 20 2021 at 12:36):

size limit on URLs

• they do exist.

John Moehrke (Aug 20 2021 at 12:38):

I did not disagree with the size limit. I disagree with the "POST being more secure than GET" --- I was very specific

Lloyd McKenzie (Aug 20 2021 at 13:21):

Size limits on URLs shouldn't be an issue for next/previous, so the only reason for those being pushed into POST would be organizational policy.

Josh Mandel (Aug 20 2021 at 13:35):

no need to decipher, simply to convert to a POST. Though for most servers, there's no reason not to do a get (unfortunately, though, that's not something that can be relied on)

As Paul points out above, we need to specify what convert means. A post with query parameters is a pretty unusual construct, but it's technically valid. Is that what we mean? Or do we mean the clients are parts the query parameters and turn them into form parameters when posting?

John Moehrke (Aug 20 2021 at 13:37):

or just using POST with URL parameters... effectively not using the POST body... which is allowed (note this has the same "security" problem that GET is "said to have")

Josh Mandel (Aug 20 2021 at 13:39):

(And to Vassil's point, it faces the same URL length limitations that GET is said to have.)

John Moehrke (Aug 20 2021 at 13:40):

can we just stop this charade? POST has important benefits (size being one).

Josh Mandel (Aug 20 2021 at 13:40):

Sorry what is the charade?

Josh Mandel (Aug 20 2021 at 13:40):

Are you saying that fhir should drop support for either get or post?

John Moehrke (Aug 20 2021 at 13:41):

The charade is that POST is more secure than GET, and this is why POST is mandated and GET is optional (for searches).

John Moehrke (Aug 20 2021 at 13:41):

POST is definitely needed

Josh Mandel (Aug 20 2021 at 13:42):

And the question on the thread is about how to support it best. I don't see a charade.

John Moehrke (Aug 20 2021 at 13:42):

Not the original thread

John Moehrke (Aug 20 2021 at 13:43):

I started this when I realized that FHIR core MANDATES POST and indicates GET is optional (for searches).

John Moehrke (Aug 20 2021 at 13:44):

that mandate is just a bit too strong. If GET is not pushed to the background, then there would not be questions like the new one on Bundle encoding and next/prev

Josh Mandel (Aug 20 2021 at 13:44):

Sorry, I've just been talking about the discussion that started here yesterday, about how to populate things like self links and next links in a search responsible; the earlier discussion about mandatory support for GET/POST based search was already brought to FHIR Infrastructure, according to the notes above.

John Moehrke (Aug 20 2021 at 13:45):

and I still think we will continue to come up against issues like this.. all over a silly misguided policy based on a fallacy.

John Moehrke (Aug 20 2021 at 13:46):

yes this self/next/prev does need to be addressed related to POST.. yes, i understand that is regardless of the mandate/optionality.

Daniel Venton (Aug 20 2021 at 13:48):

Let's throw the difference between GET and POST when it comes to default caching into the mix too! :)

John Moehrke (Aug 20 2021 at 13:49):

note, I am also interested in this encoding question because I have the same problem when recording a search in an AuditEvent

John Moehrke (Aug 20 2021 at 13:50):

my only point is that FHIR core should have never said one way or the other... the statement of preference should be an IG scope.

Lloyd McKenzie (Aug 20 2021 at 13:59):

FHIR mandates POST because everything can be done with POST and only some things can be done with GET. It was also mandated because some organizations mandate POST (for good reason or not doesn't matter) but we couldn't find any organizations that mandate GET. By requiring everyone support POST, we maximize interoperability. It wasn't about fostering false beliefs around security.

Daniel Venton (Aug 20 2021 at 13:59):

The problem I see if you allow individual IGs to specify which operations shall be supported.... Company has to comply with 2 different IGs. IG A says SHALL GET and SHALL NOT POST and IG B says SHALL NOT GET and SHALL POST. What does Company do, stand up 2 different servers to handle different IGs? What if an IG says SHALL support GET but a governmental authority prohibits it?

What if Base FHIR said, "All compliant FHIR servers SHALL support both GET and POST for searches, unless local regulations prohibit their use. Each server SHALL indicate via their /metadata which operations are supported. (perhaps metadata already does?)" By having the search operations supported in the metadata, at least a client would know how to communicate with the server.

Yunwei Wang (Aug 20 2021 at 15:09):

Can IG nullify a SHALL in FHIR base spec (Server SHALL support POST search)? I doubt about that.

All compliant FHIR servers SHALL support both GET and POST for searches, unless local regulations prohibit their use.

That basically change the SHALL to a SHOULD because anyone can use that "unless" part as an excuse for not supporting one of them

Kate Dech (Aug 20 2021 at 15:13):

@Lloyd McKenzie : FHIR-33220 has been created for this.

@Josh Mandel - we didn't have the same issue with the pagination because the link didn't have to be understood to be executable. We encrypted the parameters in the page links. As a server receiving it, we understood what they include. But the self link was more problematic because we are trying to communicate back the parameters actually used.

Josh Mandel (Aug 20 2021 at 15:17):

we didn't have the same issue with the pagination because the link didn't have to be understood to be [introspectable by clients]

That's fair; it still seems worth documenting at least if clients are expected to follow next/prev links via GET even if the original search was via POST.

Vassil Peytchev (Aug 20 2021 at 15:20):

"expected to follow next/prev links via GET" or "allowed to choose whether to follow next/prev links via GET or POST"?

Josh Mandel (Aug 20 2021 at 15:20):

Right, that's a critical question and we need to be clear.

John Moehrke (Aug 20 2021 at 15:20):

or is there a way the server can indicate that POST should be used

Josh Mandel (Aug 20 2021 at 15:21):

If there's a choice:

1. does it belong to the server or to the client?
2. If it belongs to the server, how does the client know?
3. How does a client turn a "next page" link into a POST? (presumably we want body params here, but we need to say)

Josh Mandel (Aug 20 2021 at 15:22):

I've updated my comment on the Jira issue to capture this

Vassil Peytchev (Aug 20 2021 at 15:34):

on next/prev links

presumably we want body params here, but we need to say

I actually didn't mean that. What I had in mind is that if I executed my original search using method, I could still use method for next/prev links. The server must ensure that the next/prev links are not too long for its HTTP server to handle.

For self links, I think we need to note that the client must be aware of the possibility that the self url may be too long for the server to process, and therefore the client must be able to convert it into POST with params.

The open question for me is whether there is a way to actually represent the self link as a POST link with parameters.

Daniel Venton (Aug 20 2021 at 16:01):

I have to wonder why the client, which presumably issued the initial request, would have decipher the self link in order to re-issue the request?

Vassil Peytchev (Aug 20 2021 at 16:13):

I have to wonder why the client, which presumably issued the initial request, would have decipher the self link in order to re-issue the request?

The specific case we are pondering is when the client issues a POST search with the search parameters in the body. The self link cannot represent that, as it is just a URL.

Josh Mandel (Aug 20 2021 at 18:10):

I have to wonder why the client, which presumably issued the initial request, would have decipher the self link in order to re-issue the request?

@Daniel Venton it's not so much that clients need to re-issue, but this is how clients can understand how the server processed their initial request (e.g., if the server had to ignore some params it didn't understand, the server omits these from the self link, to tell the client "here's the search I actually did for you")

Josh Mandel (Aug 20 2021 at 18:10):

From http://build.fhir.org/search.html#errors

The client can determine what parameters the server used by examining the self link in the return (see below)

Gino Canessa (Aug 20 2021 at 20:40):

Not loving the idea, but we could define an extension with a valueParameters for released versions that has the processed key-value pairs in it. That would allow servers to at least return proper context for values that are over their URL limit.

Josh Mandel (Aug 20 2021 at 20:44):

er, valueReference(Parameters[contained])? That's tough.

Daniel Venton (Aug 20 2021 at 20:50):

I think we've strayed off topic. The idea was about POST and could a client deconstruct the self link and be able to know how to do a POST vs a GET. I understand that a client might want to know which values it specified originally were ignored (or modified, _count). But I still don't see a reason why a client would have to deconstruct a self link and re-execute as POST or GET. Yes the client may have done a POST and the self link might read like a GET that doesn't mean it's going to be executed as a GET or deconstructed and reformed into a POST (if it happens to be too long to be executed as a GET).

John Moehrke (Aug 20 2021 at 21:02):

I think the current point is that it might be too big for the datatype of the self element

Gino Canessa (Aug 20 2021 at 23:07):

Josh Mandel said:

er, valueReference(Parameters[contained])? That's tough.

One day I'm going to add Parameters as an option, so that I'll be right when I repeatedly try to do this. ;-)

In reality, I'd probably change it to something like a 0..* string that has a URL-Encoded KVP then.

Alexander Kiel (Aug 26 2021 at 12:28):

http://build.fhir.org/http.html#paging says:

The links are opaque to the client, have no dictated structure, and only the server understands them.

In the case of a search, the initial request may be made via a POST, but the follow up page requests will be made via GET requests. However servers SHOULD allow for a client to convert the follow up requests to be made via a POST.

That means, it should be possible to issue a GET request using the opaque next link without any modification. If so the link has to satisfy the following requirements:

1. don't be to long to work with any HTTP client and any intermediate HTTP server (which isn't in control of the FHIR server)
2. don't expose sensitive information
3. be also requestable with POST (but without modification)
4. be cache friendly (as a bonus and good HTTP citizen)

@Kate Dech You said, that you encrypt the query params. That would satisfy 2. but not necessarily 1.

To satisfy 1. in all cases, I have no better idea as to implement it in a stateful way. That would mean that every search creates state which might complicate implementation and hurt performance.

Has someone an idea that will satisfy 1. but works without creating new state for every search?

Vassil Peytchev (Aug 26 2021 at 13:10):

Has someone an idea that will satisfy 1. but works without creating new state for every search?

Wouldn't you want new state for every multi-page search for performance reasons?

Michele Mottini (Aug 26 2021 at 13:25):

2. don't expose sensitive information

No, sensitive information in GET URL are ok

Alexander Kiel (Aug 26 2021 at 13:50):

Vassil Peytchev said:

Has someone an idea that will satisfy 1. but works without creating new state for every search?

Wouldn't you want new state for every multi-page search for performance reasons?

In my implementation, I don't need to create state for multi-page searches. My database exposes explicit snapshots to my application, so all I need to track in order to ensure consistent paging is the identifier of the database snapshot. I put this identifier in the URL. So if you will, that snapshot identifier refers to state, but that state isn't create because of the search, it's created after every transaction (write).

So just storing the search params, because they don't fit in the URL, is additional state for me.

Alexander Kiel (Aug 26 2021 at 13:51):

Michele Mottini said:

2. don't expose sensitive information

No, sensitive information in GET URL are ok

I thought that is one of the reasons for using POST instead of GET. Because URL's are usually logged but request bodies not.

Grahame Grieve (Aug 26 2021 at 13:52):

usually. Any intermediary can log the body.

Grahame Grieve (Aug 26 2021 at 13:52):

you don't secure stuff for what 'usually' happens

Grahame Grieve (Aug 26 2021 at 13:52):

storing the search params, because they don't fit in the URL

This feels like a fictional concern to me. What FHIR search would ever be longer than 1-2k?

Grahame Grieve (Aug 26 2021 at 13:53):

and if it is... reject it

Alexander Kiel (Aug 26 2021 at 13:53):

So what is the reason to use POST? What makes POST more 'secure'?

Grahame Grieve (Aug 26 2021 at 13:53):

POST is not more secure. As far as we are concerned.

Alexander Kiel (Aug 26 2021 at 13:55):

Grahame Grieve said:

storing the search params, because they don't fit in the URL

This feels like a fictional concern to me. What FHIR search would ever be longer than 1-2k?

We have the use case of expanded, say SNOMED concepts into a big or-search, because our FHIR server doesn't support modifiers like :below or :in.

Grahame Grieve (Aug 26 2021 at 13:55):

hah well, that could get long. So you'll have to make some decision then

Lloyd McKenzie (Aug 26 2021 at 15:03):

"Opaque" means "no standard way to parse", not "guaranteed to not contain sensitive information".

Lloyd McKenzie (Aug 26 2021 at 15:04):

POST is not more secure, but some implementers have policies that presume it is (because logs more typically capture URLs than POST bodies).

Alexander Kiel (Aug 26 2021 at 16:00):

But does that mean that implementers have to support the case where clients strip the query params of a next link and POST'in them to an endpoint of the remaining URL? Or does "opaque" count for the whole URL including the query params? If so and if a implementation ensures not to disclose any actual initial query params in the URL of the next link, than a GET would be safe.

Lloyd McKenzie (Aug 26 2021 at 16:36):

Ideally, previous/next links would always be accessed by GET. But I can imagine some prev/next links repeating all of the search parameters and just adding on an offset or something (not best practice, due to problems with data changing between queries, but not prohibited) and then some security person freaking out because there are names, identifiers or other PHI in a URL and saying that next/prev links need to be traversed as POSTs too. And therefore, it's something we should probably address in the specification.

Brian Forbis (Oct 05 2021 at 13:29):

We also support multi-ID searches to efficiently read resources in batches. With resource IDs maxing out at 64 characters, you can hit 1k characters with only 16 IDs in your search. For mechanisms like this, we pretty much NEED to use POST due to URL length limits. The self links returned by these bundles will not be accessible using GET for the same reason.

Sridhar (Nov 10 2021 at 01:49):

In SearchWithPost if we implement parameters only in the URI or only in the body, is that conformant to the specifications or should the parameters be in both URI and the body? @Lloyd McKenzie

Lloyd McKenzie (Nov 10 2021 at 02:44):

A given parameter SHOULD only appear in one place. If it appears in both places, then it's the same as if you'd repeated the parameter multiple times on the URL or in the body. However, it's possible for some parameters to be in the URL and others to be in the body.

Vassil Peytchev (Nov 10 2021 at 04:18):

Does that mean that servers are required to support such mix of URL and body parameters?

Sridhar (Nov 10 2021 at 05:09):

Thanks, @Lloyd McKenzie

Grahame Grieve (Nov 10 2021 at 09:10):

@Vassil Peytchev we've never said anything about that, but I think that a mix is inherent in the way HTTP works, and therefore expected. I'm interested to hear if you know of anything from elsewhere about this

Michele Mottini (Nov 10 2021 at 22:56):

We do not support parameters in the URL for POST searches, and I am actually surprised that such a thing is done / expected- seems super-confusing that you can mix

Vassil Peytchev (Nov 10 2021 at 23:02):

We support either body or URL parameters, but not mixed. I am curious if there are any clients that build POST searches in a mixed way.

Grahame Grieve (Nov 11 2021 at 00:00):

I support it because it comes naturally to people building form based solutions - the action attribute includes some parameters, the method is POST, and there's form variables. Bingo, you have mixed parameters

Jose Costa Teixeira (Nov 11 2021 at 10:45):

What happens if the form in the POST body contains _filter?

Last updated: Apr 12 2022 at 19:14 UTC