Stream: IG creation
Topic: Packages2.fhir.org
Grahame Grieve (Aug 17 2021 at 09:56):
Someone has a script that is hammering packages2.fhir.org. I'm not sure what the purpose of this is, and it's running from multiple IP addresses. I'm going to start ignoring the request that the client is making; it's made the server unstable. Obviously it shouldn't, but I can't debug
Grahame Grieve (Aug 17 2021 at 09:58):
although it's leaking threads. I can see that much...
Grahame Grieve (Aug 17 2021 at 10:24):
unfortunately, it doesn't leak threads for me...
Grahame Grieve (Aug 17 2021 at 12:58):
I'm going to ban the IP addresses from using the server at all
Chris Moesel (Aug 17 2021 at 16:33):
SUSHI uses packages2.fhir.org, but only for projects that declare R4B / R5 as their FHIR version -- and even then, only if it doesn't find the package in the FHIR cache first. I don't think that CI servers (like the FHIR IG builder) maintain a FHIR cache, so SUSHI would hit packages2.fhir.org on each CI build of a R4B/R5-based IG. But that shouldn't look like hammering. Still, if anyone discovers that SUSHI is somehow involved, please let us know.
Jens Villadsen (Aug 17 2021 at 16:44):
Ahemmmmm ... If its from Denmark I might know the culprit ...
Jens Villadsen (Aug 17 2021 at 16:45):
In terms of 'hammering' ... Whats the approx amount here? @Grahame Grieve
Grahame Grieve (Aug 17 2021 at 19:55):
~10/sec, but when the server is down, the threads bank up, so once the server comes up, it's deluged by thousands
Grahame Grieve (Aug 17 2021 at 19:56):
whatever it is is making the request http://packages2.fhir.org/packages/catalog?canonical=http://hl7.org/fhir/us/carin-bb
Chris Moesel (Aug 17 2021 at 20:17):
Oh, OK. That's definitely not SUSHI then. We don't do any queries on catalog -- and as I noted before, SUSHI is scoped to look only for R4B/R5 packages on packages2 anyway. Phew. Glad it's not me. ;-)
Grahame Grieve (Aug 17 2021 at 20:56):
well, I've 'solved' it for now by turning off non-SSL access to the server. But if whoever is running the script switches to SSL too, then the problem will come back.
Grahame Grieve (Aug 17 2021 at 20:56):
I feel like there should be a better solution, but can't think of it
Lloyd McKenzie (Aug 17 2021 at 20:57):
Having a blacklist is probably a good idea. We're going to have others who hit our registries in production systems and don't cache properly and we need to protect ourselves. Rate limiting is another possibility.
Grahame Grieve (Aug 17 2021 at 20:58):
it is rate limiting now, but that rate limits everyone.
Grahame Grieve (Aug 17 2021 at 20:58):
I can't rate limit inside the server because the denial of service is to the server surface.
Grahame Grieve (Aug 17 2021 at 20:59):
and windows firewall is utterly useless. :-(
Gino Canessa (Aug 17 2021 at 20:59):
This is running on Server, isn't it? You can block IPs fairly easily.
Grahame Grieve (Aug 17 2021 at 21:00):
well, how? I tried, but windows gives me some crap about 'has to be a secure connection', which is irrelevant to me
Grahame Grieve (Aug 17 2021 at 21:02):
"you can specify user and computer authentication only when the action is set to "Allow only secure connections."
Grahame Grieve (Aug 17 2021 at 21:02):
we do not want windows authentication here. Stupid
Grahame Grieve (Aug 17 2021 at 21:03):
or I can whitelist IPs.
Grahame Grieve (Aug 17 2021 at 21:03):
But I can't figure how out to blacklist them
Grahame Grieve (Aug 17 2021 at 21:04):
anyway @Chris Moesel this affects you. If you're using packages2.fhir.org, you have to switch to https: instead of http: as of now (sorry, but the denial of service attack gives me no choice)
Chris Moesel (Aug 17 2021 at 21:33):
Thanks for the heads up, @Grahame Grieve. I understand you're in a tight spot. I do want to clarify though, it doesn't just affect me -- it currently affects every SUSHI user who has an R4B/R5 IG and uses the autobuild system (or does not have R4B/R5 in their FHIR cache). So... we will see some failed IG builds until we push out a new SUSHI release w/ the fix. We'll try to get one out quickly.
Grahame Grieve (Aug 17 2021 at 21:48):
yes right, but they are failing already
Gino Canessa (Aug 17 2021 at 21:48):
You can block them at the system level with the firewall. You just need a custom rule in to deny access, I believe the step-by-step in this article should work.
Jens Villadsen (Aug 18 2021 at 17:15):
Ok ... that for sure aint me ... carin bb is not my thing
Corey Spears (Aug 21 2021 at 23:43):
OK, carin-bb is my thing and I am confused as to what is going on. carin-bb does use SUSHI, but does not use R4B or R5 and there has only been a couple of small changes to the IG in the last 1.5 months (back on August 3). I have no idea how carin-bb could be causing an issue and no clear way to diagnose. How can I tell if it is using packages2? How might I tell if something else is going on? Is there a way to tell where this is coming from or specifically the request that is causing the issue?
Lloyd McKenzie (Aug 22 2021 at 04:19):
It's not likely the IG-publication process. It's someone who's implementing.
Grahame Grieve (Aug 22 2021 at 05:00):
someone doing something. my gut call: they picked a package at random
Last updated: Apr 12 2022 at 19:14 UTC
