Stream: committers
Topic: LATEST(?) IG Publisher
Chris Moesel (May 17 2019 at 01:45):
Based on the fhir-ig-publisher commit logs, the latest release of the publisher should be 0.9.8-SNAPSHOT. But using the advertised link https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.hl7.fhir.publisher&a=org.hl7.fhir.publisher.cli&v=LATEST&e=jar results in org.hl7.fhir.publisher.cli-0.9.6-20190509.152505-1.jar. I understand that the "SNAPSHOT" part can be replaced by a timestamp, but there is still a discrepency between 0.9.8 and 0.9.6. Does the link actually retrieve the most recent "blessed" version of the publisher (which may be older than the last build)? Or is there something awry with the build system?
Lloyd McKenzie (May 17 2019 at 01:47):
The latter. Though it seems to be a maven issue rather than a build issue - so far as we can tell.
Grahame Grieve (May 17 2019 at 01:47):
it feels like a caching issue somewhere, though I don't know what we could do to create that
Chris Moesel (May 17 2019 at 02:05):
OK. Dug in a bit. This shows the issue: https://oss.sonatype.org/service/local/repositories/snapshots/content/org/hl7/fhir/publisher/org.hl7.fhir.publisher.cli/maven-metadata.xml
Chris Moesel (May 17 2019 at 02:06):
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
<groupId>org.hl7.fhir.publisher</groupId>
<artifactId>org.hl7.fhir.publisher.cli</artifactId>
<versioning>
<latest>0.9.6-SNAPSHOT</latest>
<release></release>
<versions>
<version>0.9.0-SNAPSHOT</version>
<version>0.9.1-SNAPSHOT</version>
<version>0.9.2-SNAPSHOT</version>
<version>0.9.3-SNAPSHOT</version>
<version>0.9.4-SNAPSHOT</version>
<version>0.9.5-SNAPSHOT</version>
<version>0.9.6-SNAPSHOT</version>
<version>0.9.7-SNAPSHOT</version>
<version>0.9.8-SNAPSHOT</version>
</versions>
<lastUpdated>20190515022659</lastUpdated>
</versioning>
</metadata>
Chris Moesel (May 17 2019 at 02:12):
Notice that the value for <latest> is wrong. Found an article about this indicating that LATEST is not reliable: https://articles.javatalks.ru/articles/32 (sigh).
Grahame Grieve (May 17 2019 at 02:24):
well, that's crap. how do we tell people to get the latest then? there must be a way that works...?
Chris Moesel (May 17 2019 at 02:42):
There seem to be some scripts that can do this -- but that's not exactly user-friendly. Perhaps if HL7 could host a microservice that uses Sonatype's API (or one of the scripts found on google) to correctly resolve the latest version and return a redirect?
Chris Moesel (May 17 2019 at 02:44):
Or perhaps some Javascript that does this (or even grabs the maven-metadata.xml, parses it, and uses an algorithm to determine the latest from the list of versions) -- then modifies the DOM to include a link to it.
Chris Moesel (May 17 2019 at 02:44):
Maybe if I can snag a little time this weekend I could try the latter approach. It's crazy, but it just might work.
Grahame Grieve (May 17 2019 at 02:48):
so that prompted me to actually read that article. It was making sense in a this-is-a-train-wreck kind of way until I got to this paragraph:
Also note, that if you think you need the latest versions, you’re probably moving in a wrong direction, I haven’t seen good workflows which require latests.
I'm always happy to be told that my learning is as yet incomplete, but if someone asks me to fix a bug, and fix it a release a new IG publisher, what other workflow is there than 'use the latest'?
Chris Moesel (May 17 2019 at 02:56):
I wonder if they are advocating that you should point people to specific releases rather than a link that always goes to the "latest". For example, we can still get to the real latest -- we just need to use the actual link: https://oss.sonatype.org/service/local/repositories/snapshots/content/org/hl7/fhir/publisher/org.hl7.fhir.publisher.cli/0.9.8-SNAPSHOT/org.hl7.fhir.publisher.cli-0.9.8-20190515.022459-1.jar
Grahame Grieve (May 17 2019 at 02:58):
well, so I just tell everyone to update to the latest... seems wonderfully inefficient
Chris Moesel (May 17 2019 at 03:00):
I also wonder if perhaps part of the issue is that you're using SNAPSHOTS? In addition to the "LATEST" tag, there is also a "RELEASE" tag -- that I think is supposed to represent the latest formal release. (I don't think SNAPSHOTs are considered real releases since they're intended to represent daily builds).
Chris Moesel (May 17 2019 at 03:00):
So maybe if you released the publisher not as a SNAPSHOT then the we'd have a way forward. But I don't know for sure.
Grahame Grieve (May 17 2019 at 04:01):
I’m waiting for @James Agnew to figure out some crypto issue before we can do that
Grahame Grieve (May 17 2019 at 04:01):
But we do want to do that
Chris Moesel (May 17 2019 at 12:28):
The other approach (which I didn't think of last night) is to continue with SNAPSHOT but don't rev the version number (except maybe with major releases). For example, if you set the version to 1.0.0-SNAPSHOT, that would typically be interpreted as a daily build leading up to an eventual 1.0.0 release. You could publish 1.0.0-SNAPSHOT many times, and clients (or URLs) that ask for 1.0.0-SNAPSHOT would always get the most recently published 1.0.0-SNAPSHOT.
Grahame Grieve (May 17 2019 at 21:22):
but then I wouldn't have any idea what 1.0.0 people were running. nor would I be able to manage the release of new builds of Core FHIR which wouldn't pick up new copies of the core code
Chris Moesel (May 20 2019 at 12:45):
As for your first concern, when you publish the SNAPSHOT to Sonatype, the actual filename of the jar will be generated with a timestamp (just like in the existing ones; e.g., org.hl7.fhir.publisher.cli-0.9.6-20190509.152505-1.jar). That should help with knowing what version someone is running (although you may still need to do date compares to find where it is in the commit history). As for your second concern, I don't think I understand it well enough to comment.
Chris Moesel (May 20 2019 at 12:48):
@Grahame Grieve -- if you log into the oss.sonatype.org server and browse/search to the org.hl7.fhir.publisher.cli artifact, does it give you a button or link to rebuild the maven metadata XML? If so, then doing that should reset the latest in the metadata file as well. I just don't know if sonatype exposes that to you.
Grahame Grieve (May 20 2019 at 13:00):
not that I can see
Grahame Grieve (May 20 2019 at 13:00):
it's another version behind now
Grahame Grieve (May 21 2019 at 03:07):
i can't find out how to update the latest....
Chris Moesel (May 21 2019 at 04:21):
It sounds like it may not be possible, based on that article I linked the other night. I was hoping maybe they gave you a way to at least manually force it, but apparently not. I think you'll likely need to explore other approaches. :-(
Chris Moesel (May 21 2019 at 04:22):
If you want a static URL that always returns the latest, I can think of three choices (all of which I think we previously discussed):
[1] Publish as real releases (not SNAPSHOT) and _hopefully_ the "RELEASE" tag works
[2] Publish as a steady SNAPSHOT version (e.g., 1.0.0-SNAPSHOT) and rarely increment the number
[3] Write a small microservice that will download the maven meta XML, parse it, and redirect to the _real_ latest (but that requires a server)
Grahame Grieve (May 21 2019 at 04:28):
[4] add to my little build script that builds to push it somewhere else
Grahame Grieve (May 21 2019 at 04:28):
https://github.com/HL7/fhir-ig-publisher/blob/master/release.bat
Grahame Grieve (May 21 2019 at 04:32):
@Josh Mandel the latest link for the IG-publisher just isn't working. It's causing problems across the eco-system. Should I go back to pushing to build.fhir.org (how?)? or on www.fhir.org (ftp)?
Josh Mandel (May 21 2019 at 14:05):
The idea of pushing to the same snapshot version was no good?
Josh Mandel (May 21 2019 at 14:06):
Maybe just push to a fixed location in a cloud storage bucket or even GitHub (like a force push to a branch like latest-build that just contains the latest)
Grahame Grieve (May 21 2019 at 19:50):
ok: https://github.com/FHIR/latest-ig-publisher/blob/master/org.hl7.fhir.publisher.jar
Grahame Grieve (May 21 2019 at 19:53):
which makes the download link https://github.com/FHIR/latest-ig-publisher/raw/master/org.hl7.fhir.publisher.jar
Lloyd McKenzie (May 21 2019 at 20:00):
Is this working? (I.e. do I change my templates to leverage this right now?)
Grahame Grieve (May 21 2019 at 20:08):
it's go right now yes
Grahame Grieve (May 21 2019 at 20:10):
@Josh Mandel can you update the ci-build to use this link? (unless it's already sorted using maven somehow). (btw - I created a new repo rather than using a branch since it's simpler overall)
Lloyd McKenzie (May 21 2019 at 20:21):
Shouldn't it be org.hl7.fhir.igtools.publisher.jar?
Grahame Grieve (May 21 2019 at 20:24):
maybe but I was working from the namespace in maven not java.
Chris Moesel (May 21 2019 at 20:25):
I thought it was org.hl7.fhir.publisher.cli.jar. :shrug:
Josh Mandel (May 21 2019 at 20:36):
I'll configure the auto builder to use this link, sure @Grahame Grieve. How time sensitive?
Lloyd McKenzie (May 21 2019 at 20:39):
The Get doesn't seem to be paying any attention to the timestamp - it retrieves it always, even if it hasn't changed. That wasn't the case with how we had it stored before the transition to Maven.
Grahame Grieve (May 21 2019 at 20:45):
"The get" ?
Grahame Grieve (May 21 2019 at 20:45):
@Josh Mandel well a couple of CDA IGs are broken but the authors aren't jumping up and down right now
Josh Mandel (May 21 2019 at 20:48):
I think Lloyd is talking about support for a conditional headers, by time stand or ETAG
Josh Mandel (May 21 2019 at 20:48):
This is actually a pretty important issue...
Grahame Grieve (May 21 2019 at 20:50):
github are not doing this?
Josh Mandel (May 21 2019 at 20:50):
I'm not saying that. But I need to take a look.
Lloyd McKenzie (May 21 2019 at 20:51):
I think Ant relies on timestamp. It doesn't have anywhere to put an etag.
Josh Mandel (May 21 2019 at 21:03):
Correct about ant.
Lloyd McKenzie (May 21 2019 at 21:06):
My base requirement is that I want to be able to launch the IGPublisher in such a way that, if I wish, I can force a download of the most recent publisher before I run it. And I only want the 70+MB download to happen if something has actually changed. Whether I do that with Ant or whether we figure out some other cross-platform mechanism, I don't really care.
Josh Mandel (May 21 2019 at 21:06):
I understand.
Josh Mandel (May 21 2019 at 21:06):
Github supports etag headers (and if-none-match negotiation) for regular file access like https://github.com/FHIR/latest-ig-publisher/raw/master/org.hl7.fhir.publisher.jar. It should support last-modified headers (and if-modified-since negotiation) if you go via gh-pages, like https://fhir.github.io/latest-ig-publisher/org.hl7.fhir.publisher.jar
Josh Mandel (May 21 2019 at 21:07):
Though please note that this introduces some caching.
Josh Mandel (May 21 2019 at 21:07):
Do you want to give this a shot? @Lloyd McKenzie
Josh Mandel (May 21 2019 at 21:15):
(@Grahame Grieve I turned on "github pages" access for the repo at https://github.com/FHIR/latest-ig-publisher/settings to make this work.)
Josh Mandel (May 21 2019 at 21:24):
Meanwhile, I've updated the auto IG builder to use https://github.com/FHIR/latest-ig-publisher/raw/master/org.hl7.fhir.publisher.jar -- should be effective now.
Josh Mandel (May 21 2019 at 21:25):
BTW @Grahame Grieve shall we move auto-ig-builder from https://github.com/hl7-fhir to https://github.com/fhir ?
Grahame Grieve (May 21 2019 at 21:27):
probably in theory but it's not a pressing issue
Josh Mandel (May 21 2019 at 21:27):
Just did a test and https://build.fhir.org/ig/HL7/cda-tools-template/branches/master/build.log seems to indicate things are working with the IG Publisher coming from the right place.
Josh Mandel (May 21 2019 at 21:28):
Sure not pressing but I might as well just hit the button on it, no? If we "transfer repo" links to https://github.com/hl7-fhir/auto-ig-builder will still work (redirected). Only links to https://hl7-fhir.github.io/auto-ig-builder/builds.html might break... which is pretty much internals. (And they might get redirected, I'm just not sure.)
Grahame Grieve (May 21 2019 at 21:28):
great
Josh Mandel (May 21 2019 at 21:31):
OK, updated to https://github.com/fhir/auto-ig-builder . Let me know if anything blows up.
Josh Mandel (May 21 2019 at 21:31):
I fixed the link at https://dev.azure.com/fhir-build/build.fhir.org/_wiki/wikis/build.fhir.org.wiki? but don't know where else might still be pointing there. Some official docs for the IG Publisher presumably.
Josh Mandel (May 21 2019 at 21:31):
ah, http://wiki.hl7.org/index.php?title=Process_for_Publishing_a_FHIR_IG
Josh Mandel (May 21 2019 at 21:32):
Updated that too.
Grahame Grieve (May 21 2019 at 21:33):
and then just everyone's local bookmarks
Josh Mandel (May 21 2019 at 21:51):
https://hl7-fhir.github.io/auto-ig-builder/builds.html should help with that :)
Lloyd McKenzie (May 21 2019 at 22:11):
@Josh Mandel Yes, that works. Thanks!! :) (When you mentioned 'caching issues', what kind of issues should we expect/guard against?
Josh Mandel (May 21 2019 at 22:16):
If you use the GH Pages URL, you should expect that files won't be updated immediately after a push to github. Would expect a minute or two of delay in general. Beyond that... I think you'll have to try it and see .
Josh Mandel (May 21 2019 at 22:16):
If it's unsuitable, we'll move to a cloud storage bucket somewhere. But I think it's probably going to be okay.
Lloyd McKenzie (May 21 2019 at 22:19):
A few minutes delay is totally fine.
Josh Mandel (May 21 2019 at 22:29):
Good :) Let's see how well that matches reality next time we have an update.
Last updated: Apr 12 2022 at 19:14 UTC
