Stream: genomics/committers
Topic: FSH branch
Kevin Power (Oct 05 2020 at 17:13):
Thanks to @May Terry we have a new FSH branch:
branch -> https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh
deployed -> http://build.fhir.org/ig/HL7/genomics-reporting/branches/genomics-reporting-fsh/index.html
Kevin Power (Oct 05 2020 at 18:56):
@May Terry - If you could also switch over an example so that everyone can have an example of how to do an example, that would be great.
And for everyone else #genomics/committers two things:
- I suggested that we just do 'in-place' upgrades of the artifacts versus the completely separate approach she started with. We can always compare FSH upgraded rendered version back to the master branch.
- Don't forget, you have to install SUSHI (if you haven't already) (https://fshschool.org/docs/sushi/installation/)
May Terry (Oct 05 2020 at 18:58):
@Kevin Power - yup! I can do that. Lemme see what I can have done by tomorrow. I'm currently working on replacing the CG profiles I showed earlier so that the FSH ones are replacing them rather than a copy. I should have that in my next remote push later today.
May Terry (Oct 05 2020 at 20:10):
Just pushed another commit to https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh which replaces the existing artifacts from the original StructureDefinition to its FSH-authored equivalent: Specimen, GenomicsBase, TMB, and High/Low codes.
deployed list of artifacts here: http://build.fhir.org/ig/HL7/genomics-reporting/branches/genomics-reporting-fsh/artifacts.html.
Note: I've currently labeled all of the fsh ones only in the artifact labels. It's done on purpose for tracking purposes. They'll all be removed when its closer to merging.
Patrick Werner (Oct 06 2020 at 15:28):
great work May! Thanks a lot.
Kevin Power (Oct 06 2020 at 16:16):
Indeed, looks great @May Terry !
May Terry (Oct 06 2020 at 16:17):
thanks all! I'll be committing the Variant profile to the branch later today.
Kevin Power (Oct 06 2020 at 16:35):
@May Terry -- What is the best strategy for the number of .fsh files? You started with files by type of artifact, which is great except maybe for Profiles since we have a lot of them. Do you expect all of our profiles to be in the single .fsh file, or should we have some grouping?
If we split, would something like this work?
CGGeneral.fsh (servicerequest, specimen, genomics-report, genomics-base, etc ...)
CGObservationFindings.fsh (genomic finding, overall interp, region studied, grouper, variant, haplotype, genotype, seq phase reltn, 'biomarkers')
CGObservationImplications.fsh (genomic implication, diagnostic, therapeutic, med usage, recommended followup)
May Terry (Oct 06 2020 at 16:38):
That's a really good question, and suggestion, @Kevin Power - it's really up to the modeling team that will be collaborating and eventually maintaining the IG. The SUSHI compiler looks at all .fsh file extensions and doesn't distinguish it by file name so it's really up to us.
Happy to re-organize the FSH tank to the file structure you suggested and push it in the next commit. We might also want to have a sub-team meeting for anyone that wants to actively contribute.
Kevin Power (Oct 06 2020 at 16:42):
I would vote for the structure I outlined above, but welcome input from others: tagging @Jamie Jones @Patrick Werner @ Bob Milius @Bret H
May Terry (Oct 06 2020 at 16:45):
I can also start a GoogleSheet on representation differences and issues for update by CG FSH contributors. That is, unless we want to have a different tracking system for this work. Thoughts?
Kevin Power (Oct 06 2020 at 16:49):
I think a Google Sheet is fine for that. If we find issues that have some sort of material impact we will need to consider logging new JIRAs to track those changes. But I am assuming most will fall into “QA” type issues
Jamie Jones (Oct 06 2020 at 16:51):
Agreed. I won't have bandwidth to contribute to this but am happy to help review any changes in the sdefs and familiarize myself with the new structures for future edits
Kevin Power (Oct 06 2020 at 16:54):
I can spend some time this week, how about others? @ Bob Milius, @Bret H, and @Bob Dolin specifically?
Bob Milius (Oct 06 2020 at 21:43):
I'm just catching up with this thread. Great work! I can spend some time on this later in the week.
Bob Dolin (Oct 06 2020 at 21:47):
@Kevin Power Let me know how I can help.
May Terry (Oct 06 2020 at 22:01):
It might help to coordinate who's doing what and managing contributions (e.g.: are they all branches off of https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh, who will merge if there are conflicts, etc.). Perhaps a meeting for the coalition of the willing? I'm available either tomorrow or Thursday, 3-5p. LMK what 1 hour block works for everyone and I can send out an invite.
Kevin Power (Oct 06 2020 at 22:21):
My thoughts for a plan:
1 - We should ONLY use the single branch, creating multiple branches will cause lots of confusion.
2 - I am afraid my calendar is not very friendly right now, and if I get a chance to work on anything, it will likely be during off hours.
3 - I think we can work off of this thread - if someone wants to convert an artifact, please post the artifact name to this topic. Everyone else can leave that artifact alone.
4 - Once you complete an artifact, build locally to test.
5 - Once you confirm everything looks good locally, do a git commit/push.
6 - Update this thread as you complete the artifact and note when you are ready to pick up the next one.
7 - If you have questions/issues, let's bring there here as well. We might need a spreadsheet at some point, but I suggest we start simple.
Kevin Power (Oct 07 2020 at 01:31):
I am working on 'servicerequest'
May Terry (Oct 07 2020 at 01:50):
@Kevin Power - please wait for the file reorganization. I don't want to do merge conflict resolution.
Kevin Power (Oct 07 2020 at 01:51):
Have you started it (the reorg)? If not, I can do it now. I was about to commit/merge :smile:
May Terry (Oct 07 2020 at 01:51):
haven't done it yet. Variant is taking a while.
Kevin Power (Oct 07 2020 at 01:52):
Is it OK if I do it now?
May Terry (Oct 07 2020 at 01:53):
lemme commit what I currently have. one min.
May Terry (Oct 07 2020 at 01:54):
ok. pushed. Variant is still WIP but good stopping point for tonight anyway. I'll finish it in the morning. :-)
May Terry (Oct 07 2020 at 01:54):
so...feel free to reorg. :-)
Kevin Power (Oct 07 2020 at 02:43):
OK, I have committed. Everyone should sync up before working again. And probably obvious, but @May Terry the Variant profile is now in CGFindings.fsh
May Terry (Oct 07 2020 at 02:51):
Thanks @Kevin Power! Did you successfully generate the IG with the latest changes?
Kevin Power (Oct 07 2020 at 02:52):
Yup. There are a lot of errors right now but many are related (I think) to variant which will go away once you are done :+1:
May Terry (Oct 07 2020 at 02:53):
hmm...I didn't get any SUSH compiler errors before committing. Do you mean QAReport?
Kevin Power (Oct 07 2020 at 02:53):
Yes
May Terry (Oct 07 2020 at 02:53):
ok. makes sense. thanks!
May Terry (Oct 07 2020 at 04:32):
ok. Currently looking at the QAReport output for variant. I see there are a number of LOINC code resolution issues. I'll look into them further in the morning. I finished a first pass at the variant profile in the meantime with the disclaimer that it's still WIP.
Kevin Power (Oct 07 2020 at 11:33):
If you see similar LOINC errors in other profiles it could be tooling problems. There are more than a few of those.
May Terry (Oct 07 2020 at 12:55):
Seems like there are a couple of different kinds of errors, some which I see in other profiles which were not migrated to FSH. I'll try to separate them out to qualify the different types problems. On a related note, I noticed that there is some odd narrative rendering for some of the FHIR examples, both in the ones I authored in FSH, and also a couple of others that were from the original IG. The examples however seem to pass the validator so I suspect it's related to the IG Publisher. So not the highest priority but probably makes sense to resolve those before STU2 publication.
May Terry (Oct 07 2020 at 13:49):
FYI for those interested in the latest FSH/SUSHI updates:
Please join our discussion tomorrow, Thursday October 8 at 9-10 am Eastern US time. We will be focusing on the SUSHI 1.0 release.
Join Microsoft Teams Meeting
+1 540-492-5664 United States, Roanoke (Toll)
Conference ID: 889 990 654#
May Terry (Oct 07 2020 at 17:31):
This migration work seems to be doubling as a deep FHIR modeling review/scrub of the CG profiles, especially with Variant. I found that original Variant StructureDefinition incorrectly specified component::variant-length.value[x] and component::copy-number.value[x] to the Count datatype through a profile reference:
"type": [
{
"code": "Quantity",
"profile": [
"http://hl7.org/fhir/StructureDefinition/Count"
]
}
]
Instead, per this Zulip chat thread with Grahame (https://chat.fhir.org/#narrow/stream/179177-conformance/topic/Narrow.20Quantity.20to.20Duration.20and.20Age/near/200727989), Count is done through a fhirpath specification. Yeh, it's really klunky, but the options are... 1) constrain the value to a different data type instead of Count (e.g.: integer), 2) keep it high level as Quantity, 3) change your StructureDefinition as an invariant.
For FSH, I'm going to try option 3 but it would be nice for the sake of simplicity to agree to option 1.
Thoughts?
Jamie Jones (Oct 07 2020 at 17:57):
Per https://jira.hl7.org/browse/FHIR-27146 we can remove variant-length. For copy-number, positiveInt seems appropriate??
Jamie Jones (Oct 07 2020 at 17:59):
I understand a copy number of 1 normally means a single deletion (for diploid regions) so copy number of 0 should be supported, implies unsignedInt might be best
May Terry (Oct 07 2020 at 18:02):
Thanks @Jamie Jones - We can remove variant-length, and also note to update the examples accordingly since it'll might further increase the number of QAReport known errors in specifying something that no longer exists.
Jamie Jones (Oct 07 2020 at 18:02):
Yeah we've got a lot of outdated refs in examples to clean up :space_invader:
Kevin Power (Oct 08 2020 at 03:51):
Hey all, I hope no one was working on something, as I forgot to put a notice here. I just committed the following:
Convert Genomics Implication (abstract) and RelatedArtifact extension. Note the base implication is missing two slices on derivedFrom (genotype and haplotype), it seems they need to be converted before we can reference them. Plus a few minor corrections.
May Terry (Oct 08 2020 at 12:45):
@Kevin Power - thanks for doing these changes. Unfortunately, I pulled them in and yes, it overwrote my changes because we modified the same file. I think the workflow serves better from a traceability and code control perspective to branch and merge moving forward.
Kevin Power (Oct 08 2020 at 12:48):
I am not sure how familiar you are with Git, but you should be able to 'stash' your changes (basically does a local commit), and then you can compare and merge in your changes?
May Terry (Oct 08 2020 at 12:48):
without doing this, we'll need to do local merges and less optimally, not being even able to easily see diffs for reconciling them. For FSH, VSCode provides a really nice way to see these and merge them dynamically.
May Terry (Oct 08 2020 at 12:48):
so...traceability and diffs. please.
Kevin Power (Oct 08 2020 at 12:49):
I am OK with multiple branches.
Kevin Power (Oct 08 2020 at 12:50):
Honestly, we typically have so little work going on that this isn't a concern :smile: And I have to remind myself that we don't have the spreadsheets to worry about merging any longer.
Kevin Power (Oct 08 2020 at 12:51):
(though I have done the stash approach successfully multiple times)
May Terry (Oct 08 2020 at 12:54):
yeh...I used to use git stash/pop at my last job. it's just that it's probably a more sustainable workflow for folks to do code reviews, dynamically comment on specific lines of code, and more easily track down where changes occurred. doing git stash/pop hides all that and from an auditing perspective, loses the ability to find out who did what, at least without doing a lot of serial backtracking.
May Terry (Oct 08 2020 at 12:59):
Now that I pulled in the latest, I guess I'll create a new branch off of genomics-reporting-fsh. We don't necessarily have to go overboard and do pull requests unless we feel necessary, but at least it'll catch the conflicts when we merge and pull. :-)
Kevin Power (Oct 08 2020 at 12:59):
Fair enough. How would you propose a branch strategy? One per committer?
May Terry (Oct 08 2020 at 13:00):
yup.
May Terry (Oct 08 2020 at 13:01):
off of the fsh branch. when it's in a stable spot, then we can do a team wide joint review before folks feel comfortable enough to merge to master. That is unless we feel comfortable enough with the current environment to have each committer directly branch off of master. imho, I'm not sure I'm comfortable with that...at least for my contributions. :-)
Kevin Power (Oct 08 2020 at 14:21):
I am fine with branching off the fsh branch.
May Terry (Oct 08 2020 at 14:33):
@Kevin Power - probably more for my knowledge, but I was modifying genomics.xml to note in the Artifacts list which profiles have been converted to FSH. I noticed in the latest commit that this file has been moved to the parking-lot directory and the original artifact labels I used to mark them is now also gone. Is there some other location where I should be doing this? Without noting them, it would help to note somewhere for QA purposes which profiles have been FSH authored.
As another convention, rather than deleting the original StructureDefinition that was replaced by FSH, I was moving them to the parking-lot directory. This allowed me more easily to do direct comparisons within the same project without having to open 2 projects. Would you prefer to continue deleting them or moving them over to parking-lot? By moving them into parking-lot, it is a lot easier to backtrack the entire StructureDefinition if we run into issues with the FSH equivalent.
May Terry (Oct 08 2020 at 14:50):
I'll be tackling region-studied and msi profiles next.
Jamie Jones (Oct 08 2020 at 14:51):
is variant live somewhere? happy to help QA
May Terry (Oct 08 2020 at 14:53):
@Jamie Jones - http://build.fhir.org/ig/HL7/genomics-reporting/branches/genomics-reporting-fsh/StructureDefinition-variant.html. Still a bit buggy and needs changing for the elements with the Count datatype, but good for a first-pass review.
Kevin Power (Oct 08 2020 at 16:22):
May Terry said:
Kevin Power - probably more for my knowledge, but I was modifying genomics.xml to note in the Artifacts list which profiles have been converted to FSH. I noticed in the latest commit that this file has been moved to the parking-lot directory and the original artifact labels I used to mark them is now also gone. Is there some other location where I should be doing this? Without noting them, it would help to note somewhere for QA purposes which profiles have been FSH authored.
As another convention, rather than deleting the original StructureDefinition that was replaced by FSH, I was moving them to the parking-lot directory. This allowed me more easily to do direct comparisons within the same project without having to open 2 projects. Would you prefer to continue deleting them or moving them over to parking-lot? By moving them into parking-lot, it is a lot easier to backtrack the entire StructureDefinition if we run into issues with the FSH equivalent.
Sorry about that, I thought we had decided to just do the "inplace" upgrades and just compare to the master branch. So I removed those FSH notations. If we are going to have multiple branches, I suppose each branch can do the validation as they see fit, but I don't know that we need those notations as we merge into the one 'genomics-report-fsh' branch? What do others think?
Jamie Jones (Oct 08 2020 at 17:15):
Great start with Variant! In the differential view, I'm seeing a "Unordered, Open by type:$this" slicing rendering for most components with a value[x] datatype of valueCodeableConcept. I don't think the slicing is intentional, can anyone confirm?
May Terry (Oct 08 2020 at 17:24):
@Jamie Jones - yep! good catch. :-) Coincidentally, I actually noted this exact behavior in Variant on the FSH call this morning. I think per Chris Moesel, what's there is equivalent to what you originally had. In essence, what we've constrained is value[x] with a slice of 1. However, so that it's not confusing, I think the FSH syntax can be re-specified so it shows as before. Lemme try that out and change it for the next commit.
Jamie Jones (Oct 08 2020 at 17:28):
I'm not an expert on slicing, but this formulation may actually be MORE CORRECT and remove some warnings in our examples... Technically we want to say that one of the valueCodeableConcept's codings must contain something from a specific answer list. Additional codings are permitted.
Jamie Jones (Oct 08 2020 at 17:28):
as far as i know, these were throwing 'no known slice...' warnings whenever we added an additional coding
Jamie Jones (Oct 08 2020 at 17:30):
It's a tradeoff we may want to discuss, informational warnings vs simplicity in the sdef. Personally I think either is fine
May Terry (Oct 08 2020 at 17:30):
good point again. I think this is why I wanted to have a GoogleSheet with modeling questions and issues that crop up while we're doing the migration. Whether we use it or not, I've been making my own punchlist and we can discuss it in Monday's modeling meeting if you think it's worth adding to the agenda.
Jamie Jones (Oct 08 2020 at 17:32):
sheets.new is a good friend of mine, if you make one I would be happy to comment :)
Kevin Power (Oct 08 2020 at 20:11):
I have made some notes in the FSH files, but if you want to share your punch list @May Terry feel free :smile:
May Terry (Oct 08 2020 at 22:25):
@Kevin Power below is the punch list of what I've captured so far. @Jamie Jones - lmk if you have a spreadsheet created for this. I prefer to have it there than embedded in FSH files. I add comments in there but it's not meant for code review/QA issue tracking.
o Migration of IG to FSH
Q: why are there 2 bodySite elements in Genomics base?
Bug?: the following elements are in the Diff table but not constrained to anything:
• basedOn
• status
• effectiveDateTime
• issued
New change: category shows as sliced. Could be equivalent and a cosmetic difference. Verify this.
Fixed:
• secondaryFinding extension
• bodySite – duplicate element name but represented differently. Q: what was the rationale?
Bug (fixed): High/Low codes value set incorrectly defined the value set instead of the code system. Should be http://terminology.hl7.org/CodeSystem/v3-ObservationInterpretation
Bug: Variant.component::copy-number.value[x] incorrectly represented Count in its StructureDefinition. Per zulip thread with Grahame, this should be an invariant.
Kevin Power (Oct 08 2020 at 22:54):
May Terry said:
Q: why are there 2 bodySite elements in Genomics base?
Fixed:
• bodySite – duplicate element name but represented differently. Q: what was the rationale?
Regarding the bodySite extension, it's data type is BodyStructure (http://hl7.org/fhir/R4/bodystructure.html) where the standard attribute is just a CodeableConcept. This was part of the first draft of the profile from Lloyd, and we have just never changed it. Perhaps we should at least rename the extension to be more clear (though since it is an extension, it is probably technically OK)?
May Terry (Oct 09 2020 at 00:23):
@Kevin Power et al - I created a PullRequest to merge. No conflicts so to not slow down the process, I'll just merge it. You can see it just in case: https://github.com/HL7/genomics-reporting/pull/8. It contains changes to Variant to reflect (unbound), and the Count data type (which shows as Quantity with an invariant, as recommended by Grahame). Also included are 2 new migrated profiles for Region Studied and MSI.
May Terry (Oct 09 2020 at 00:29):
Will probably start this tomorrow, but I'd like to do Operation: $find-subject-variants next if that's okay with everyone.
Jamie Jones (Oct 09 2020 at 00:32):
It has some errors currently, let me know how it goes
Jamie Jones (Oct 09 2020 at 13:40):
It has some errors currently, let me know how it goes
May Terry (Oct 10 2020 at 15:39):
First pass of OperationsDefinition $find-subject-variants completed in https://github.com/HL7/genomics-reporting/pull/9 so I merged it to https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh. Please update your fsh development branches accordingly.
Jamie Jones (Oct 12 2020 at 13:44):
@May Terry @Kevin Power would either of you like some time on call today to highlight updates from this effort?
May Terry (Oct 12 2020 at 13:44):
@Jamie Jones - sure. I can give an update.
Kevin Power (Oct 12 2020 at 14:03):
I have a conflict the first 1/2 of the meeting, but hope I can join the second 1/2
Jamie Jones (Oct 12 2020 at 14:06):
we can do examples and trackers first then. branch review shouldn't take too long
May Terry (Oct 12 2020 at 16:32):
For anyone wanting to contribute either in FSH authoring or QA, we're meeting tomorrow at 3p ET (1p MT) to go over contribution workflow and QA. Here's the meeting invite: https://teams.microsoft.com/l/meetup-join/19%3ameeting_NTkyOTkxZmQtMzY2MC00MjcwLWFjZDctYTYzZDc2OTI1MjVl%40thread.v2/0?context=%7b%22Tid%22%3a%22c620dc48-1d50-4952-8b39-df4d54d74d82%22%2c%22Oid%22%3a%22912e5167-1db2-4628-b366-47b76d2062f8%22%7d
May Terry (Oct 12 2020 at 19:57):
Looks like the file reorganization to SUSHI 1.0 might cause merge conflicts with current FSH branches. Let me create the next PR and branch merge so that folks can pull in the changes. I'll update the stream when it's ready to pull.
May Terry (Oct 12 2020 at 20:14):
Okay....I merged https://github.com/HL7/genomics-reporting/pull/11 to genomics-reporting-fsh branch which updates thet directory structure to support SUSHI 1.0. Please pull the changes if you have an active branch.
Patrick Werner (Oct 13 2020 at 10:30):
Jamie Jones said:
I'm not an expert on slicing, but this formulation may actually be MORE CORRECT and remove some warnings in our examples... Technically we want to say that one of the valueCodeableConcept's codings must contain something from a specific answer list. Additional codings are permitted.
Both is possible, the solution on the FSH branch introduces an unneeded slicing on value[x]. We are having components with defined value[x] data types. Slicing value[x] makes sense if we are expecting different data types for a component.value.
Patrick Werner (Oct 13 2020 at 10:31):
To express: "Additional codings are permitted". Everything was correct in the "old branch". value[x] set to a defined type, + Binding (which never disallows additional bindings.
Jamie Jones (Oct 13 2020 at 12:31):
Thanks Patrick
Bob Milius (Oct 13 2020 at 14:03):
Should we be standardizing a version of sushi (1.0?) and publisher during development. Or just use the latest version?
Patrick Werner (Oct 13 2020 at 14:10):
would go for the newest version at the moment as new features are still added, right?'
edit: i'm using SUSHI v0.16.0 (implements FHIR Shorthand specification v0.13.x)
Patrick Werner (Oct 13 2020 at 14:13):
and now i'm on the 1.0.0-beta (thanks for bringing that up)
Patrick Werner (Oct 13 2020 at 14:14):
@May Terry i love the cli outputs of sushi :heart:
May Terry (Oct 13 2020 at 14:25):
We should be using the latest sushi 1.0 version, even if beta.
May Terry (Oct 13 2020 at 14:26):
I have a CG-FSH tracker spreadsheet started so we can potentially sign up for migrating profiles. Would it be helpful if I post this to as a GoogleSheet and share? If so, what's the GoogleDoc folder that I can use?
Patrick Werner (Oct 13 2020 at 14:26):
@May Terry had a look at the variant generated by your branch. Could this be resolved by removing the line (and others alike): * component[dna-chg].value[x] only CodeableConcept ?
May Terry (Oct 13 2020 at 14:27):
Possibly. Haven't tried it. Give it a try and let me know! :-)
Patrick Werner (Oct 13 2020 at 14:27):
trying, but running into technical issues on my side (i think) will keep you updated
Patrick Werner (Oct 13 2020 at 14:28):
@May Terry you can place the file here: https://drive.google.com/drive/folders/1zmeWkjA4DXF_D9xv5gLn5Tr0gWkdHNpC?usp=sharing
Patrick Werner (Oct 13 2020 at 14:33):
i'm getting the org.hl7.fhir.exceptions.FHIRException: Cannot find definition for MedicationUsage Exception, and later:
## Memory (MB): Use = 1367, Free = 1081, Total = 2449, Max =4096
Jekyll: /usr/local/Cellar/ruby/2.7.2/lib/ruby/2.7.0/rubygems.rb:277:in `find_spec_for_exe': can't find gem jekyll (>= 0.a) with executable jekyll (Gem::GemNotFoundException) (03:45.0345)
Jekyll: from /usr/local/Cellar/ruby/2.7.2/lib/ruby/2.7.0/rubygems.rb:296:in `activate_bin_path' (03:45.0345)
Jekyll: from /usr/local/bin/jekyll:23:in `<main>' (03:45.0346)
Jekyll has failed. Complete output from running Jekyll: (03:45.0348)
Note: Check that Jekyll is installed correctly (03:45.0349)
Publishing Content Failed: Process exited with an error: 1 (Exit value: 1) (03:45.0350)
May Terry (Oct 13 2020 at 14:33):
Thanks @Patrick Werner - here's the uploaded Googlesheet. Feel free to modify it in any way you find helpful: https://docs.google.com/spreadsheets/d/1OVDHPQ24aVQE_ztCin38yF23P0CHNnkk/edit#gid=850969131
Patrick Werner (Oct 13 2020 at 14:33):
on the genomics-reporting-fsh branch
Patrick Werner (Oct 13 2020 at 14:33):
sushi and jekyll are up-to-date
Patrick Werner (Oct 13 2020 at 14:33):
IG publisher updated to the newest version
May Terry (Oct 13 2020 at 14:35):
yup. I'm getting that as well. Not sure it's related to FSH/SUSHI as I don't think this IG is even referencing MedicationUsage. I suspect it's the IG Publisher. I ignore it for now since it doesn't seem to affect any of the CG profiles I see thus far.
May Terry (Oct 13 2020 at 14:35):
But we probably should be adding this to our MigrationTracker just in case.
Patrick Werner (Oct 13 2020 at 14:36):
Resolving the MedicationUsage thing is assigned to myself. Will investigate this week (was on vacation the last 2 weeks)
Patrick Werner (Oct 13 2020 at 14:37):
So you are seeing the expection, but the build finishes? Without jekyll error?
May Terry (Oct 13 2020 at 14:38):
I'll run it again, but I get no unhandled exceptions and am able to generate the IG in spite of it. Is the IG Publisher crashing on your side?
Patrick Werner (Oct 13 2020 at 14:39):
getting this: https://pastebin.com/qRdE2sMC
Patrick Werner (Oct 13 2020 at 14:39):
so i see the expection, it continues, and fails in the Jekyll step
May Terry (Oct 13 2020 at 14:41):
I'll send my output shortly. currently running it again.
May Terry (Oct 13 2020 at 14:42):
yeh...works for me.
org.hl7.fhir.exceptions.FHIRException: Cannot find definition for MedicationUsage
at org.hl7.fhir.r5.renderers.ProfileDrivenRenderer.render(ProfileDrivenRenderer.java:110)
at org.hl7.fhir.r5.renderers.ProfileDrivenRenderer.render(ProfileDrivenRenderer.java:99)
at org.hl7.fhir.r5.renderers.ResourceRenderer.build(ResourceRenderer.java:55)
at org.hl7.fhir.r5.renderers.BundleRenderer.render(BundleRenderer.java:298)
at org.hl7.fhir.igtools.publisher.Publisher.getXhtml(Publisher.java:7807)
at org.hl7.fhir.igtools.publisher.Publisher.saveDirectResourceOutputs(Publisher.java:7135)
at org.hl7.fhir.igtools.publisher.Publisher.generateHtmlOutputs(Publisher.java:6581)
at org.hl7.fhir.igtools.publisher.Publisher.generate(Publisher.java:4857)
at org.hl7.fhir.igtools.publisher.Publisher.createIg(Publisher.java:881)
at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:725)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:8335)
Generating Summary Outputs (02:47.0926)
Sending Usage Stats to Server (02:52.0790)
## Memory (MB): Use = 3226, Free = 869, Total = 4096, Max =4096
Reclaiming memory...
## Memory (MB): Use = 1382, Free = 2713, Total = 4096, Max =4096
Jekyll: Source: /Users/mayt/cg/genomics-reporting/temp/pages (02:57.0923)
Jekyll: Generating... (02:57.0924)
Jekyll: done in 8.149 seconds. (03:06.0073)
Installing hl7.fhir.uv.genomics-reporting#dev to the package cache
Fetching:.
Installing: . done.
Checking Output HTML (03:10.0216)
found 1123 files (03:10.0285)
........................................
........................................
... 1123 html files, 0 pages invalid xhtml (0%) (03:40.0792)
... 456819 links, 22 broken links (0%) (03:40.0792)
Build final .zip (03:40.0793)
Final .zip built (03:46.0232)
Finished. Times: loading: 00:13.0820, generate: 02:52.0303, narrative generation: 00:25.0029, realm-rules: 00:00.0000, previous-version: 00:08.0532, jekyll: 00:11.0039, validation: 00:08.0308 (#58), template: 00:03.0749 (#3) (03:46.0601)
context [anonymous] 2:13 attribute path isn't defined
context [anonymous] 2:13 attribute path isn't defined
Validation output in /Users/mayt/cg/genomics-reporting/output/qa.html
Errors: 1399, Warnings: 20, Info: 365, Broken Links = 22 (03:50.0722)
Done. Note that this IG is good for your local use but is not suitable for use as a final publication (consult Confluence for publishing advice if you are actually building with intent to publish) (03:50.0724)
(base) Tue Oct 13 10:42:20 genomics-reporting (genomics-reporting-fsh-mlt) $```
Patrick Werner (Oct 13 2020 at 14:44):
so it is a jekyll issue :(
May Terry (Oct 13 2020 at 14:44):
Maybe uninstall/reinstall Jekyll? Also not sure if you have other Ruby or Ruby-on-Rails applications that you're also working with but I go through unexplained Jekyll errors when I upgrade my rubygems environment because of a different application on a different project.
Patrick Werner (Oct 13 2020 at 14:44):
on my side
Patrick Werner (Oct 13 2020 at 14:45):
yes i'm also working with Inferno at the moment ...
Patrick Werner (Oct 13 2020 at 14:49):
npm, ruby gems, maven, brew packages, fhir packages... tooo much :dizzy:
May Terry (Oct 13 2020 at 14:49):
yeh...likely the case. It's probably because I'm not a Ruby gems / rvm expert, but I so hesitate upgrading anything related to Ruby on my dev laptop. After each upgrade I spend at least 3-4 hours troubleshooting and uninstalling/reinstalling/retesting all of my Ruby/RoR applications each time. :-(
Patrick Werner (Oct 13 2020 at 14:54):
never used ruby gems a lot, but already hating them :D
Patrick Werner (Oct 13 2020 at 15:19):
i think re-installing jekyll and fiddling with my gem $PATH vars did the trick
Kevin Power (Oct 13 2020 at 15:23):
Not sure what is happening for me, but I did a quick test to try and build locally, and am getting a new error:
FHIR IG Publisher Version 1.1.34 (Git# b37a5ab15769). Built 2020-10-02T04:41:27.374Z (11 days old)
File log: /var/folders/8j/4zywd4p529sdc4hq43bkykq524lzvj/T/fhir-ig-publisher-tmp.log
Detected Java version: 14.0.1 from /Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home on x86_64 (64bit). 8192MB available
Parameters: -ig .
dir = /Users/cerkyp/Documents/GitHub/genomics-reporting, path = /Users/cerkyp/.gem/ruby/2.7.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Vbrick/nodejs/bin
Run time = Tuesday, October 13, 2020 at 10:21:05 AM Central Daylight Time (2020-10-13T10:21:05-05:00)
Package Cache: /Users/cerkyp/.fhir/packages (00:00.0029)
Run Sushi on /Users/cerkyp/Documents/GitHub/genomics-reporting (00:00.0031)
Sushi: info Running SUSHI v0.16.0 (implements FHIR Shorthand specification v0.13.x) (00:01.0147)
Sushi: info SUSHI detected a "fsh" directory in the input path. As a result, SUSHI will operate in "IG Publisher integration" mode. This means: (00:01.0148)
Sushi: - the "fsh" directory will be used as the input path (00:01.0149)
Sushi: - the parent of the "fsh" directory (e.g., "../fsh") will be used as the output path unless otherwise specified with --out option (00:01.0149)
Sushi: - generation of publisher-related scripts will be suppressed (i.e., assumed to be managed by you) (00:01.0150)
Sushi: info Extracting FSHOnly configuration from input/genomics-reporting.xml. (00:01.0233)
Sushi: info Checking local cache for hl7.fhir.r4.core#4.0.1... (00:01.0233)
Sushi: info Found hl7.fhir.r4.core#4.0.1 in local cache. (00:04.0103)
Sushi: info Loaded package hl7.fhir.r4.core#4.0.1 (00:04.0105)
Sushi: error Invalid path to FSH definition folder. (00:04.0105)
Sushi: Usage: sushi [path-to-fsh-defs] [options] (00:04.0106)
Sushi: (00:04.0107)
Sushi: Options: (00:04.0108)
Sushi: -o, --out <out> the path to the output folder (00:04.0108)
Sushi: -d, --debug output extra debugging information (00:04.0109)
Sushi: -s, --snapshot generate snapshot in Structure Definition output (default: (00:04.0109)
Sushi: false) (00:04.0110)
Sushi: -i, --init initialize a SUSHI project (00:04.0111)
Sushi: -v, --version print SUSHI version (00:04.0111)
Sushi: -h, --help output usage information (00:04.0112)
Sushi: (00:04.0113)
Sushi: Additional information: (00:04.0124)
Sushi: [path-to-fsh-defs] (00:04.0125)
Sushi: Default: "." (00:04.0126)
Sushi: If fsh/ subdirectory present, it is included in [path-to-fsh-defs] (00:04.0126)
Sushi: -o, --out <out> (00:04.0127)
Sushi: Default: "build" (00:04.0128)
Sushi: If fsh/ subdirectory present, default output is one directory above fsh/ (00:04.0129)
Sushi couldn't be run. Complete output from running Sushi : (00:04.0130)
Note: Check that Sushi is installed correctly ("npm install -g fsh-sushi". On windows, get npm from https://www.npmjs.com/get-npm) (00:04.0130)
Exception: Process exited with an error: 1 (Exit value: 1) (00:04.0131)
Publishing Content Failed: Process exited with an error: 1 (Exit value: 1) (00:04.0131)
(00:04.0132)
Use -? to get command line help (00:04.0133)
(00:04.0134)
Stack Dump (for debugging): (00:04.0135)
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:153)
at org.hl7.fhir.igtools.publisher.Publisher.runFsh(Publisher.java:1544)
at org.hl7.fhir.igtools.publisher.Publisher.initialize(Publisher.java:1437)
at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:719)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:8335)
Patrick Werner (Oct 13 2020 at 15:26):
you need to update to sushi 1.0.0 (i think)
Patrick Werner (Oct 13 2020 at 15:30):
build is running again. But removing the line doesn't resolve the issue
Patrick Werner (Oct 13 2020 at 15:39):
asked about restricting the type here: https://chat.fhir.org/#narrow/stream/215610-shorthand/topic/restricting.20value.5Bx.5D.20without.20slicing
May Terry (Oct 13 2020 at 16:51):
I added BobM to the spreadsheet to migrate haplotype and genotype. As a reminder, the spreadsheet is on GoogleDrive here: https://docs.google.com/spreadsheets/d/1OVDHPQ24aVQE_ztCin38yF23P0CHNnkk/edit#gid=850969131
Feel free to sign up for other profiles to ensure no duplicate efforts. :-)
May Terry (Oct 13 2020 at 16:53):
one more thing. I suggest we leave GenomicsReport for last or almost last since it has dependencies on so many other profiles.
May Terry (Oct 14 2020 at 02:58):
Created new PullRequest migrating the ConceptMap for dna-change-type in https://github.com/HL7/genomics-reporting/pull/12. Note: I changed the id to dna-change-type-map to remove the name ambiguity with the Variant component dna-change-type which has the same exact spelling. Please do a git pull origin genomics-reporting-fsh if you have an active branch from it.
May Terry (Oct 14 2020 at 03:00):
I updated the FSH Migration Tracker spreadsheet to migrate Grouper and Overall Interpretation next. Should have this done sometime tomorrow.
Patrick Werner (Oct 14 2020 at 08:54):
Patrick Werner said:
asked about restricting the type here: https://chat.fhir.org/#narrow/stream/215610-shorthand/topic/restricting.20value.5Bx.5D.20without.20slicing
and got a answer which resolves the issue, updating profiles now
Patrick Werner (Oct 14 2020 at 09:50):
updated, tested, commited, pushed, PRed, merged:
https://github.com/HL7/genomics-reporting/pull/13
Patrick Werner (Oct 14 2020 at 09:50):
doesn't build because of the deprecated sushi version in the autobuild?
Patrick Werner (Oct 14 2020 at 09:50):
http://build.fhir.org/ig/HL7/genomics-reporting/branches/genomics-reporting-fsh/failure/build.log
Kevin Power (Oct 14 2020 at 11:39):
I can confirm - that was the error I was getting locally before I had to install the most recent version of Sushi.
Kevin Power (Oct 14 2020 at 11:59):
And reviewing the #committers/notification stream, I am seeing other auto-build failures with the same error.
May Terry (Oct 14 2020 at 13:10):
can you upgrade to the latest SUSHI version?
Kevin Power (Oct 14 2020 at 13:15):
Can we control that in the auto-build?
May Terry (Oct 14 2020 at 13:35):
Kevin Power said:
Can we control that in the auto-build?
Dunno. @Patrick Werner - can you post this issue on the #shorthand stream?
Kevin Power (Oct 14 2020 at 13:51):
I posted a quick one (though won't have time to do much more today)
May Terry (Oct 14 2020 at 13:52):
No worries. Thanks for doing this! I'll also try to follow-up internally if we don't get a prompt reply.
May Terry (Oct 14 2020 at 14:43):
Follow-up on the autobuild issue. The SUSHI team will be working with Grahame to see if they can resolve the issue within the next couple of days. In the meantime, our workaround could be to work with local builds until SUSHI 1.0 is production (which hopefully will be too long). We can also take zip file snapshots and make them available for QA folks to download and review on their local systems. Sorry about that.
Kevin Power (Oct 15 2020 at 02:19):
Added fsh.ini, committed/pushed change (directly to the '-fsh' branch), and the autobuild was successful.
May Terry (Oct 15 2020 at 02:56):
oy... @Kevin Power because there was no pull request tracking, I also added that file and merged as well.
May Terry (Oct 15 2020 at 02:58):
https://github.com/HL7/genomics-reporting/pull/14 merged which also adds the fsh.ini file. I suppose there would be a merge conflict if it wasn't compatible but you might want to pull from the merged branch to verify.
Kevin Power (Oct 15 2020 at 03:21):
Sorry should have posted here :worried:
May Terry (Oct 15 2020 at 03:22):
No worries. all good. simple change and no merge conflicts. :smile:
May Terry (Oct 15 2020 at 03:23):
Realized that Grouper has too many dependencies, so I'll be tackling DiagnosticImplication next. Updated on tracker spreadsheet.
Kevin Power (Oct 15 2020 at 03:25):
(We need more fish emojis)
May Terry (Oct 15 2020 at 04:26):
First pass migration of DiagnosticImplication in PR https://github.com/HL7/genomics-reporting/pull/15 and merged. Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh.
May Terry (Oct 19 2020 at 03:02):
First pass migration of TherapeuticImplication in PR https://github.com/HL7/genomics-reporting/pull/16 and merged.
Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh.
May Terry (Oct 19 2020 at 15:08):
First pass migration of OverallInterpretation in PR https://github.com/HL7/genomics-reporting/pull/17 and merged.
Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh.
Bob Milius (Oct 22 2020 at 16:33):
Sorry I'm late to this party. So I created a branch (genomics-reporting-fsh-rpm) from genomics-reporting-fsh and updated sushi to 1.0beta3 and publisher to the latest version (1.1.35), then ran _genonce.sh. The build completed but there are a slew of errors (Errors: 1548, Warnings: 20, Info: 350, Broken Links = 34). Is this expected? Are the errors different pieces that haven't been ported to FSH? I'm assuming the workflow to port genotype and haplotype profiles are to ignore everything except errors related to those profiles? and then once those are complete, then work on the examples that are based on those?
Bob Milius (Oct 22 2020 at 16:46):
And should genotype and haplotype be defined inside of CGFindings.fsh since its basedefinition is finding?
Bob Milius (Oct 22 2020 at 16:57):
Or should it be inside of CGGeneral.fsh as indicated in the spreadsheet?
Kevin Power (Oct 22 2020 at 21:54):
Yea, we are introducing a lot of new QA errors that we haven't had time to resolve yet.
To you next question, I say CGFindings.fsh is best for genotype and haplotype.
May Terry (Oct 26 2020 at 15:02):
First pass migration of task-med-chg in PR https://github.com/HL7/genomics-reporting/pull/18 and merged.
Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh.
May Terry (Oct 26 2020 at 15:26):
First pass migration of task-rec-followup in PR https://github.com/HL7/genomics-reporting/pull/19 plus moving the original task-med-chg json to parkinglot. merged.
Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh.
May Terry (Oct 26 2020 at 15:51):
First pass migration of ServiceRequest in PR https://github.com/HL7/genomics-reporting/pull/20 plus fixed typo in recommended-followup. Note that I had to change the id of the original ServiceRequest profile from servicerequest to service-request to avoid name collision error.
Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh.
May Terry (Oct 30 2020 at 19:23):
First pass migration of RecommendedAction and SupportingInformation in PR https://github.com/HL7/genomics-reporting/pull/21 and merged.
Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh to bring in the latest changes to your branch.
May Terry (Nov 01 2020 at 21:35):
First pass migration of Sequence Phase Relationship value set, code system, and profile in https://github.com/HL7/genomics-reporting/pull/22 and merged. Note that the code system id has changed to SequencePhaseRelationshipCS. This might require updates of examples. Also, SequencePhaseRelationship profile currrently does not include the derivedFrom slide to haplotype. This slice will be included once the profile is completed by @ Bob Milius.
@ Bob Milius - please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh to bring in the latest changes to your branch.
May Terry (Nov 02 2020 at 02:29):
First pass migration of Genomics Report and Grouper in https://github.com/HL7/genomics-reporting/pull/23 and merged. Note that both profiles are incomplete, pending completion of genotype and haplotype profiles.
Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh to bring in the latest changes to your branch.
Patrick Werner (Nov 02 2020 at 16:33):
regarding the errors: https://chat.fhir.org/#narrow/stream/179252-IG-creation/topic/Links.20cannot.20be.20resolved
maybe this is related?
Patrick Werner (Nov 02 2020 at 16:34):
i don't think so, but this was the last link resolving error on zulip
May Terry (Nov 03 2020 at 13:57):
Yeh...not sure. I'll look at the SDs for that pattern just in case.
Patrick Werner said:
i don't think so, but this was the last link resolving error on zulip
Jamie Jones (Nov 03 2020 at 13:59):
I did some analysis of the qa errors, all the ones in the SDs themselves are due to a mismatch between the snapshot and differential, and only for TBD elements
Jamie Jones (Nov 03 2020 at 14:00):
I don't know anything about how they are generated but even if the order is different between the two they may get these errors
May Terry (Nov 03 2020 at 14:00):
yeh...lemme try migrating the TBD code system and referencing the FSH authored one to see if that does it.
May Terry (Nov 03 2020 at 15:08):
I think I found the bug on the order. Checking now. :-)
May Terry (Nov 03 2020 at 15:43):
Completed first pass migration of genotype and haplotype; fixed multiple build errors related to tbd-codes. migrated tbd-codes. Note that because of the FSH naming convention of CodeSystem, id changed from tbd-codes to TbdCodes. Examples must change to reflect this. Changes in PR https://github.com/HL7/genomics-reporting/pull/24 and merged. Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh to bring in the latest changes to your branch.
Kevin Power (Nov 03 2020 at 15:52):
WOW. Just did a quick double check, and our error numbers dropped from like 1500ish down to 363! Great work @May Terry !!!
May Terry (Nov 03 2020 at 20:03):
added haplotype references to other profiles in https://github.com/HL7/genomics-reporting/pull/25 and merged. Please pull in the latest https://github.com/HL7/genomics-reporting/tree/genomics-reporting-fsh to bring in the latest changes to your branch.
Jamie Jones (Nov 12 2020 at 19:13):
okay, i've gotten the FSH branch to build locally and it's cool to merge with /replace what's in Master in my opinion. How is everyone else feeling about it? @Kevin Power @May Terry @Patrick Werner ?
May Terry (Nov 12 2020 at 19:15):
Go for it! :fish:
Jamie Jones said:
okay, i've gotten the FSH branch to build locally and it's cool to merge with /replace what's in Master in my opinion. How is everyone else feeling about it? Kevin Power May Terry Patrick Werner ?
Kevin Power (Nov 12 2020 at 19:17):
Time to 'reel' in that 'FSH' you might say? :smile:
May Terry (Nov 12 2020 at 19:20):
@Kevin Power - I don't have any new changes so once it's merged, might make sense to delete the fsh branch. We can advertise it ahead of time but I'd like to do this asap after the master merge. Thoughts?
Kevin Power (Nov 12 2020 at 19:21):
I concur.
Jamie Jones (Nov 12 2020 at 19:21):
i'm for it, so folks dont accidentally contribute there
Kevin Power (Nov 12 2020 at 19:23):
I am even OK with @Jamie Jones doing it as part of the merge to master. If someone still has a branch off the FSH branch with pending changes, I am pretty certain they can re-base to master and then contribute their changes.
Patrick Werner (Nov 13 2020 at 11:54):
looking good
Jamie Jones (Nov 20 2020 at 01:38):
anyone able to try pulling the current build and running it? I'm getting a jekyll error for a generated file missing that was working before recent publisher update :)
Jekyll has failed. Complete output from running Jekyll: Liquid Exception: Could not locate the included file 'StructureDefinition-diagnostic-implication-dict.xhtml'
Jamie Jones (Nov 20 2020 at 04:21):
Chased the build-breaking errors upstream to null pointer exceptions on org.hl7.fhir.igtools.renderers.StructureDefinitionRenderer.txDetails(StructureDefinitionRenderer.java:585). Looks like an added trim() on top of something that was gracefully failing before (our good friends the missing LOINC codes 92822-6, 93044-6, and 94076-7 from tx.fhir.org/r4). Publisher commit was https://github.com/HL7/fhir-ig-publisher/compare/1.1.41...1.1.42
Kevin Power (Nov 20 2020 at 04:25):
Yuck
Kevin Power (Nov 20 2020 at 04:27):
Did you post the problem anywhere?
Jamie Jones (Nov 20 2020 at 04:30):
just posted https://chat.fhir.org/#narrow/stream/179252-IG-creation/topic/SD.20renderer.20null.20pointer.20exception
Kevin Power (Nov 20 2020 at 15:04):
No surprise, but I recreated the same issue.
May Terry (Nov 20 2020 at 16:15):
I have an older version of the publisher. let me run that to see if we can revert on publisher versions.
May Terry (Nov 20 2020 at 16:18):
yup. no jekyll exceptions with FHIR IG Publisher Version 1.1.35 (Git# a4a66bd10efb). Built 2020-10-14T21:26:37.503Z (36 days old). Probably won't fix the CI build generation online but it at least shouldn't block one from continuing to make edits and testing them locally.
Jamie Jones (Nov 20 2020 at 16:19):
that is a good thought, i may revert my publisher and do some extra work this weekend. The variant changes were easy in FSH! a few others shouldn't be hard
May Terry (Nov 20 2020 at 16:21):
pressing forward...are there specific changes or additions you want me to work on with this IG?
e.g.: examples creation or migration? e.g.: Not 100% sure with the ballot process, but at least or mCODE publication, the TSC now required there be an illustrative example attached to every profile in order to be approved.
May Terry (Nov 20 2020 at 16:22):
I can work on creating some of the more straightforward examples in FSH so others can see what this would look like.
May Terry (Nov 20 2020 at 16:34):
Also, fyi... if you've upgraded to SUSHI 1.0, then please remove the sushi-version=beta line from the fsh.ini file. Otherwise, the publisher will run with the SUSHI 1.0.0 beta 4 version. I just tested it with both SUSHI versions and that older version of the IG publisher and SUSHI 1.0 upgrade doesn't seem to break any backwards compatibility at least for what we've done in our IG so I think the change should be fine.
Kevin Power (Nov 20 2020 at 17:50):
May Terry said:
pressing forward...are there specific changes or additions you want me to work on with this IG?
e.g.: examples creation or migration? e.g.: Not 100% sure with the ballot process, but at least or mCODE publication, the TSC now required there be an illustrative example attached to every profile in order to be approved.
I do think adding examples is always time well spent. We have had a focus on creating 'bundles' to represent a full report, along with a few 'single resource' examples. I think having a single example per profile would be very helpful.
May Terry (Nov 20 2020 at 19:38):
sounds good. I'll take a first pass at creating FSH-generated examples for the "atomic" profiles with minimal dependencies to start (e.g.: TMB, MSI) to start and then build on them incrementally as they're referenced by other more complex profiles (e.g.: variant). I'll create a PR before Monday's modeling call and be ready to go over them if needed. thanks!
Patrick Werner (Nov 24 2020 at 09:45):
Jamie Jones said:
Chased the build-breaking errors upstream to null pointer exceptions on org.hl7.fhir.igtools.renderers.StructureDefinitionRenderer.txDetails(StructureDefinitionRenderer.java:585). Looks like an added
trim()on top of something that was gracefully failing before (our good friends the missing LOINC codes 92822-6, 93044-6, and 94076-7 from tx.fhir.org/r4). Publisher commit was https://github.com/HL7/fhir-ig-publisher/compare/1.1.41...1.1.42
Patrick Werner (Nov 24 2020 at 09:45):
http://tx.fhir.org/r4/CodeSystem/$validate-code?code=92822-6&system=http://loinc.org
Patrick Werner (Nov 24 2020 at 09:46):
The tx Server knows these codes! weird...
Jamie Jones (Nov 24 2020 at 14:24):
I hear it may be a latency issue
Patrick Werner (Nov 24 2020 at 14:29):
read the same
Patrick Werner (Nov 24 2020 at 14:29):
Fixed Publisher jar:
https://1drv.ms/u/s!AiblQ6yMWNkSm5oPQozBJAEJYY7QGA?e=hPtwEF
Patrick Werner (Nov 24 2020 at 14:30):
as the PR wasn't merged
Patrick Werner (Nov 24 2020 at 14:30):
yet
May Terry (Nov 24 2020 at 19:21):
Looks like the latest publisher is checking for unhandled profiles. It caught this one... org.apache.commons.lang3.NotImplementedException: type org.hl7.fhir.r5.model.RelatedArtifact not handled - should not be here.
Patrick Werner (Nov 24 2020 at 19:25):
saw that too. Will investigate. Same for the MedicationUsage thing popping up.
Patrick Werner (Nov 24 2020 at 19:27):
@May Terry seeing you online:
SUSHI can't handle: * item[0].extension[http://hl7.org/fhir/StructureDefinition/questionnaire-itemControl].valueCoding = http://molit.eu/fhir/CodeSystem/questionnaire-codes-tbd#vassyntax.
Patrick Werner (Nov 24 2020 at 19:27):
Patrick Werner (Nov 24 2020 at 19:28):
if you have an opinion.
May Terry (Nov 24 2020 at 19:31):
I don't know that I would reference extension assignments like that. what kind of extension are you actually creating? Can you send me the entire profile that you profiled in FSH?
Jamie Jones (Nov 24 2020 at 20:02):
We need to fix the relatedartifact implementation. We have the sd but the profiles don't properly reference it. It's beyond my current understanding of FSH
Patrick Werner (Nov 25 2020 at 09:48):
Jamie Jones said:
We need to fix the relatedartifact implementation. We have the sd but the profiles don't properly reference it. It's beyond my current understanding of FSH
will give it a go today
Patrick Werner (Nov 25 2020 at 15:48):
ok:
https://chat.fhir.org/#narrow/stream/179252-IG-creation/topic/type.20org.2Ehl7.2Efhir.2Er5.2Emodel.2ERelatedArtifact.20not.20handled
https://chat.fhir.org/#narrow/stream/179252-IG-creation/topic/Cannot.20find.20definition.20for.20MedicationUsage
Patrick Werner (Nov 25 2020 at 15:48):
RelatedArtifact is a bug in the narrative generator i think.
Patrick Werner (Nov 25 2020 at 15:49):
MedicationUsage is still a bug in the Publisher, internally (for snapshoting and validation) everything is converted to R5 resources.
I think there is a bug there, not paying attention that in R4 the resource is named MedicationStatement
Jamie Jones (Nov 25 2020 at 15:56):
i do think there is an error in how we are referencing/defining relatedArtifact beyond the narrative. When i navigate to its page locally, i see a cyclic Reference(Related Artifact) rather than the actual structure of the RelatedArtifact complex dataType.
Patrick Werner (Nov 25 2020 at 17:13):
looking good, the narrative generator chokes on the DataType
Patrick Werner (Nov 25 2020 at 17:14):
Could you point me to the page with the cyclic reference?
Jamie Jones (Nov 25 2020 at 17:22):
there's a 'Reference' on http://build.fhir.org/ig/HL7/genomics-reporting/StructureDefinition-RelatedArtifact.html that points to http://build.fhir.org/ig/HL7/genomics-reporting/StructureDefinition-RelatedArtifact.html instead of a valueRelatedArtifact.
Patrick Werner (Nov 25 2020 at 17:27):
oh!
Patrick Werner (Nov 25 2020 at 17:28):
good catch! totally oversaw that. Fixed, building...
Patrick Werner (Nov 25 2020 at 17:29):
Thate explains the error, the publisher searched for a Resource called RelatedArtifact
Patrick Werner (Nov 25 2020 at 17:31):
And the error is still there :octopus:
Jamie Jones (Nov 25 2020 at 17:33):
i tried a couple way to get the valueRelatedArtifact to build right and didn't get it :shrug:
Patrick Werner (Nov 25 2020 at 17:37):
hmmm the generated SD looks weird, i think thats a sushi bug. But will check again before reporting
Patrick Werner (Nov 25 2020 at 19:48):
found it :-)
Patrick Werner (Nov 25 2020 at 19:51):
no sushi bug... but maybe a missing feature.
The Extension was named RelatedArtifact, wich introduced some weird behaviour as the name is identical to the data type
Patrick Werner (Nov 25 2020 at 19:51):
Would be neat if suhsi warns you as soon you are overwriting core stuff
Patrick Werner (Nov 25 2020 at 20:05):
fixing examples....
Patrick Werner (Nov 25 2020 at 20:13):
https://github.com/HL7/genomics-reporting/pull/28
Patrick Werner (Nov 25 2020 at 20:19):
please review and merge
Patrick Werner (Nov 27 2020 at 11:55):
Update on the MedicationUsage Bug: Grahame found the source of the problem (bug) and fixed it. Will be contained in the next release.
Patrick Werner (Dec 01 2020 at 13:51):
merged MSI example branch
Jamie Jones (Dec 01 2020 at 14:00):
I've got some pending changes in a separate branch as well
Patrick Werner (Dec 01 2020 at 14:02):
PR?
Patrick Werner (Dec 01 2020 at 14:03):
i will fix the grouper errors, the slicing has to be modified to be a target profile based slicing
Jamie Jones (Dec 01 2020 at 14:04):
I did that! A lot of our slicing had errors
Patrick Werner (Dec 01 2020 at 14:06):
uh
Patrick Werner (Dec 01 2020 at 14:06):
nice!
Jamie Jones (Dec 01 2020 at 14:07):
https://github.com/HL7/genomics-reporting/pull/30 was a bit of a rabbit hole for QA
Patrick Werner (Dec 01 2020 at 14:23):
you had sushi beta still enabled @Jamie Jones
Jamie Jones (Dec 01 2020 at 14:25):
Whoops, I can try running it without
Jamie Jones (Dec 01 2020 at 14:31):
i tweaked the examples a bit and didn't pull in the relatedArtifact fix yet
Patrick Werner (Dec 01 2020 at 14:32):
i already fixed the beta issue, also found another issue while test-building. Already fixed (double categories defined)
Jamie Jones (Dec 01 2020 at 14:36):
yeah category was a weird one. :upside_down: We still have to apply the DR changes FHIR#19831 and FHIR#25418
Patrick Werner (Dec 01 2020 at 14:36):
:+1:
Jamie Jones (Dec 01 2020 at 14:39):
updated publisher is working?
Patrick Werner (Dec 01 2020 at 14:40):
yes
Patrick Werner (Dec 01 2020 at 14:41):
what did fix all the reference errors?
Jamie Jones (Dec 01 2020 at 14:43):
all of our hasmember and derivedfrom slices i switched to #profile, and some others I tried switching the pattern from "code" to "coding.code". "coding" may be more accurate?
Jamie Jones (Dec 01 2020 at 14:45):
i think declaring the pattern on "code" wasn't working because we weren't then explicitly stating which "code" had to be present, so it never got validated somehow...
Patrick Werner (Dec 01 2020 at 15:20):
Jamie Jones said:
all of our hasmember and derivedfrom slices i switched to #profile, and some others I tried switching the pattern from "code" to "coding.code". "coding" may be more accurate?
Will have a look at the coding slices.
Patrick Werner (Dec 01 2020 at 15:20):
merged your PR, merged Mays PR
Patrick Werner (Dec 01 2020 at 15:21):
what about: https://github.com/HL7/genomics-reporting/pull/29
Jamie Jones (Dec 01 2020 at 15:22):
Thanks! I think where I landed on the slicing wasn't quite there yet. I am testing the diagnosticreport changes but keep timing out on tx.fhir.org so I've slowed down a bit :joy:
Patrick Werner (Dec 01 2020 at 15:22):
the PR changes 108 files?
Jamie Jones (Dec 01 2020 at 15:23):
might be safest to do a manual review of the converted FSH files for pull 29. I split a coupe of the bundles up into atomic examples in my branch
Patrick Werner (Dec 01 2020 at 15:25):
I think we should remove all Bundle Examples and reference the atomic examples via their id.
Patrick Werner (Dec 01 2020 at 15:27):
did you discuss moving the ballot to march yesterday?
Jamie Jones (Dec 01 2020 at 15:28):
Isn't it May? Clem thinks we shouldn't delay ;) Realistically there is still a lot left to barely pass FMG's checks
Patrick Werner (Dec 01 2020 at 15:44):
thats my concern. FMG deadline tomorrow.
Things to avoid in IT: last minute changes before a deadline :smile:
Patrick Werner (Dec 01 2020 at 15:44):
and yes it's May
Jamie Jones (Dec 01 2020 at 15:54):
just pushed up DR changes and tidied slicing on category
Patrick Werner (Dec 01 2020 at 15:55):
weird: ignoreWarnings contains: The code "92822-6" is......
Patrick Werner (Dec 01 2020 at 15:56):
but still these errors are shown in the build
Jamie Jones (Dec 01 2020 at 15:56):
seems we can only suppress build errors, not validation on examples/SDs. that or i didn't suppress them right :)
Patrick Werner (Dec 01 2020 at 15:57):
:-|
Jamie Jones (Dec 01 2020 at 15:58):
i was really hoping we could suppress those failing LOINC refs, as it stops the validator from finding the right profile sometimes, and validates the instance against Observation itself :)
Patrick Werner (Dec 01 2020 at 16:04):
i just asked about it here: https://chat.fhir.org/#narrow/stream/179252-IG-creation/topic/Error.20with.20examples.20using.20external.20codes/near/218441910
Errors aren't supressable in the build, only warnings....
Patrick Werner (Dec 01 2020 at 16:09):
weird: http://tx.fhir.org/r4/CodeSystem/$lookup?system=http://loinc.org&code=92822-6
Patrick Werner (Dec 01 2020 at 16:09):
works
May Terry (Dec 01 2020 at 16:29):
weirdly, when I put that API call (http://tx.fhir.org/r4/CodeSystem/$lookup?system=http://loinc.org&code=92822-6) in Postman, I get the rendered webpage in return rather than a JSON.
Patrick Werner (Dec 01 2020 at 16:29):
you have to use postman and set the accept header to fhir+json
Patrick Werner (Dec 01 2020 at 16:34):
or add the &_format=jsonparameter to the url
Patrick Werner (Dec 02 2020 at 09:43):
Ok, the loinc errors are gone if you delete your tx-cache locally. Unfortunately the cache in the CI build is re-used every time and only manually deleted before publication/voting cycles.
Kevin Power (Dec 02 2020 at 15:34):
Do we have to delete the cache every time? Or just when we see errors like that? Seems like a very problematic cache strategy.
Patrick Werner (Dec 02 2020 at 17:06):
i removed the cache folder from the repository to fix the errors on the CI build.
Patrick Werner (Dec 02 2020 at 17:06):
i also added the txcache to the gitignore.
Patrick Werner (Dec 02 2020 at 17:07):
So please delete your local input-cache oder txcache as well
Patrick Werner (Dec 02 2020 at 17:08):
i updated .gitignore, ignoremessages as well
Patrick Werner (Dec 02 2020 at 17:14):
broken links = 0, errors = 163, warn = 78, info = 282
Patrick Werner (Dec 02 2020 at 17:14):
all loinc errors are gone now
Patrick Werner (Dec 02 2020 at 17:20):
All profiles without error, but still a lot of errors from the examples
Jamie Jones (Dec 02 2020 at 17:34):
I'm seeing a lot of Unable to resolve value set (from http://tx.fhir.org/r4) where we didn't before, that's probably timing out errors?
Jamie Jones (Dec 02 2020 at 17:35):
eg., for component[2] on http://build.fhir.org/ig/HL7/genomics-reporting/Observation-VariantExample2.json.html (also I love the new example view, now that most of the bugs around it are resolved!!)
Patrick Werner (Dec 02 2020 at 17:52):
investigating the valueSet errors
Patrick Werner (Dec 02 2020 at 17:54):
i still hate the huuuuuuge profiles due to the narrative. If we populate the narrative it won't get generated and the resources are ~50% the size
Jamie Jones (Dec 02 2020 at 17:55):
Would we want the downloadable defs to not have the narrative??
Jamie Jones (Dec 02 2020 at 17:55):
I've had trouble uploading them to servers :)
Patrick Werner (Dec 02 2020 at 17:58):
Yes, i would put the name of the profile in the narrative, thats it. The IG publisher always generates narratives if no narrative is present. Therefore i'd like to put in a minimal one.
Patrick Werner (Dec 02 2020 at 17:58):
Previous POST attempt returned HTTP<422> from url
Patrick Werner (Dec 02 2020 at 17:58):
ist the reason for the errors
Patrick Werner (Dec 02 2020 at 17:58):
hmmm
Patrick Werner (Dec 02 2020 at 18:03):
The coding producing the the error seems to be:
{
"coding": [
{
"system": "http://terminology.hl7.org/CodeSystem/observation-category",
"code": "laboratory"
}
]
}
Patrick Werner (Dec 02 2020 at 18:03):
which doesn't make sense
Kevin Power (Dec 05 2020 at 19:04):
Noticed something while doing a quick bit of QA today:
Variant profile, sample-allelic-frequency in STU1:
image.png
Variant profile, sample-allelic-frequency in current build (after I changed it from CodeableConcept back to Quantity):
image.png
Might need someone else to look at it, but from the rendering it doesn't appear to be limiting the valueQuantity.value to decimal, even with this in the FSH file:
* component[sample-allelic-frequency].code = LNC#81258-6
* component[sample-allelic-frequency] ^short = "Sample Allelic Frequency"
* component[sample-allelic-frequency].value[x] only Quantity
* component[sample-allelic-frequency].valueQuantity.value only decimal
Jamie Jones (Dec 05 2020 at 19:13):
Some errors with valueQuantity merit further investigation, I agree
Kevin Power (Dec 05 2020 at 19:17):
My change dropped the number of errors significantly. The FSH change had
component[sample-allelic-frequency].value[x] as CodeableConcept
which was wrong. When I switched it to Quantity, all the examples which used this component stopped generated a bunch of errors.
To be clear, I assume I didn't do something right in my 'fix' because it doesn't seem to limit valueQuantity.value to decimal only. Or least the rendered profile doesn't show that. We need to also have a fixed value on the system as well.
Patrick Werner (Dec 06 2020 at 14:09):
Did we decide to move away from Quantity here?
Patrick Werner (Dec 06 2020 at 14:09):
To decimal?
Patrick Werner (Dec 06 2020 at 14:11):
Because Decimal is not allowed in Obs.component. Would stick to Quantity
Kevin Power (Dec 06 2020 at 14:15):
Oh duh my bad. Just quantity is ok then. We just need to add the fixed value for system then to stay consistent with STU1
Patrick Werner (Dec 06 2020 at 14:16):
For which system?
Patrick Werner (Dec 06 2020 at 14:17):
PR for allelic freuqency:; https://github.com/HL7/genomics-reporting/pull/33
Kevin Power (Dec 06 2020 at 14:17):
unitofmeasure.org (I think is what I remember from STU1)
Patrick Werner (Dec 06 2020 at 14:18):
ah! yes. Will update my PR in a second
Patrick Werner (Dec 06 2020 at 14:27):
updated, now wit fixed system
Patrick Werner (Dec 07 2020 at 11:12):
ok spotted another minor issue in Variant: copy number is sliced, but doesn't need to be
Patrick Werner (Dec 07 2020 at 11:12):
https://github.com/HL7/genomics-reporting/pull/34
Jamie Jones (May 25 2021 at 21:11):
I'm noticing that our FSH build process is not populating some basic info on the trunk of our profile structuredefinitions like "short", "definition", or "comments" about the profile itself. These properties can be populated for elements like components, but for instance all of our Observation sdefs have this info inherited from base Observation rather than what is defined for that profile. Was investigating but local build seems to be broken for me currently due to publisher updates :)
Kevin Power (May 25 2021 at 21:29):
Interesting, do we need to ask on the #shorthand channel?
Patrick Werner (May 26 2021 at 09:06):
will have a look today
Jamie Jones (May 26 2021 at 14:49):
after digging a bit, it seems like the issue is that in the generated structureDefinitions, the 'snapshot' contains a first element called 'Observation' and FSH can't assign anything to that element
Jamie Jones (May 26 2021 at 14:57):
for ease of use, they strip the base type from the syntax, so assigning a "short" name to (for example code) is done via code ^short rather than Observation.code ^short. I can't find a way to assign a short to the base "Observation" element in the profile. It should just be ^short but the syntax doesn't understand that should point to the first elementDefinition in the snapshot and not the base of the profile's structureDefinition
Kevin Power (May 26 2021 at 14:58):
How does that affect the display as of today? What are you trying to put in there?
Jamie Jones (May 26 2021 at 15:04):
the first line of the tables show "Observation Measurements and simple assertions" rather than the profile's name, and if you navigate to Detailed Descriptions, you won't find the definition or name of the profile. This is true for any FSH-defined sdef as far as I can see, for example http://hl7.org/fhir/us/mcode/2021MAY/StructureDefinition-mcode-tumor-size.html, http://build.fhir.org/ig/HL7/genomics-reporting/StructureDefinition-variant.html, or http://build.fhir.org/ig/HL7/genomics-reporting/StructureDefinition-grouper.html but not our old profiles like http://hl7.org/fhir/uv/genomics-reporting/somatic-diagnostic.html
Kevin Power (May 26 2021 at 15:06):
Ahhhh, now I see.
To me, as long as we can have the definition in the table at the top of the page, I am OK with that one. But I think we could go FSHing on #shorthand and see if we get a byte. :smile:
Jamie Jones (May 26 2021 at 15:08):
I don't think it's the end of the world if that first elementDefinition is generic rather than specific to the profile, I'll draft something for the shorthand stream
Kevin Power (May 26 2021 at 15:10):
I would also like to understand who/what makes the determination of the 'attribute names', especially for extenstions:
image.png
We have quite a hodge-podge of naming conventions, and to be honest, it seems the name displayed isn't consistant.
Kevin Power (May 26 2021 at 15:19):
Actually, maybe it is consistant for extensions. When I see in our Obs profiles, espeically for our components, it uses slice name. However, for extensions, it seems to use the name of the extension profile. For example, the CGRelatedArtifact is NOT the slice name (which shows sliceName), but CGRelatedArtifact is the name of the profile:
snippet:
{
"id": "DiagnosticReport.extension:relatedArtifact",
"path": "DiagnosticReport.extension",
"sliceName": "relatedArtifact",
"short": "Related Artifact",
"definition": "Captures citations, evidence and other supporting documentation for the observation or report.",
"min": 0,
"max": "*",
"base": {
"path": "DomainResource.extension",
"min": 0,
"max": "*"
},
"type": [
{
"code": "Extension",
"profile": [
"http://hl7.org/fhir/uv/genomics-reporting/StructureDefinition/CGRelatedArtifact"
]
}
],
Kevin Power (May 26 2021 at 15:20):
But as the screen shot above shows, the attribute name gets rendered as CGRelatedArtifact.
Jamie Jones (May 26 2021 at 15:33):
i think all extensions are inherently slices on the resource's extension element? if that helps justify it...
Kevin Power (May 26 2021 at 15:34):
But at least in Obs.component where we slice that list, the tooling sticks what we put into 'sliceName' as the name. But for extension, it pulls the id of the profile (or at least from what I can tell).
Jamie Jones (May 26 2021 at 15:44):
(question answered on https://chat.fhir.org/#narrow/stream/215610-shorthand/topic/Assigning.20to.20a.20Profile's.20base.20elementDefinition) we are free to add shorts and defs back into our profiles at the top level
Patrick Werner (May 26 2021 at 17:04):
my day was packed. With the carret syntax hint by FSH we should be able to modify als fields as desired, correct?
Patrick Werner (May 26 2021 at 17:07):
but i am also getting a build error which looks familiar. We had the same in the VITU IG, cause was a sushi update
Patrick Werner (May 26 2021 at 17:07):
trying to fix the build atm
Patrick Werner (May 26 2021 at 17:10):
found the problem, we are assigning VS urls where CodeSystems should be used. As these locations are all having VS bindings it is enough to write: * status = #draftinstead of * status = PubStat#draft
Patrick Werner (May 26 2021 at 17:11):
i am also removing the then obsolete Aliases
Patrick Werner (May 26 2021 at 17:42):
down to 0 sushi errors
Patrick Werner (May 26 2021 at 17:46):
created a PR: https://github.com/HL7/genomics-reporting/pull/44
Patrick Werner (May 26 2021 at 17:46):
waiting for the CI build to run through
Kevin Power (May 26 2021 at 17:51):
Uh oh ...
http://build.fhir.org/ig/HL7/genomics-reporting/branches/pw-fix-build/failure/build.log
Kevin Power (May 26 2021 at 17:56):
Uh, OK? No error now?
http://build.fhir.org/ig/HL7/genomics-reporting/branches/pw-fix-build/build.log
Patrick Werner (May 26 2021 at 18:05):
yes. First commit was incomplete
Patrick Werner (May 26 2021 at 18:05):
merged the working branch. Build should be ok now.
Patrick Werner (May 26 2021 at 18:19):
also make sure to have the latest sushi version (1.3.2) Apr 23
May Terry (Jun 10 2021 at 22:48):
Answering my own question after all this time just for closure in case any relative newbie wants to know and searches Zulip for a similar question... to resolve this tx.fhir.org request in Postman, I needed to add the headers Content-type and Accept headers both set to application/json. :upside_down:
May Terry said:
weirdly, when I put that API call (http://tx.fhir.org/r4/CodeSystem/$lookup?system=http://loinc.org&code=92822-6) in Postman, I get the rendered webpage in return rather than a JSON.
Last updated: Apr 12 2022 at 19:14 UTC
