FHIR Chat · unable to track local build changes · committers/git-help

Stream: committers/git-help

Topic: unable to track local build changes

Eric Haas (Sep 12 2018 at 15:16):

As I feared, I made a change in a branch committed and pushed ( forgot to close excel and is all @#$ up but that is separate issue ) checkout back to master branch run the build and voila it is as if I never made a change. So how to I keep a local build copy of all the cumulative changes? so I can move forward without having to wait for a PR to be approved and updating the master before I can make the next incremental change?

Eric Haas (Sep 12 2018 at 15:56):

so if I run the build ( specifically the publish.sh bash) in the current branch it reflects the changes. if I checkout to the master branch and then run the build, it does not. Is this expected behavior?

Lloyd McKenzie (Sep 12 2018 at 16:10):

That's expected behavior. However, you should be able to merge the change from your branch into your local master

Eric Haas (Sep 12 2018 at 17:23):

You should be able to merge the change from your branch into your local master

How do I do that? ( by how I mean step by step command lines that i can literally copy and paste into my shel l )

What happens when I do a git pull origin master if I merge a branch with the master? will I get conflicts?

Geoff Low (Sep 12 2018 at 21:52):

master is usually considered a special branch; so usual pattern is to:
1. make your changes on a feature branch (which will be initially a copy of the master branch),
2. add any new files
3. commit the changes,
4. push the commits to the remote (GH in this case),
5. create a PR
6. merge the PR (this is often expected to be done by a third party under the guise of code review)
7. checkout master
8. git pull

If master is updated in the meantime, a git pull will reflect the changes upstream (it by default updates all local branches) ; you can then merge them into your feature branch by running a git merge master- here you might get conflicts if the changes you have made are on the same files as what other people have been updating; this can be painful, but there are some good tools to assist. W.r.t to running builds, by default the current branch will be the upstream version plus the changes you've made.

Geoff Low (Sep 12 2018 at 22:02):

To do a local merge:

$ git checkout master
$ git pull
$ git merge my-branch

1. The first checks out the master branch
2. The second runs a git pull to sync with the remote master
3. The third merges the commits from your feature branch into the current branch (master)
If there are conflicts, it will tell you here; I recommend a tool for managing the conflicts; the Jetbrains tools work pretty well.

There is a second command git fetch which just fetches the changes; ==> git pull does a fetch, followed by a merge

Grahame Grieve (Sep 12 2018 at 22:24):

IU

Grahame Grieve (Sep 12 2018 at 22:26):

I'm still far from understanding all the implicit stuff going on in the background, but my PRs initially claimed to containing past changes. I think that this happened because I was trying to merge my local changes into my local master to do what Eric's trying to do

Eric Haas (Sep 12 2018 at 22:29):

between this issue and the build error and the build using all my cpu I am giving up for a while and will try again when things are better sorted.

Grahame Grieve (Sep 12 2018 at 22:29):

here's a pretty funny page: https://www.quora.com/Why-is-Git-so-hard-to-learn - series of completely contradictory answers. The problems for me are all because the 'this is easy' answers don't deal with real world work flow - pretending that our work is a series of cleanly separated tasks. I especially like the 'learn git slowly' answers - as if git is actually our job

Grahame Grieve (Sep 12 2018 at 22:29):

giving up for a while

other committers are too scared to try. From my point of view, we have a serious problem here

Josh Mandel (Sep 13 2018 at 00:34):

I saw Eric's comment here, but I wasn't sure if it was related to Git in particular, vs a broken build.

Grahame Grieve (Sep 13 2018 at 00:35):

both, but Git is what changed.

Josh Mandel (Sep 13 2018 at 00:35):

What else are you hearing ? I want to 1) be responsive to serious issues, 2) allay any unnecessary fears, and 3) treat this as an experiment that that is reversible if we're not happy.

Josh Mandel (Sep 13 2018 at 00:36):

I just think it would be a shame to treat "I was too scared to try" as a deciding vote (vs, I tried and couldn't make it work)

Josh Mandel (Sep 13 2018 at 00:37):

For perspective, pretty much every serious software product now uses some flavor of distributed version control; it's not part like we're peddling a giant fad (IMO at least).

Grahame Grieve (Sep 13 2018 at 00:42):

well, I hear that committers are watching this and deciding not to contribute for now, in the hope that documentation will improve.

Grahame Grieve (Sep 13 2018 at 00:43):

I'm not hopeful on that front right now - the workflow I am following is really very complicated; the documentation you wrote is right in principle, but doesn't deal with our reality. I don't know how many committers interlace work the way that Eric/Lloyd/I do, but I suspect that most of us do.

Grahame Grieve (Sep 13 2018 at 00:45):

I understand that the general movement is in this direction, but I very much feel as though real issues are dismissed as FUD when they represent a significant increase in the workflow complexity - once you have figured out how to explain the issues in language that git users can understand.

Josh Mandel (Sep 13 2018 at 00:58):

Okay. Well I'm here to help, but I'm not going to push. Please let me know what I can do that's useful, beyond the experiment we're running with Mergify and the updated merge announcements we've configured on Zulip.

Grahame Grieve (Sep 13 2018 at 01:59):

well, what might help is :
- agreeing on 1 or a few UI enhanced clients that we document, and documenting for them. For instance, I think that most of us on windows use tortoisegit. command line documentation I'll be ignoring. I simply don't understand how you can meaningfully review changes in 100s of files using a command line tool
- documenting a workflow that supports maintaining a single local folder you work in, and committing sets of changes out of it as separate PRs (while maintaining the integrity of your local workspace)

Josh Mandel (Sep 13 2018 at 02:33):

Great! The first item @Bryn Rhodes has agreed to help with. I've not had much luck personally with GUIs for git or svn, but I know this is very much a matter of personal preference.

Josh Mandel (Sep 13 2018 at 02:34):

The second item is a sticking point if I'm not mistaken -- unless you've seen a solution somewhere in our recent discussion/exploration.

Grahame Grieve (Sep 13 2018 at 02:34):

it's really weird how there's no one workable single UI

Grahame Grieve (Sep 13 2018 at 02:34):

the second item is critical; we can at least document a least worst practice

Bryn Rhodes (Sep 13 2018 at 02:34):

I use TortoiseGit and am happy to write docs for that, I'll work on that.

Josh Mandel (Sep 13 2018 at 02:35):

What is the least worst? I'm not sure I've seen one that meets your criteria.

Bryn Rhodes (Sep 13 2018 at 02:35):

However, I've never had any success with merging my local branches that I've pushed and then sync'ing. That's a recipe for really difficult to resolve conflicts, it seems to me.

Bryn Rhodes (Sep 13 2018 at 02:36):

I've always pushed, then merged to master, then pulled from master.

Bryn Rhodes (Sep 13 2018 at 02:36):

Is the mergify going to auto-merge commits?

Bryn Rhodes (Sep 13 2018 at 02:37):

er, I mean PRs?

Grahame Grieve (Sep 13 2018 at 02:37):

actually, the most practical idea I've seen yet is to have a sibling folder you do git from. you commit from that. but that has it's own problems

Josh Mandel (Sep 13 2018 at 02:37):

Only in an opt-in fashion for now, but that's what we're looking into.

Grahame Grieve (Sep 13 2018 at 02:37):

mergify: yes, for approved editors. at this time, I'm the only approved editor while we test

Josh Mandel (Sep 13 2018 at 02:38):

For the sibling folder idea, how do you manage changes between the main folder and the sibling folder?

Josh Mandel (Sep 13 2018 at 02:38):

Are they both git clones, or just the sibling is a git clone?

Grahame Grieve (Sep 13 2018 at 02:40):

well, I think there's problems, and I haven't tried it, but if I tried, here's what I'd do: tie my working copy to master. Only sync into it from master. Have a batch file that copies everything from the working copy to my git-commit copy. Use that after I switch that git-commit copy to the branch I'm going to commit to. Resync it after committing

Grahame Grieve (Sep 13 2018 at 02:41):

it just seems like so much work compared to the really simple pattern with svn or git when committing to the master directly: choose a set of changes, and commit them.

Grahame Grieve (Sep 13 2018 at 02:41):

I feel as though git should be able to do this: choose a set of changes, and commit to a clone of the master, and tell me if that's a problem

Lloyd McKenzie (Sep 13 2018 at 03:02):

A slightly crazy idea here - what if each individual had their own branch (and only one). When you pushed, if the branch built, it would auto-merge. You could commit and push at will, always working in one branch. No switching. No stashing. But still having the ability to test before merging. Is that viable/technically possible? We'd have 40-50 branches, though probably only 10 or so that would be terribly active. I'm sure this notion will make git afficianados cringe (and I do get why), but what I'm interested in is maximum simplicity, minimum pain and getting the one and only thing we really looking for from Git - not breaking builds. (And I think that we ought to also be able to get the "changes must be reviewed if they're impacting normative content" capability with this mode too which was the major "nice-to-have" we'd been hoping for.)

Bryn Rhodes (Sep 13 2018 at 03:17):

Another option would be to consider a "develop" branch where committers can commit directly, and commits would be either manually or automatically (on some gate) merged to master.

Lloyd McKenzie (Sep 13 2018 at 03:18):

The challenge there is that every developers commits impact everyone else. If I push something to the developer branch that breaks building the developer branch, everyone's twiddling their thumbs until that change gets fixed.

Lloyd McKenzie (Sep 13 2018 at 03:20):

If everyone has their own branch, they can only break themselves - and are on the hook for fixing themselves. Only downside would be someone who pushes something and doesn't notice that the build failed. They think they're done, but their stuff never got migrated. So we might need to set up something to ping people occasionally (every day? every week?) if their committed branch is unmerged.

Grahame Grieve (Sep 13 2018 at 04:14):

what if each individual had their own branch (and only one). When you pushed, if the branch built, it would auto-merge

Well, I'll give it a go. but how does it get synced against the master for other people's changes?

Lloyd McKenzie (Sep 13 2018 at 04:17):

When you pull, you'd want to pull from the master. I presumed that was possible, but don't know for sure.

Geoff Low (Sep 13 2018 at 12:07):

Just as a query, what editors are people using? I use Jetbrains (PyCharm, IntelliJ, etc) and the tooling for creating, pushing, merging and resolving conflicts works pretty well.

Geoff Low (Sep 13 2018 at 12:31):

I think the theory is that CI would help Lloyds case; if he does his work in isolation on a feature branch; pushes the branch to GH, and you have travis-ci running on the PR with merging being blocked until the build passes (on your feature branch), then develop after someone merges the PR will still be stable and working. Anyone pulling from develop at that time would be guaranteed to have a working branch.

Features are aimed at change requests, so for example you'd have a branches called:

• feature/forge_101_add_valueset_to_patient
• fix/forge_102_amend_status_description_for_patient

These branches which would be merged into develop via a PR, that if gated by a status check will prove that the develop branch is always in a working state.

When someone takes on a new task (Say FORGE-103), they would:
1. git checkout develop
2.git pull
3. git branch feature/forge_103_add_coding_decision_workflow (which creates a copy of develop under the new branch)
4. git checkout feature/forge_103_add_coding_decision_workflow
5. do some work, commit your changes
6. locally run the build on the feature branch; if it passes, move to the next step - if it doesn't then the issue lies with changes on the branch, which need to be fixed before moving on
7. git push origin feature/forge_103_add_coding_decision_workflow
8. Create a PR - NOTE GH will warn you if you are out of date so you can mitigate locally before pushing again
9. Wait for the CI status checks to pass (which they should, because you ran them locally) and then someone will merge into develop
10. Goto 1.

You can of course replace all of my references to develop by master but the concept remains the same. Change Requests are tracked by the branch and pull requests. You could have multiple clones of the repository, and in doing so switch contexts to work on another task while the build is running on the current (ie /dev/fhir1 /dev/fhir2), but that could get confusing fast. So long as you follow the git pull then git branch you should be able to stay sane as develop in the repository will always be stable; the nightly build would run off the develop branch and per the contract will always be in a working state.

well, that's the theory

Josh Mandel (Sep 13 2018 at 13:50):

I think that theory is exactly what we're doing with git this week (on "master" rather than "develop") and exactly what seems to be causing some angst.

The key challenges we're hearing are: 1) extra steps compared with the subversion workflow, and 2) the inability to change branches while actively running a debugger.

Rob Hausam (Sep 13 2018 at 14:46):

I think that Lloyd's idea might work. Each person can create and work with and push from their own 'develop' branch (like 'rhausam-develop'). I think that could be managed locally, and wouldn't necessarily need to be prescribed or enforced (particularly if some committers prefer a different workflow). If the local master branches are kept clean, then they can always be fast-forward updated (without conflicts) by git pull and then those updates can be merged into the 'develop' branch via git checkout 'local-develop-branch' and git merge master.

Lloyd McKenzie (Sep 13 2018 at 15:02):

We lose semanticly meaningful branches (at least for the people who follow this workflow), but we never had it before and it wasn't something we were looking to gain with the transition to Git. If someone can reflect how to do this properly using TortoiseGit, I'd be willing to apply a couple of change requests to see how it works. :)

Rob Hausam (Sep 13 2018 at 15:06):

I use the command line and haven't tried TortoiseGit yet, so I can't help with that.

Geoff Low (Sep 13 2018 at 15:07):

Ok, and is the workflow task based? i.e. for the next hour I'm fixing one thing, or adding one feature?
Is the prior development on an rolling HEAD basis - be it master or $(USER)_develop?

Geoff Low (Sep 13 2018 at 15:08):

I'm installing tortoise git, I can try and put some screenshots together and share a document

Lloyd McKenzie (Sep 13 2018 at 15:15):

The standard workflow for Grahame, myself and quite a few others is "I'm fixing a bunch of stuff simultaneously. Some of it is done and other stuff won't be done for hours or days. The stuff that's done, I want integrated into the mainline now (because someone's waiting on it). We don't create different branches for different features because we can only run one build locally and we can't wait for a build to finish before we work on other stuff.

Lloyd McKenzie (Sep 13 2018 at 15:16):

For people who only work on one thing at a time and don't mind waiting for a 30-45 minute build to complete before they commit, push and switch, a more traditional workflow is fine.

Bryn Rhodes (Sep 13 2018 at 15:36):

@Geoff Low I'm working on this right now, just want to make sure we're not duplicating effort...

Geoff Low (Sep 13 2018 at 15:40):

Cool, I have some examples of creating a PR, followed by a failing status check, and then passing if that helps

Geoff Low (Sep 13 2018 at 15:42):

from https://travis-ci.org/synthetichealth/synthea/jobs/428219650

Bryn Rhodes (Sep 13 2018 at 16:50):

Okay, I've made it to the Pull Request step. I've got a submitted PR, once that's merged to master, I can finish up the "simplest case" commit scenario: https://github.com/HL7/fhir/wiki/Using-TortoiseGit-with-FHIR

Bryn Rhodes (Sep 13 2018 at 16:50):

Comments welcome on the content, it makes sense to me, but I'm fairly bad at explaining things, I think.

Lloyd McKenzie (Sep 13 2018 at 17:10):

I really don't like the notion of having to go to the website to create a pull request. Why is it hard to do that with tortoise? (Should also highlight that a Git commit is different than an SVN commit)

Bryn Rhodes (Sep 13 2018 at 17:17):

The TortoiseGit interface just doesn't default the information, so you have to go look it up. It also doesn't provide all the nice Github stuff like being able to request a reivewer, etc.

Bryn Rhodes (Sep 13 2018 at 17:17):

Honestly, for me, Github is what makes Git usable. If I wasn't able to use Github, I wouldn't use Git.

Lloyd McKenzie (Sep 13 2018 at 17:30):

Having to go to an external website in order to use source control doesn't sound super-useable to me...

Bryn Rhodes (Sep 13 2018 at 17:48):

You don't have to, it's just easier :)

Lloyd McKenzie (Sep 13 2018 at 18:20):

"Going to an external website is easier" is an interesting notion of "useable"...

Geoff Low (Sep 13 2018 at 20:10):

According to https://tortoisegit.org/docs/tortoisegit/tgit-dug-patch.html#tgit-dug-patch-pullrequest - you can create the PR directly from TortoiseGit.

You then just need to get someone to merge it through the website; although github has a pretty nice API

Lloyd McKenzie (Sep 13 2018 at 20:17):

Once it's set up properly, the merging should happen automatically for most things (and for other things, someone else would need to review and do the merge.)

Geoff Low (Sep 13 2018 at 20:17):

once the branch is pushed to GH, anyone can check it out. So if they need it urgently, so long as you have pushed it they can checkout your working branch locally and carry on; at some point in the future your changes will be merged into the master branch. But the master/develop branch stays stable, and working

Geoff Low (Sep 13 2018 at 20:32):

Docs look good; Couple of comments:

• what is the vscache used for?
• do you need to mention a git pull before checking out masterin the Switch back to master section

Bryn Rhodes (Sep 13 2018 at 20:44):

My intent is to show that pull after the PR is approved, so pulling the feature branch changes that are now in the remote master into your local master, so I was waiting to talk about that until that PR is merged and I can grab the screenshots of that happening.

Bryn Rhodes (Sep 13 2018 at 20:45):

The vscache is the terminology cache the build uses so it doesn't have to expand everything every time.

Geoff Low (Sep 13 2018 at 22:04):

ok, I was referring to the fact that you need to do a git pull after the PR merge; otherwise your changes (plus the merge commit) won't have been retrieved from the remote prior to checkout

Bryn Rhodes (Sep 14 2018 at 15:23):

So, I thought that's what I was saying with this:

NOTE: This means the changes you just made on your feature branch are no longer in your working directory, they are in a pull request that needs to be reviewed and merged into master. Once the change has been merged to the master branch, use "Pull" to get those changes into your local repository's master branch.

I'm concerned that either I'm not understanding what you're saying or I haven't explained that well in the documentation. Thoughts?

Geoff Low (Sep 16 2018 at 22:07):

Ok, I misread - the commandline tool just does a checkout (without the pull). From the docs I now see that tortoisegit seems to do a fetch and merge from the origin at the same time. Please ignore my comment.

Last updated: Apr 12 2022 at 19:14 UTC