Stream: IG creation
Topic: Batch files
Grahame Grieve (Mar 27 2020 at 05:52):
Many editors that run on windows use batch files, typically named _genOnce.bat or something to get the latest copy of the igpublisher, and then run the ig publisher
Grahame Grieve (Mar 27 2020 at 05:52):
Lloyd has introduced those batch files into the base template as a way to distribute them but this is a problem - I can't build anything because my a/v is making all the tools blow up when the try to handle the file
Grahame Grieve (Mar 27 2020 at 05:53):
and I can't really fault it - downloading executable code by a batch file sounds like a really insecure way to do things
Lloyd McKenzie (Mar 27 2020 at 06:05):
I think Grahame's question is - is there an alternative way to handle launch other than batch files that would work on all Windows environments and would be less likely to upset AV software
Vassil Peytchev (Mar 27 2020 at 13:32):
Git for windows includes a Bash shell. Can shell scripts be used instead? On my machine, the location is C:\Program Files\Git\git-bash.exe if you need to check that it exists.
Lloyd McKenzie (Mar 27 2020 at 14:09):
Is Bash better than .bat from an AV/security perspective? The biggest challenge is that we need something we can double-click on in the Windows explorer folder that will "just run". I expect that would be true of a .sh, so we'd need a .bat to invoke it (though it would mean we could put all the logic in the .sh file and have the .bat file be super simple & static). Are we confident that everyone who's installed any sort of Git client on a Windows machine will always have git-bash in that location?
Rik Smithies (Mar 27 2020 at 15:11):
can't we just exclude certain folders from A/V scanning?
Grahame Grieve (Mar 27 2020 at 18:10):
many people cannot exclude any folders - corporate managed
Jens Villadsen (Mar 27 2020 at 18:12):
couldn't people that have such challenges rely on the pipeline currently set up using webhooks on github?
Lloyd McKenzie (Mar 27 2020 at 18:15):
There's a decent number of people who need to be able to test when they don't have an internet connection. Also, they don't always want to commit their stuff every time they make a change and want a build. Some users of the tool chain won't use the CI build at all.
John Moehrke (Mar 27 2020 at 18:18):
Jens, this is possible and is used. I have had IG editors that only used that, never had build local ability. As Lloyd points out, there is still a need for local builds, and making them better should always be on the table
Jens Villadsen (Mar 27 2020 at 18:23):
sure ... as I understand it, the challenge is that the tool is used by a very broad set of people
Jens Villadsen (Mar 27 2020 at 18:23):
omitting bat/sh files makes it easier for some while it makes it a bit more difficult for others
Jens Villadsen (Mar 27 2020 at 18:24):
what about putting the sh/bat files in another repo?
Grahame Grieve (Mar 27 2020 at 18:24):
what is it that the batch files do?
Lloyd McKenzie (Mar 27 2020 at 18:33):
The batch files (and shell scripts) do three things: The first is responsible for downloading the current IGPublisher and putting it into either a project-specific folder or a parent folder (to allow sharing amongst a bunch of IGs). It does this in a way that supports a variety of Windows versions. The second checks to see where the IGPublisher is located and whether the system is connected to the internet or not and then launches the publisher with an appropriate tx server flag based on whether it's connected or not. It also ensures Java uses UTF-8. The final one does the same as the second, but launches the publisher in 'continuous' mode.
Grahame Grieve (Mar 27 2020 at 19:20):
@Josh Mandel I'm interested in your comment on this thread please
Josh Mandel (Mar 27 2020 at 19:29):
Re: packaging and setup, one end of the spectrum is to release managed packages per platform, like going through the Windows Store, getting developer keys, signing release, etc. But that doesn't sound like what's required here...
Josh Mandel (Mar 27 2020 at 19:30):
I'm unclear on why your tools are blowing up, though Grahame. If there's a setup.sh file (or whatever), why are you loading or running this when you go to build an IG? I would have thought that's a one-time developer-facing script that you run at the beginning of a process?
Grahame Grieve (Mar 27 2020 at 19:53):
I was simply copying it when installing the template package into the package cache. the A/V was preventing me from even having the batch file on my drive
Josh Mandel (Mar 27 2020 at 20:57):
But does not having the file break your build? I wouldn't think you need this file.
Lloyd McKenzie (Mar 27 2020 at 20:59):
The file is what kicks off the build
Grahame Grieve (Mar 27 2020 at 20:59):
failing to install a package breaks everything. I won't tolerate dropping files at random from packages and having mystifying downstream errors
Lloyd McKenzie (Mar 27 2020 at 21:00):
Wrong thread?
Grahame Grieve (Mar 27 2020 at 21:01):
no the right thread. I'm explaining what the problem with the batch file was - it was in a template, and my A/V wouldn't let it exist on my drive. I didn't want to run it. Only instal the package that contained it
Josh Mandel (Mar 27 2020 at 22:27):
Hmm. What does it mean to install a package? Is it just cloning the git repository or something more?
Lloyd McKenzie (Mar 27 2020 at 22:36):
What had happened was as follows: We had added the .bat files into the IG template, which caused them to be incorporated into an NPM package. The IGPublisher uses its NPM package manager to retrieve the package, stick it in the local cache, then extract the contents into a local 'template' folder for use in IG execution. Some part of that process caused Grahame's AV to be unhappy and yank some of the content, which caused other parts of the process to not copy the full set of content into the template folder, which led to a failure of the IGPublisher.
We've yanked that part of the process and will (shortly) be exploring a process where the .bat files try to update themselves directly from the source sitting in Git somewhere else. That'll avoid dealing with NPM or package extraction, though it's possible the process will still make Grahame's AV unhappy - hope to know that soon.
Josh Mandel (Mar 28 2020 at 02:42):
I'm so confused. Probably because I don't know where/why IG authors (or just template authors?) are using these .bat files, and why they need to "update themselves". Also regarding the underlying issue: Grahame, can't you tell your antivirus software not to complain about this? Basically every npm package anywhere (like, in the public npm registry) contains executable (js) scripts, shell scripts, etc. This can't all be broken, right?
Lloyd McKenzie (Mar 28 2020 at 03:14):
Authors are using the batch files to grab the most recent publisher and to kick off a local build. We want to auto-update them because sometimes we make enhancements to how the batch files function. (This conversation started, in part, because we were considering using the batch/shell files to do some FSH pre-processing before Grahame revamped the IG publisher to make that unnecessary. There will undoubtedly be future changes. Right now, we have a straight-forward process of making sure people are running with the current publisher and current templates, but there's no good way to make sure they're running with the current batch/shell files. The objective is to try to fix that.
Grahame Grieve (Mar 28 2020 at 04:10):
Grahame, can't you tell your antivirus software not to complain about this?
yes, well, in principle I could though it's getting harder to. but I actually had some conceptual sympathy for the AB here - a batch script that downloads itself from an internet server....
The real issue is that not everyone can configure their anti-virus like that
Jose Costa Teixeira (Mar 28 2020 at 05:45):
But with the bat files out of the template the problem is gone, right?
Lloyd McKenzie (Mar 28 2020 at 06:06):
Maybe. Haven't had a chance to test yet. You still have bat files updating bat files, which it's possible some AV systems may be unhappy with
Grahame Grieve (Mar 28 2020 at 06:18):
I might need to issue a signed executable to do the updating
Lloyd McKenzie (Mar 28 2020 at 06:38):
We'll see. If it's too big a pain to make work, we can just badger people on #committers/announce to download the new ones from sample-ig or wherever. First, let's see if a direct download causes grief.
Lloyd McKenzie (Mar 28 2020 at 06:39):
The problem with a signed executable is then you have the issue of ensuring people have an up-to-date signed executable :)
Jose Costa Teixeira (Mar 28 2020 at 06:46):
Lloyd McKenzie said:
The problem with a signed executable is then you have the issue of ensuring people have an up-to-date signed executable :)
...and for that (updating the executable) we can make a batch file :) <-- this should have been a joke
Jose Costa Teixeira (Mar 28 2020 at 06:46):
Lloyd McKenzie said:
Maybe. Haven't had a chance to test yet. You still have bat files updating bat files, which it's possible some AV systems may be unhappy with
Obviously my AV is very lax, but it does not complain about the self-updating batch files.
Lloyd McKenzie (Mar 28 2020 at 06:51):
We'll test it w/ Grahame and then stress test it w/ @John Moehrke :)
Jens Villadsen (Mar 28 2020 at 09:13):
If the only/primary concern is to have the most recent version of the IG publisher - then consider https://stackoverflow.com/questions/4002462/how-can-i-write-a-java-application-that-can-update-itself-at-runtime or something alike. There are a couple of tools that does this kind of thing
Jens Villadsen (Mar 28 2020 at 09:14):
eg. https://github.com/update4j/update4j
Grahame Grieve (Mar 28 2020 at 11:06):
i'm not reproducing OSGi
Jens Villadsen (Mar 28 2020 at 11:35):
I don't believe you are
Jens Villadsen (Mar 28 2020 at 11:41):
IMHO - if you want to distribute software that ensures that you run on the latest stuff, either use docker, an msi installer (with embedded updater) or something like https://github.com/update4j/update4j. You could also embed into the publisher that it advises you to stop the current run if there's an internet connection and it can detect a newer version online. If it is crucial business logic, put it into the publisher itself. - ... just a thought
Jose Costa Teixeira (Mar 28 2020 at 12:41):
Installer is machine-wide.
Jose Costa Teixeira (Mar 28 2020 at 12:42):
We need to support several IGs in the same machine, some of them using different versions of the publisher.
The batch files allow us to use a "shared" publisher jar for several IGs, or individual jars for some IGs.
Jose Costa Teixeira (Mar 28 2020 at 12:43):
They are a simple click-and-go solution, which we can zip with along with the IG, and tell a non-techie (after they install jelkyll...) - "here, unzip this, click that, then that, and watch the output folder."
I thought of a Launcher as well, but if the current solution works...
Jens Villadsen (Mar 28 2020 at 13:41):
Jose Costa Teixeira said:
We need to support several IGs in the same machine, some of them using different versions of the publisher.
The batch files allow us to use a "shared" publisher jar for several IGs, or individual jars for some IGs.
sounds a bit contradicting. How can a "shared" publisher be different versions of the publisher?
Jens Villadsen (Mar 28 2020 at 13:42):
have there been cases for using older versions of the publisher?
Jose Costa Teixeira (Mar 28 2020 at 14:08):
In my machine I use the latest publisher version for all IGs, but when something goes wrong -or when I want to test a specific version of the publisher in one of the IGs, then I use a different jar for that IG, while all the others use the "main" publisher
Jens Villadsen (Mar 28 2020 at 14:22):
do many other users share the same need to be able to run a multitude of different versions of the publisher?
Lloyd McKenzie (Mar 28 2020 at 14:41):
I'd say about 2/3 of people publishing only have one IG. But a small percentage of folks work with multiple IGs - and supporting different publisher versions is an uncommon but necessary capability for those folks.
Jose Costa Teixeira (Mar 28 2020 at 14:42):
And we don't need to have N times 100 mb to download and use disk space. One shared jar handles that
Jose Costa Teixeira (Mar 28 2020 at 14:45):
Btw, this is why I personally don't really care about a test branch of the IG Publisher. I can keep building locally by fetching the n-1 version of the jar ( @Mark Kramer not sure if this helps)
Jens Villadsen (Mar 28 2020 at 14:46):
i guess my own use is not representative for the average user then as I don't lack bandwidth nor diskspace and prefer to wrap it in docker an usually only use the latest version as I have no expectations of feature fixes back in time of the IG
Last updated: Apr 12 2022 at 19:14 UTC
