Stream: connectathon mgmt
Topic: github api questions
Sean McIlvenna (Sep 15 2019 at 17:38):
Is there anyone here at the connectathon that is familiar with the GitHub Data API? I have a problem where I'm attempting to commit too many files to GitHub at one time, and GitHub is flagging my requests as "suspected abuse". Hoping someone here at the Connectathon is familiar enough with it that they can discuss a better way of accomplishing what I'm trying to do...
Abbie Watson (Sep 15 2019 at 17:42):
Have you done a directory analysis? Usually when that happens you're checking in log files or maybe a dependency cache. You can add those directories to your .gitignore file to keep the overall number of files down.
Sean McIlvenna (Sep 15 2019 at 17:58):
I'm currently relying on the GitHub API to determine what has changed, which means I need to get ALL the files on GitHub's servers first, and then have GitHub build the change tree from the blobs.
Abbie Watson (Sep 15 2019 at 18:08):
Try a network monitor and check whether your query to the Data API is in the wrong scope and firing repeatedly?
Sean McIlvenna (Sep 15 2019 at 18:20):
no.. the query (POST) is good
Sean McIlvenna (Sep 15 2019 at 18:20):
but, there's too many
Sean McIlvenna (Sep 15 2019 at 18:21):
I have over 400+ files to create blobs out of
Sean McIlvenna (Sep 15 2019 at 18:21):
and after 100 requests, github rejects me for suspected abuse
Sean McIlvenna (Sep 15 2019 at 18:22):
I'm told @Josh Mandel might have some ideas about this issue
Josh Mandel (Sep 15 2019 at 18:32):
Not sure who mentioned that -- I haven't used this API, but @Dan Gottlieb has, a bit, through a browser-based app ;-)
Josh Mandel (Sep 15 2019 at 18:32):
What's your use case btw for submitting these files through the GH API vs via git?
Sean McIlvenna (Sep 15 2019 at 18:34):
The use case is for having #trifolia-on-fhir export data from a FHIR server to a GitHub repository.
Sean McIlvenna (Sep 15 2019 at 18:34):
All of the logic is in the client-side (browser) application
Sean McIlvenna (Sep 15 2019 at 18:35):
So, we don't use git.exe and don't have access to a file system even if we wanted to
Sean McIlvenna (Sep 15 2019 at 18:35):
All of the data is in memory...
Josh Mandel (Sep 15 2019 at 18:35):
Understood -- yeah, Dan has the same use case (but not thousands of files)
Sean McIlvenna (Sep 15 2019 at 18:35):
For commits with < 100 files, this approach has worked great
Sean McIlvenna (Sep 15 2019 at 18:35):
as soon as we throw in the "framework" directory, it fell apart
Sean McIlvenna (Sep 15 2019 at 18:36):
Is @Dan Gottlieb at Connectathon? If so, can you introduce?
Josh Mandel (Sep 15 2019 at 18:37):
He is, through this afternoon I think -- generally hanging around the Bulk Data table. (I'm in breakouts for the next 90min or so)
Sean McIlvenna (Sep 15 2019 at 18:50):
spoke to him... ultimately, he's not committing as many files as I am and hadn't experienced that problem
Sean McIlvenna (Sep 15 2019 at 18:50):
however, he gave me an idea for a partial solution
Sean McIlvenna (Sep 15 2019 at 18:51):
still, unfortunately, not going to work in all use-cases
Sean McIlvenna (Sep 15 2019 at 18:51):
so, still searching for a full solution
Sean McIlvenna (Sep 15 2019 at 19:24):
@Dan Gottlieb https://developer.github.com/v3/git/trees/#create-a-tree
Sean McIlvenna (Sep 15 2019 at 19:24):
I think maybe this would be the answer to my problem...
Sean McIlvenna (Sep 15 2019 at 19:33):
creating a tree allows me to specify multiple files/paths/content
Last updated: Apr 12 2022 at 19:14 UTC
