Stream: smart/scheduling-links
Topic: One-liner to fetch
Josh Mandel (Apr 16 2021 at 19:22):
export BULK_PUBLISH='https://raw.githubusercontent.com/jmandel/wba-appointment-fetch/gh-pages/$bulk-publish'
export PARALLEL_CONNECTIONS=16
wget -qO- $BULK_PUBLISH | tee 'manifest.json' | jq -r '.output | .[] | .url' | xargs -n 1 -P $PARALLEL_CONNECTIONS wget -q
For me this fetches 211 files with 28k resources in ~5 seconds.
(Edit to fix back-ticks, add teeto save manifest file.)
Robert Scanlon (Apr 16 2021 at 19:44):
Thanks for sharing, this is great
Last updated: Apr 12 2022 at 19:14 UTC
