Faster bulk downloads from Internet Archive with aria2

A simple and quick way to download a large collection of files from an internet archive collection.

Screenshot of a file listing on the Internet Archive website
Screenshot of an Internet Archive file listing

First create a list of files to download by navigating into the file listing of the archive then grab the URL.

Next, use a combination of curl, grep and sed to extract the download links to a file.

curl -s URL | grep -oP 'archive\\.org\\/download[^"]*' | sed -e 's/^/https:\/\//'

Now either use wget -i file or for much faster parallel downloads use aria2.

aria2c -j4 -x8 -c -i file

-j4 Number of files to download at one time

-x8 Number of connections to the server for each file, x2 for each file

-c Continue on if download is stopped or paused

-i List of files to download, one per line