Collects media files from mods/games and puts them in a media directory, also creates an index.mth file in the MTHS format

collectstatic.sh

#!/bin/bash

###### Options ######
MINETESTDIR=/home/username/minetest
GAMENAME=minetest_game
WORLDDIR=none
MEDIADIR=./media
LEGACY_SYMLINKS=0

# When settings this up be aware that the Minetest client
# will send a POST request to index.mth. You need to configure
# your web server to serve the file on POST requests for this to work.
#####################

die () {
	echo "$1" >&2
	exit 1
}

collect_from () {
	echo "Processing media from: $1"
	find -L "$1" -type f -name "*.png" -o -name "*.ogg" -o -name "*.x" -o -name "*.b3d" | while read f; do
		basename "$f"
		hash=`openssl dgst -sha1 <"$f" | cut -d " " -f 2`
		cp "$f" $MEDIADIR/$hash
		[ $LEGACY_SYMLINKS -eq 1 ] && ln -nsf $hash $MEDIADIR/`basename "$f"`
	done
}

[ -d "$MINETESTDIR" ] || die "Specify a valid Minetest directory!"
which openssl &>/dev/null || die "OpenSSL not installed."

mkdir -p $MEDIADIR

[ ! $GAMENAME == none ] && collect_from "$MINETESTDIR/games/$GAMENAME/mods"
[ ! "$WORLDDIR" == none ] && collect_from "$WORLDDIR/worldmods"
[ -d "$MINETESTDIR/mods" ] && collect_from "$MINETESTDIR/mods"

printf "Creating index.mth... "
printf "MTHS\x00\x01" >$MEDIADIR/index.mth
find $MEDIADIR -type f -not -name index.mth | while read f; do
	openssl dgst -binary -sha1 <$f >>$MEDIADIR/index.mth
done
echo "done"

This is awesome, thanks! Wondering is there a way you could modify it to leave the file extensions intact? Naming as a hash is fine, but by removing the .png extension (for example) this breaks a proxy cache, which could be leveraged to hugely improve the performance of the given server.

For example, a .png file will be cached by the proxy based on the filename something**.png** - but naming it instead 1b2e0ad7a04c551be776c3c0ed851c30 without a file extension makes it so the proxy cache just passes through the file every time since it does not know it is actually a png... so every request results in a pull from the server, rather than the cache. That also means the user is accessing the file from the main server, rather than a CDN node.

Naming the file instead, 1b2e0ad7a04c551be776c3c0ed851c30**.png** would lead the proxy to cache it locally. In the case of a CDN (which is our case) that would mean the file would be served from a local server to the player and not pulled from our main server.

Would love to see this changed so we can leverage the proxy cache on our servers!!

Thanks!
RobbieF

No, that's not possible. Minetest looks for the files with just the sha1 hash, adding file extensions would break it.

Can't you tell your CDN to forcefully cache everything in a single folder?

Depends on which cache I put it on, but the one I would like to use requires extensions as that's how it determines the cache policy to use.

Is it possible to have the hashed file + the index.mth on different servers? Or does the mht file have to be located in the same folder as all the files? This is also a problem since the ideal cache server doesn't allow POST operations (static file serving only).

Nope, both files need to be at the same path.

K thanks sfan5. Not perfect, but still will help offload the game server itself.

I made this as an alternative: https://github.com/ShadowNinja/mt_media_collector

I'd like to edit this a little and share - what license do you wish to detail please?