Takes a large image as the input, outputs map tiles at the appropriate size and file structure for use in frameworks like leaflet.js, MapBox, etc.

GenerateSlippyMapTiles.py

# -*- coding: utf-8 -*-

import argparse, magic, re, os, math, glob, shutil

'''
GENERATE SLIPPY MAP TILES
Jeff Thompson | 2016 | jeffreythompson.org

NOTE!
@danizen has an updated version for Python 3:
https://github.com/danizen/campaign-map/blob/master/gentiles.py

Takes a large image as the input, outputs map tiles
at the appropriate size and file structure for use
in frameworks like leaflet.js, MapBox, etc.

ARGS:
input_file		large image file to split (JPG, PNG, or TIFF)
zoom_level          	zoom level(s) to generate (0 to 18); either 
			integer or range (ex: 2-6)
output_folder       	folder name to write tiles to (will be created 
			if does not exist)

OPTIONAL:
-h, --help          	show this help message and exit
-w --resize_width	dimension in pixels for outputted tiles (default 256px)
-q, --quiet         	suppress all output from program (useful for 
			integrating into larger projects)

DETAILS:
Resulting tiles are 256px square, regardless of the
size of the source image. The number of tiles wide/
high is determined by the "zoom level", which is
2^zoom. In other words, a zoom level of 3 = 8 tiles,
each resized to 256 pixels square.

Way more info here:
http://wiki.openstreetmap.org/wiki/Slippy_map_
tilenames#Resolution_and_Scale

REQUIRES:
ImageMagick and Python bindings for splitting 
images, resizing tiles, etc
http://www.imagemagick.org
https://github.com/ahupp/python-magic

FILE STRUCTURE
Slippy maps require tiles to be stored in a specific
file structure:
	output_folder/zoom_level/x/y.png

This is the standard arrangement (some frameworks let
you specify others), and should be noted in your Javascript.
For example, if using leaflet.js, you would use:
	tiles/{z}/{x}/{y}.png

ADDING MORE ZOOM LEVELS
Want to add more levels? Just run this script again; it 
will append the new zoom level to the same location.

CREATING A SOURCE IMAGE
If combining many smaller images, the easiest method
is to use ImageMagick's 'montage' command.

Your images should be the same size, or at least the
same height. You can do this using ImageMagick as well:
	mogrify -geometry x400 *.jpg

Arguments:
	x400			height to set images to
	*.jpg 			gets all jpg images from a folder

Then combine into a single image:
	montage *.jpg -gravity center -tile NxN -geometry +0+0 output.jpg

Arguments:
	*.jpg 			gets all jpg images from a folder
	-gravity		centers rows/columns
	-tile			how many images per row/column in final image
	-geometry		no extra space between images (or +N+N for padding)
	-background		none or "rgb(255,255,255)"
	output.jpg		output filename and format

VERY LARGE IMAGES:
When working with extra big images, ImageMagick makes
some suggestions where RAM may run out:
http://www.imagemagick.org/Usage/files/#massive

'''

# ==============

def power_of(num, base):
	''' checks if a number is a power another '''
	while(num % base == 0):
		num = num / base
	return num == 1

def generate(input_file, output_folder, zoom_level, resize_width, quiet):
	''' generates slippy map tiles from large image '''

	# how many tiles will that be?
	num_tiles = pow(2, zoom_level)
	if not quiet: print 'Zoom level ' + str(zoom_level) + ' = ' + str(num_tiles) + ' tiles'

	# get image dims (without loading into memory)
	# via: http://stackoverflow.com/a/19035508/1167783
	if not quiet: print 'Getting source image dimensions...'
	t = magic.from_file(input_file)
	try:
		if input_file.endswith('.jpg') or input_file.endswith('.jpeg'):
			dims = re.search(', (\d+)x(\d+)', t)
			width = int(dims.group(1))
			height = int(dims.group(2))
		elif input_file.endswith('.tif') or input_file.endswith('.tiff'):
			width = int(re.search('width=(\d+)', t).group(1))
			height = int(re.search('height=(\d+),', t).group(1))
		elif input_file.endswith('.png'):
			dims = re.search(', (\d+) x (\d+)', t)
			width = int(dims.group(1))
			height = int(dims.group(2))
		else:
			if not quiet: print 'ERROR: Unknown source image type; JPG, TIFF, or PNG only! Quitting...'
			exit(0)
	except:
		if not quiet: print 'ERROR: Could not parse source image dims! Quitting...'
		exit(0)
	
	if not quiet: print '- ' + str(width) + ' x ' + str(height) + ' pixels'

	# errors and warnings
	if not power_of(int(width), 2):
		if not quiet: print 'WARNING: Source image dims should be power of 2! Continuing anyway...'
	if width != height:
		if not quiet: print 'ERROR: Source image should be square! Quitting...'
		exit(0)

	# get details for ImageMagick
	if not quiet: print 'Splitting to...'
	tile_width = int(math.ceil(width / num_tiles))
	if not quiet: print '- ' + str(tile_width) + ' x ' + str(tile_width) + ' px tiles'
	pad = len(str(num_tiles * num_tiles))

	# split using ImageMagic, then resize to the expected tile size
	# use filename padding so glob gets them in the right order
	if not os.path.exists(output_folder):
		os.mkdir(output_folder)
	cmd = 'convert ' + input_file + ' -quiet -crop ' + str(tile_width) + 'x' + str(tile_width) + ' -resize ' + str(resize_width) + 'x' + str(resize_width) + ' ' + output_folder + '/%0' + str(pad) + 'd.png'
	os.popen(cmd)
	if not quiet: print '- done!'

	# rename/move images into tile server format
	if not quiet: print 'Moving files into column folders...'

	# 1. make cols
	for x in range(0, num_tiles):
		folder = output_folder + '/' + str(zoom_level) + '/' + str(x)
		if not os.path.exists(folder):
			os.makedirs(folder)

	# 2. move tiles into their column folders
	tiles = glob.glob(output_folder + '/*.png')
	for i, tile in enumerate(tiles):
		col = i % num_tiles
		f = i / num_tiles
		dst = output_folder + '/' + str(zoom_level) + '/' + str(col) + '/' + str(f) + '.png'
		shutil.move(tile, dst)
	if not quiet: print '- done!'

# ==============

if __name__ == '__main__':
	p = argparse.ArgumentParser(description='Takes a large image as the input, outputs map tiles at the appropriate size and file structure for use in frameworks like leaflet.js, MapBox, etc. Much more info in the source code.', usage='python GenerateSlippyMapTiles.py input_file zoom_level output_folder [options]')
	p.add_argument('input_file', help='large image file to split (JPG, PNG, or TIFF)')
	p.add_argument('zoom_level', help='zoom level(s) to generate (0 to 18); either integer or range (ex: 2-6)')
	p.add_argument('output_folder', help='folder name to write tiles to (will be created if does not exist)')
	p.add_argument('-w', '--resize_width', help='dimension in pixels for outputted tiles (default 256px)', metavar='', type=int, default=256)
	p.add_argument('-q', '--quiet', help='suppress all output from program (useful for integrating into larger projects)', action='store_true')
	args = p.parse_args()

	input_file = args.input_file
	zoom_level = args.zoom_level
	output_folder = args.output_folder
	resize_width = args.resize_width
	quiet = args.quiet

	if not quiet: print 'GENERATING SLIPPY-MAP TILES'
	if not quiet: print ('- ' * 14)

	# if multiple zoom levels, run them all
	# otherwise, run just once
	if '-' in zoom_level:
		try:
			match = re.search(r'([0-9]+)-([0-9]+)', zoom_level)
			zoom_min = int(match.group(1))
			zoom_max = int(match.group(2))
		except:
			if not quiet: print "ERROR: Couldn't parse zoom levels; should be int or 'min-max'! Quitting..."
			exit(0)
		for z in range(zoom_min, zoom_max+1):
			generate(input_file, output_folder, z, resize_width, quiet)
			if not quiet: print ('- ' * 14)
	else:
		generate(input_file, output_folder, int(zoom_level), resize_width, quiet)
		if not quiet: print ('- ' * 14)

	# that's it!
	if not quiet: print 'FINISHED!'

Official (ie more likely to be updated) version in this repo: https://github.com/jeffThompson/EmptyApartments/tree/master/GenerateSlippyMapTiles

Hi. Thanks for sharing this. I don't know if it's the same for you, but I had to change line 158 from

tiles = glob.glob(output_folder + '/.png')
to
tiles = sorted(glob.glob(output_folder + '/.png'))

in order to have the tiles in the correct order. I'm currently running Python 2.7.16 on Ubuntu 19.04.

@calsurferpunk – hmm, it shouldn't need that, I don't think? But it probably can't hurt either.

This doesn't seem to be creating any files, just directories.

@Qwerty-Space – so weird! Do you get a full set of nested directories (ie in the format of tiles/{z}/{x}/{y}.png)? Are you getting any errors?

No errors, it just completes with the directories in place

I was getting all of the files in each set of folders, but they weren't correct for the needed coordinates. I'm pretty sure I had empty folders when I tried to use an incorrect image size (i.e. not perfectly square).

> pip2 install python-magic
> python2 GenerateSlippyMapTiles.py --help
usage: python GenerateSlippyMapTiles.py input_file zoom_level output_folder [options]

Takes a large image as the input, outputs map tiles at the appropriate size
and file structure for use in frameworks like leaflet.js, MapBox, etc. Much
more info in the source code.

positional arguments:
  input_file            large image file to split (JPG, PNG, or TIFF)
  zoom_level            zoom level(s) to generate (0 to 18); either integer or
                        range (ex: 2-6)
  output_folder         folder name to write tiles to (will be created if does
                        not exist)

optional arguments:
  -h, --help            show this help message and exit
  -w , --resize_width   dimension in pixels for outputted tiles (default
                        256px)
  -q, --quiet           suppress all output from program (useful for
                        integrating into larger projects)

Added magic file specification parameter: https://gist.github.com/Pysis868/99e4b6f6592cbc0e59c5849316d0f86c
On my Mac with Homebrew it was at /usr/local/Cellar/libmagic/5.37/share/misc/magic.
It uses the executable from file-formula and requires the dependency libmagic that provides this.

Sorry all that I haven't had time to look into this.

@Pysis868 – did this solve the issue and you now get tiles correctly?

Hi Jeff,
I have a question that is related but not exactly about your tiler.

I have looked through many os resources where codes are available to create tiles, based on the assumption that the source image is available. But I would love to understand more about the creation of the source image first. Do you have any tip or publicly available resource on creating a source image from a data file using python3?

E.g.,

1. How to make a source image from a netCDF file available from:
   ftp://ftp.nodc.noaa.gov/pub/data.nodc/ghrsst/L4/GLOB/JPL/MUR/2020/
   These are 1 KM spatial resolution data, meaning effectively 36000 x 18000 pixels.

2. Is the so-called source image a large raster image with 36000 x 18000 pixels (pre-generated)?

3. Then how to generate multiple CRS datasets for WGS84 projection? Namely, EPSG 4326, 3413, 3031. I am not very much interested in the Google Mercator EPSG 3857. Can the same source map be reused for each of these CRSs or does it have to be generated separately for each CRS.

I understand this is unrelated to your tiler (which needs a source map you generated using imageMagick), but some pointers would be useful to reuse the tiler subsequently.

Hope you are keeping safe in these turbulent time in your location.
Thanks

@prasanjitdash – sorry, not sure I can be much help here! I don't use GIS or mapping tools much. I think you'd probably look for existing satellite data, then combine them into a single image (maybe using ImageMagick or a similar tool) and then split it into tiles? I know NASA and ESA have some data, as does NOAA too I believe? Or is your question what to do once you have the imagery downloaded already?

Thanks @jeffThompson for your response. Satellite data availability is no issue - there is plenty (as in the example link in my question). The question is rather amateurish and related to the procedure - what is a 'Source Map' and how to make it. Tiling is the second step.

1) Source Map is a full resolution big image? or you make small high-res images and stitch?

2) Making a big image of the earth, let's say with 100 m resolution will be a very huge file, approx 360000 x 180000 size and bytes

3) Do you have to make Source Maps with various CRS, depending on your target tiles or one Big Source Map can be reused for all?

I will do some more digging and will get back here for mentioning if I find a procedural answer. Thas again Jeff for your response and stay safe.

@prasanjitdash - aha, got it! Thanks for clarifying. If I remember correctly, yes: you make one really big image(mine was 150,000 pixels square) which then gets sliced into the tiles. The files will be huge but you don't need more than one source image – basically this code just slices it into different sized squares, one for each zoom level. Hope that helps!

@jeffThompson Hi Jeff, Thanks, again. I will move in this direction then of generating huge source images and then do the slicing. If some other ideas dawn upon me on some of the uneventful evenings, I will get back here to report as a source for those who may be pondering over the same question. Something like a code to generate a source image, sample input, sample output, and sliced tiles with a few different CRS would be useful I guess. Regards

have

@jeffThompson Hi Jeff, Thanks, again. I will move in this direction then of generating huge source images and then do the slicing. If some other ideas dawn upon me on some of the uneventful evenings, I will get back here to report as a source for those who may be pondering over the same question. Something like a code to generate a source image, sample input, sample output, and sliced tiles with a few different CRS would be useful I guess. Regards

have you try ncWMS? if the netcdf not change much i thik ncWMS will be good

@jeffThompson, I've updated this to Python 3 and cleaned up the logging a bit. Hope you don't mind and feel free to steal back - https://github.com/danizen/campaign-map/blob/master/gentiles.py

@danizen – awesome, thanks for doing that! I'll add a note in the code pointing to your version

@jeffThompson - I am preparing to go a little further. By using "Farey Sequences", we can figure out the aspect ratio of the map, and divide it into tiles without first making it square.

@jeffThompson
Want to try your script, but I have one question: what if I want to tile non-square image? Is there some hint that I can apply to make your script working (I see from the code that it will quit in this case)

@MikitaBelikau – hmm I think so? You could certainly try it and see if it works

@jeffThompson , maybe i didn't make it clear, so I will rephrase it: I see from the code, that script will terminate if image is not square.
So, my question was: do you know some hint which I can apply to non-squared image to make it possible for script to process it?

@MikitaBelikau – bah sorry! You were clear, I'm still not fully awake :)

I think this would take some work throughout, sadly. The code assumes square input so things like tile size are also square and are based on the width of the image. You could try it by commenting out lines 134–136 so it will run anyway, but not sure if it will look nice or break.

Hope that helps!

I have used other scripts to generate tile from non-square images. If you want to use this one, maybe preprocess the image onto a square canvas using ImageMagic first.