Sending maps outside of coop edit

README.md

Sending maps outside of coop edit

Sometimes it would be useful if a server were able to send a custom map to its clients, in modes different from coop edit. Sendmap/getmap operations are usually blocked in such modes, both server-side and client-side. However, most other map editing messages are allowed (clients cannot send them, but they can receive and parse them). Here, I will provide a method for sending custom maps using a sequence of allowed messages.

We are going to build a modded client capable of exporting a map into a custom file format, which the server will then read and use to send data to its clients. ogz files cannot be sent directly.

Notes

Integers

Keep in mind that when encoding 32-bit integers (e.g. via putint() and similar functions) they are not always encoded to four bytes, as described here.

N_CLIENT packets

Some messages need to be wrapped in a N_CLIENT packet. Such packets are encoded like this: N_CLIENT CLIENTNUM BUFLEN BUF The easiest thing to do in this case is to first build the buffer and take its length, and then build the N_CLIENT packet.

Selections

A selection (struct selinfo) is an object that identifies a region of the map where editing messages should be applied. It is defined like this:

struct selinfo
{
    int corner;
    int cx, cxs, cy, cys;
    ivec o, s;
    int grid, orient;
    selinfo() : corner(0), cx(0), cxs(0), cy(0), cys(0), o(0, 0, 0), s(0, 0, 0), grid(8), orient(0) {}
/* ... */
};

o contains the absolute coordinates of the starting corner of the selection (the one with the lowest x, y, z values). grid is the gridsize (2^gridpower). s is the size of the selection along each axis in gridsize units. All the other values are irrelevant to us and can be set to zero.

The best way to understand selections is to make a little function that prints their values, then go into coop edit and start selecting cubes and print the selection. Such a function can be added to engine/octaedit.cpp:

void printsel() {
	conoutf("sel: corner(%d), cx(%d), cxs(%d), cy(%d), cys(%d), o(%d,%d,%d), s(%d,%d,%d), grid(%d), orient(%d)", sel.corner, sel.cx, sel.cxs, sel.cy, sel.cys, sel.o.x, sel.o.y, sel.o.z, sel.s.x, sel.s.y, sel.s.z, sel.grid, sel.orient);
}
COMMAND(printsel, "");

Also open fpsgame/client.cpp and search for addmsg(N_EDITF to see how messages that require selection data are sent.

The custom map format

We need to come up with a file format that we can use to export our map from the client, and import into the server. We want to export everything that can be sent over the network: entities, map vars and geometry (the latter includes textures, vslots and materials).

Entities

Entities are easy to encode. Three floats for the position (x, y, z) and six integers (type, attr1, attr2, attr3, attr4, attr5). So each entity can be encoded as: X Y Z TYPE ATTR1 ATTR2 ATTR3 ATTR4 ATTR5

Map vars

Map vars can be of three types: integer, float and string. We need a field to indicate the type. I went with 0 = integer, 1 = float, 2 = string. Then we need the variable name and its value. Variables will be encoded like this: TYPE NAME VAL

Geometry (chunks)

Then we have geometry. We will load geometry data into a compressed buffer that can be sent directly inside of a N_CLIPBOARD packet. The problem is that if the map is too large we won't be able to export it all at once. So we will split it into cubic chunks. Each chunk will have a position (x, y, z) (it will be the absolute position of the starting corner divided by 2^gridscale) and a gridscale to represent its size (the corresponding selection will have s = (1, 1, 1)). We also need the uncompressed (unpacked) and compressed (packed) length of the buffer, and the buffer itself. Chunks will be encoded like this: GRIDSCALE X Y Z UNPACKEDLEN PACKEDLEN BUFFER

Final file format

We also need information about how big the map is (mapscale). Now we have everything we need to define our custom file format:

MAPSCALE NUMENTS NUMVARS NUMCHUNKS
X Y Z TYPE ATTR1 ATTR2 ATTR3 ATTR4 ATTR5     // for each entity
TYPE NAME VAL                                // for each var; TYPE = 0: int, 1: float, 2: string
GRIDSCALE X Y Z UNPACKEDLEN PACKEDLEN BUFFER // for each chunk

Client-side code: exporting the map

This is the code I added to my client (engine/octaedit.cpp, after function previewprefab) in order to export a map:

struct chunk {
	uchar *buf;
	int unpackedlen, packedlen;
	int x, y, z, gridscale;
	chunk(int x, int y, int z, int gridscale) : x(x), y(y), z(z), gridscale(gridscale) {buf = NULL; unpackedlen = packedlen = 0;}
	~chunk() {
		free(buf);
	}
};

bool exportcube(int x, int y, int z, int gridscale, vector<chunk *> &expdata) {
//	conoutf("CHUNK (%d, %d, %d) @ %d", x, y, z, gridscale);
	if(gridscale <= 6) return false; /* remove this line if you don't mind exporting very small, very dense chunks that will almost certainly cause lag; just make sure that gridscale never goes below zero */
	selinfo sel;
	sel.grid = pow(2, gridscale);
	sel.o = ivec(x * sel.grid, y * sel.grid, z * sel.grid);
	sel.s = ivec(1,1,1);
	editinfo *ei = NULL;
	mpcopy(ei, sel, false);
	chunk *c = new chunk(x, y, z, gridscale);
//	conoutf(">> trying to pack");
	if(!packeditinfo(ei, c->unpackedlen, c->buf, c->packedlen)) {
//		conoutf(">> trying smaller size");
		bool valid = true;
		for(int i = 0; i <= 1; i++) for(int j = 0; j <= 1; j++) for(int k = 0; k <= 1; k++) {
//			conoutf(">> %d, %d, %d @ %d", 2*x + i, 2*y + j, 2*z + k, gridscale-1);
			if(!exportcube(2*x + i, 2*y + j, 2*z + k, gridscale-1, expdata)) {
				valid = false;
				break;
			}
		}
		if(!valid) {
			if(ei) free(ei);
			return false;
		};
	} else {
//		conoutf(">> OK [%d => %d]", c->unpackedlen, c->packedlen);
		expdata.add(c);
	}
	if(ei) free(ei);
	return true;
}

int exportentities(vector<uchar> &buf) {
	vector<extentity *> &ents = entities::getents();
	loopv(ents) {
		extentity *e = ents[i];
		putfloat(buf, e->o.x);
		putfloat(buf, e->o.y);
		putfloat(buf, e->o.z);
		putint(buf, e->type);
		putint(buf, e->attr1);
		putint(buf, e->attr2);
		putint(buf, e->attr3);
		putint(buf, e->attr4);
		putint(buf, e->attr5);
	}
	return ents.length();
}

void exportmaptofile(char *fn, int numents, vector<uchar> entbuf, int numvars, vector<chunk *> chunks) {
	stream *of = openrawfile(fn, "wb");

	vector<uchar> buf;

	// header
	putint(buf, worldscale);
	putint(buf, numents);
	putint(buf, numvars);
	putint(buf, chunks.length());

	// ents
	buf.put(entbuf.getbuf(), entbuf.length());

	// vars
	enumerate(idents, ident, id, {
		if((id.type!=ID_VAR && id.type!=ID_FVAR && id.type!=ID_SVAR) || !(id.flags&IDF_OVERRIDE) || id.flags&IDF_READONLY || !(id.flags&IDF_OVERRIDDEN)) continue;
		putint(buf, id.type == ID_VAR ? 0 : id.type == ID_FVAR ? 1 : 2);
		sendstring(id.name, buf);
		switch(id.type) {
			case ID_VAR: {
				putint(buf, *id.storage.i);
				break;
			}
			case ID_FVAR: {
				putfloat(buf, *id.storage.f);
				break;
			}
			default: {
				sendstring(*id.storage.s, buf);
			}
		}
	});

	// chunks
	loopv(chunks) {
		chunk *c = chunks[i];
		putint(buf, c->gridscale);
		putint(buf, c->x);
		putint(buf, c->y);
		putint(buf, c->z);
		putint(buf, c->unpackedlen);
		putint(buf, c->packedlen);
		buf.put(c->buf, c->packedlen);
	}

	of->write(buf.getbuf(), buf.length());
	of->close();
}

void exportmap(char *fn) {
	if(!fn || !fn[0]) {
		conoutf("no file specified");
		return;
	}
	vector<uchar> entbuf;
	int numents = exportentities(entbuf);

	int numvars = 0;
	enumerate(idents, ident, id, {
		if((id.type == ID_VAR || id.type == ID_FVAR || id.type == ID_SVAR) && id.flags&IDF_OVERRIDE && !(id.flags&IDF_READONLY) && id.flags&IDF_OVERRIDDEN) numvars++;
	});

	vector<chunk *> chunks;
	int gridscale = worldscale - 1;
	for(int x = 0; x <= 1; x++) for(int y = 0; y <= 1; y++) for(int z = 0; z <= 1; z++) {
		vector<chunk *> expdata;
		if(!exportcube(x, y, z, gridscale, expdata)) {
			conoutf("failed to export map data");
			return;
		}
		loopv(expdata) {
			chunks.add(expdata[i]);
		}
	}
	conoutf("map exported into %d chunk%s, %d ent%s, %d var%s",
		chunks.length(), chunks.length() != 1 ? "s" : "",
		numents, numents != 1 ? "s" : "",
		numvars, numvars != 1 ? "s" : ""
	);

	exportmaptofile(fn, numents, entbuf, numvars, chunks);
}
COMMAND(exportmap, "s");

After this addition, in order to export a map, open the client, load the desired map and type /exportmap filename.bin. The file will appear in ~/.sauerbraten/ or the equivalent directory on your system.

Server-side code: importing the map

Now we want to import the map from the server. We will store it in a struct custommap, and we'll have a vector of those, so that multiple maps can be stored and sent to clients. The code below should be added to fpsgame/server.cpp, possibly as an #included header to keep things clean.

struct chunk {
	uchar *buf;
	int unpackedlen, packedlen;
	int x, y, z, gridscale;
	chunk(int _x, int _y, int _z, int _gridscale) : x(_x), y(_y), z(_z), gridscale(_gridscale) {buf = NULL; unpackedlen = packedlen = 0;}
	void setbuf(uchar *_buf, int _unpackedlen, int _packedlen) {
		if(buf) free(buf);
		unpackedlen = _unpackedlen;
		packedlen = _packedlen;
		buf = (uchar*)malloc(packedlen);
		memcpy(buf, _buf, packedlen);
	}
	~chunk() {
		if(buf) free(buf);
	}
};

struct varinfo {
	string name;
	int type;
};
struct varinfo_i : varinfo {
	int val;
	varinfo_i(char *_name, int _val) {
		copystring(name, _name);
		type = 0;
		val = _val;
	}
};
struct varinfo_f : varinfo {
	float val;
	varinfo_f(char *_name, float _val) {
		copystring(name, _name);
		type = 1;
		val = _val;
	}
};
struct varinfo_s : varinfo {
	string val;
	varinfo_s(char *_name, char *_val) {
		copystring(name, _name);
		type = 2;
		copystring(val, _val);
	}
};

struct custommap {
	string name;
	int mapscale;
	vector<entity> ents;
	vector<varinfo *> vars;
	vector<chunk *> chunks;

	custommap(const char *_name, int _mapscale) {
		copystring(name, _name);
		mapscale = _mapscale;
	}

	void addent(entity e) {
		ents.add(e);
	}

	void addvar(varinfo *v) {
		vars.add(v);
	}

	void addchunk(chunk *c) {
		chunks.add(c);
	}

	~custommap() {
		loopv(vars) delete vars[i];
		loopv(chunks) delete chunks[i];
		ents.setsize(0);
		vars.setsize(0);
		chunks.setsize(0);
	}
};
vector<custommap *> custommaps;

void clearcustommaps() {
	loopv(custommaps) delete custommaps[i];
	custommaps.setsize(0);
}
COMMAND(clearcustommaps, "");

void addcustommap(const char *name, const char *filename) {
	stream *f = openrawfile(filename, "rb");

	f->seek(0L, SEEK_END);
	size_t fsize = f->tell();
	f->seek(0L, SEEK_SET);
	uchar *cbuf = (uchar*)malloc(fsize);
	f->read(cbuf, fsize);
	ucharbuf buf(cbuf, fsize);

	// header
	int mapscale = getint(buf), numents = getint(buf), numvars = getint(buf), numchunks = getint(buf);
	custommap *cm;
	cm = new custommap(name, mapscale);

	// ents
	for(int i = 0; i < numents; i++) {
		entity e;
		e.o = vec(0,0,0);
		e.o.x = getfloat(buf); e.o.y = getfloat(buf); e.o.z = getfloat(buf);
		e.type = getint(buf);
		e.attr1 = getint(buf); e.attr2 = getint(buf); e.attr3 = getint(buf); e.attr4 = getint(buf); e.attr5 = getint(buf);
		cm->addent(e);
	}

	// vars
	for(int i = 0; i < numvars; i++) {
		int type = getint(buf);
		string name;
		getstring(name, buf);
		switch(type) {
			case 0: {
				cm->addvar(new varinfo_i(name, getint(buf)));
				break;
			}
			case 1: {
				cm->addvar(new varinfo_f(name, getfloat(buf)));
				break;
			}
			default: {
				string val;
				getstring(val, buf);
				cm->addvar(new varinfo_s(name, val));
			}
		}
	}

	// chunks
	for(int i = 0; i < numchunks; i++) {
		int gridscale = getint(buf), x = getint(buf), y = getint(buf), z = getint(buf);
		chunk *c = new chunk(x, y, z, gridscale);
		int unpackedlen = getint(buf), packedlen = getint(buf);
		uchar *chunkbuf = (uchar*)malloc(packedlen);
		buf.get(chunkbuf, packedlen);
		c->setbuf(chunkbuf, unpackedlen, packedlen);
		free(chunkbuf);
		cm->addchunk(c);
	}

	f->close();
	free(cbuf);
	custommaps.add(cm);
}
COMMAND(addcustommap, "ss");

custommap *getcustommap(const char *mapname) {
	loopv(custommaps) if(!strcmp(custommaps[i]->name, mapname)) return custommaps[i];
	return NULL;
}

bool j_iscustommap(const char *mapname) {
	loopv(custommaps) if(!strcmp(custommaps[i]->name, mapname)) return true;
	return false;
}

With this addition, all you have to do to load maps is (server-init.cfg):

clearcustommaps
addcustommap "map1" "path/to/filename1.bin"
addcustommap "map2" "path/to/filename2.bin"
addcustommap "map3" "path/to/filename3.bin"

The functions getcustommap and j_iscustommap can be used to determine if the current map (smapname) is a custom map and decide if it should be sent to clients.

Server-side: sending the map

Let's assume a client ci has just connected and we want to send them the map. The first thing we want to do is move the client to spectators. This is for two reasons:

1. Sending the map will likely take some time, during which the player would be able to move across an empty map, which might be considered cheating;
2. The player would spawn at the center of the map. In order for them to spawn at a valid spawnpoint, we need to kill them.

We will then unspectate the client (if it is allowed to be unspectated) after the map is sent.

Newmap

The first editing message we must send is N_NEWMAP. This ensures that the player sees a blank map of the desired size.

void j_sendnewmap(int mapscale, clientinfo *ci) {
	vector<uchar> b;
	putint(b, N_NEWMAP);
	putint(b, mapscale);

	packetbuf nq(MAXTRANS, ENET_PACKET_FLAG_RELIABLE);
	putint(nq, N_CLIENT);
	putint(nq, ci->clientnum);
	putuint(nq, b.length());
	nq.put(b.getbuf(), b.length());
	sendpacket(ci->clientnum, 1, nq.finalize());
}

Delcube (remove floor)

Now we have a blank map, but it's not empty. There is a floor. This floor can interfere with subsequent operations so we want to get rid of it with N_DELCUBE. Note that this message requires selection data.

void j_delcube(int mapscale, clientinfo *ci) { // 0 0 0 | 2 2 1
	vector<uchar> bb;
	putint(bb, N_DELCUBE);
	putint(bb, 0); putint(bb, 0); putint(bb, 0);
	putint(bb, 2); putint(bb, 2); putint(bb, 1);
	putint(bb, pow(2, mapscale-1));
	putint(bb, 0);
	putint(bb, 0); putint(bb, 0); putint(bb, 0);
	putint(bb, 0); putint(bb, 0);

	packetbuf pq(MAXTRANS, ENET_PACKET_FLAG_RELIABLE);
	putint(pq, N_CLIENT);
	putint(pq, ci->clientnum);
	putuint(pq, bb.length());
	pq.put(bb.getbuf(), bb.length());
	sendpacket(ci->clientnum, 1, pq.finalize());
}

Entities

Now our map is completely empty. Let's send entities.

void j_sendents(vector<entity> ents, clientinfo *ci) {
	vector<uchar> p;
	loopv(ents) {
		entity e = ents[i];
		putint(p, N_EDITENT);
		putint(p, i);
		putint(p, (int)(e.o.x*DMF));
		putint(p, (int)(e.o.y*DMF));
		putint(p, (int)(e.o.z*DMF));
		putint(p, e.type);
		putint(p, e.attr1);
		putint(p, e.attr2);
		putint(p, e.attr3);
		putint(p, e.attr4);
		putint(p, e.attr5);
	}

	packetbuf q(MAXTRANS, ENET_PACKET_FLAG_RELIABLE);
	putint(q, N_CLIENT);
	putint(q, ci->clientnum);
	putuint(q, p.length());
	q.put(p.getbuf(), p.length());
	sendpacket(ci->ownernum, 1, q.finalize());
}

Map vars

And now for map variables:

void j_sendvars(vector<varinfo *> vars, clientinfo *ci) {
	vector<uchar> p;
	loopv(vars) {
		varinfo *v = vars[i];
		putint(p, N_EDITVAR);
		switch(v->type) {
			case 0: {
				putint(p, ID_VAR);
				sendstring(v->name, p);
				putint(p, ((varinfo_i*)v)->val);
				break;
			}
			case 1: {
				putint(p, ID_FVAR);
				sendstring(v->name, p);
				putfloat(p, ((varinfo_f*)v)->val);
				break;
			}
			default: {
				putint(p, ID_SVAR);
				sendstring(v->name, p);
				sendstring(((varinfo_s*)v)->val, p);
			}
		}
	}

	packetbuf q(MAXTRANS, ENET_PACKET_FLAG_RELIABLE);
	putint(q, N_CLIENT);
	putint(q, ci->clientnum);
	putuint(q, p.length());
	q.put(p.getbuf(), p.length());
	sendpacket(ci->ownernum, 1, q.finalize());
}

Chunks

Now all that's left to do is send our geometry chunks. The function below is for a single chunk (just use loopv(chunks)). Note that the N_CLIPBOARD packet is a bit special. It requires four integers at the beginning (N_CLIPBOARD CN UNPACKEDLEN PACKEDLEN) and the buffer at the end. There can be empty/unused space between the integers and the buffer. This is because we don't know in advance how big the integers are, so we create a buffer that we are sure is big enough to accomodate everything. I couldn't think of a scenario where those four integers take up more than 16 bytes, but just to be safe, you can create the packetbuf with size 32 + c->packedlen instead.

void j_sendchunk(chunk *c, clientinfo *ci) {
	// send clipboard
	packetbuf q(16 + c->packedlen, ENET_PACKET_FLAG_RELIABLE);
	putint(q, N_CLIPBOARD);
	putint(q, ci->clientnum);
	putint(q, c->unpackedlen);
	putint(q, c->packedlen);
	if(c->packedlen > 0) q.subbuf(c->packedlen).put(c->buf, c->packedlen); // subbuf returns the last c->packedlen bytes of the buffer
	sendpacket(ci->clientnum, 1, q.finalize());

	// send paste event
	vector<uchar> bb;
	putint(bb, N_PASTE);
	putint(bb, c->x * pow(2, c->gridscale)); putint(bb, c->y * pow(2, c->gridscale)); putint(bb, c->z * pow(2, c->gridscale));
	putint(bb, 1); putint(bb, 1); putint(bb, 1);
	putint(bb, pow(2, c->gridscale));
	putint(bb, 0);
	putint(bb, 0); putint(bb, 0); putint(bb, 0);
	putint(bb, 0); putint(bb, 0);

	packetbuf pq(MAXTRANS, ENET_PACKET_FLAG_RELIABLE);
	putint(pq, N_CLIENT);
	putint(pq, ci->clientnum);
	putuint(pq, bb.length());
	pq.put(bb.getbuf(), bb.length());
	sendpacket(ci->clientnum, 1, pq.finalize());
}

Wrapper function

That should be all. You can optionally send a N_REMIP (N_CLIENT CN 4 N_REMIP) but that will cause additional lag and (on my machine at least) no performance improvement. Also if you want to send additional geometry mid-game, that will cause more lag because the remip will have to be undone first.

So, now let's wrap all of the above into a single function that sends a map with a given name to a certain client:

void j_sendcustommap(const char *mapname, clientinfo *ci) {
	custommap *cm = getcustommap(mapname);
	if(!cm) return;

	bool to_kill = (ci->state.state != CS_SPECTATOR);
	if(to_kill) forcespectator(ci);

	// newmap
	j_sendnewmap(cm->mapscale, ci);

	// delete existing cubes
	j_delcube(cm->mapscale, ci);

	// ents
	j_sendents(cm->ents, ci);

	// vars
	j_sendvars(cm->vars, ci);

	// chunks
	loopv(cm->chunks) {
		j_sendchunk(cm->chunks[i], ci);
	}

	if(to_kill) {
		unspectate(ci);
	}

//	// remip
//	sendf(ci->clientnum, 1, "riiii", N_CLIENT, ci->clientnum, 4, N_REMIP);
}

Caveats

1. Clients will freeze while the map is sent
2. Flags and bases will not work in ctf and capture; if the server has the ogz file, skulls will work in collect but there will be no bases
3. The minimap will reflect a blank map
4. Lightmaps won't be sent (players will have to /calclight manually)
5. The scoreboard will no longer show game time after sending N_NEWMAP

However

1. servermotd can be used to warn clients about lag
2. If the custom map has flags/bases in the exact same location as an existing map, clients can be instructed to load this existing map (via N_MAPCHANGE) and then sent the custom map on top of it; then flags/bases will work
3. The minimap can be regenerated by typing /minimapsize 10. Clients would have to do this at the start of every game, but it's easy to bind to a key and shouldn't cause any lag (and if it does, a smaller number can be used)
4. /calclight -1 should be significantly faster than regular /calclight and still provide decent results
5. /gameclock 1 solves this. It's a permanent setting so it only has to be typed once.