ue4_save_game_extractor_recompressor.py

#!/usr/bin/env python3

"""
Python 3 code that can decompress (to a .gvas file), or recompress (to a .savegame file)
the UE4 savegame file that Astroneer uses.

Though I wrote this for tinkering with Astroneer games saves, it's probably
generic to the Unreal Engine 4 compressed saved game format.

Examples:

ue4_save_game_extractor_recompressor.py --extract  --file z2.savegame  # Creates z2.gvas
ue4_save_game_extractor_recompressor.py --compress --file z2.gvas      # Creates z2.NEW.savegame
ue4_save_game_extractor_recompressor.py --test     --file z2.savegame  # Creates *.test files

---

Copyright (c) 2016-2020 Martin Falatic

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""

import argparse
import os
import sys
import zlib

HEADER_FIXED_HEX = "BE 40 37 4A EE 0B 74 A3 01 00 00 00"
HEADER_FIXED_BYTES = bytes.fromhex(HEADER_FIXED_HEX)
HEADER_FIXED_LEN = len(HEADER_FIXED_BYTES)
HEADER_RAW_SIZE_LEN = 4
HEADER_GVAS_MAGIC = b'GVAS'
COMPRESSED_EXT = 'savegame'
EXTRACTED_EXT = 'gvas'


def extract_data(filename_in, filename_gvas):
    data_gvas = bytes()
    with open(filename_in, 'rb') as compressed:
        header_fixed = compressed.read(HEADER_FIXED_LEN)
        header_raw_size = compressed.read(HEADER_RAW_SIZE_LEN)
        gvas_size = int.from_bytes(header_raw_size, byteorder='little')
        header_hex = ''.join('{:02X} '.format(x) for x in header_fixed)
        if HEADER_FIXED_BYTES != header_fixed:
            print(f"Header bytes do not match: Expected '{HEADER_FIXED_HEX}' got '{header_hex}'")
            sys.exit(1)
        data_compressed = compressed.read()
        data_gvas = zlib.decompress(data_compressed)
        sz_in = len(data_compressed)
        sz_out = len(data_gvas)
        if gvas_size != sz_out:
            print(f"gvas size does not match: Expected {gvas_size} got {sz_out}")
            sys.exit(1)
        with open(filename_gvas, 'wb') as gvas:
            gvas.write(data_gvas)
        header_magic = data_gvas[0:4]
        if HEADER_GVAS_MAGIC != header_magic:
            print(f"Warning: Raw save data magic: Expected {HEADER_GVAS_MAGIC} got {header_magic}")
        print(f"Inflated from {sz_in:d} (0x{sz_in:0x}) to {sz_out:d} (0x{sz_out:0x}) bytes as {filename_gvas}")
    return data_gvas


def compress_data(filename_gvas, filename_out):
    data_gvas = None
    data_compressed = bytes()
    with open(filename_gvas, 'rb') as gvas:
        data_gvas = gvas.read()
    header_magic = data_gvas[0:4]
    if HEADER_GVAS_MAGIC != header_magic:
        print(f"Warning: Raw save data magic: Expected {HEADER_GVAS_MAGIC} got {header_magic}")
    with open(filename_out, 'wb') as compressed:
        compress = zlib.compressobj(
            level=zlib.Z_DEFAULT_COMPRESSION,
            method=zlib.DEFLATED,
            wbits=4+8,  # zlib.MAX_WBITS,
            memLevel=zlib.DEF_MEM_LEVEL,
            strategy=zlib.Z_DEFAULT_STRATEGY,
        )
        data_compressed += compress.compress(data_gvas)
        data_compressed += compress.flush()
        compressed.write(HEADER_FIXED_BYTES)
        compressed.write(len(data_gvas).to_bytes(HEADER_RAW_SIZE_LEN, byteorder='little'))
        compressed.write(data_compressed)
        sz_in = len(data_gvas)
        sz_out = len(data_compressed)
        print(f"Deflated from {sz_in:d} (0x{sz_in:0x}) to {sz_out:d} (0x{sz_out:0x}) bytes as {filename_out}")
    return data_compressed


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="UE4 Savegame Extractor/Compressor")
    parser.add_argument('--filename')
    parser.add_argument('--extract', action='store_true')
    parser.add_argument('--compress', action='store_true')
    parser.add_argument('--test', action='store_true')
    args = parser.parse_args()

    argerrors = False
    if not args.filename:
        print("Error: No filename specified")
        argerrors = True
    if (args.extract and args.compress):
        print("Error: Choose only one of --extract or --compress")
        argerrors = True
    if (args.extract or args.compress) and args.test:
        print("Error: --test switch stands alone")
        argerrors = True
    if argerrors:
        sys.exit(1)

    filename = args.filename
    dirname, basename = os.path.split(filename)
    rootname, extname = os.path.splitext(basename)

    if args.extract:
        filename_in = filename
        filename_gvas = os.path.join(dirname, f'{rootname}.{EXTRACTED_EXT}')
        data_gvas = extract_data(filename_in=filename_in, filename_gvas=filename_gvas)
    elif args.compress:
        filename_gvas = filename
        filename_out = os.path.join(dirname, f'{rootname}.NEW.{COMPRESSED_EXT}')
        data_compressed = compress_data(filename_gvas=filename, filename_out=filename_out)
    elif args.test:
        filename_in = filename
        filename_gvas_1 = os.path.join(dirname, f'{rootname}.{EXTRACTED_EXT}.1.test')
        filename_out = os.path.join(dirname, f'{rootname}.NEW.{COMPRESSED_EXT}.test')
        filename_gvas_2 = os.path.join(dirname, f'{rootname}.{EXTRACTED_EXT}.2.test')
        data_gvas = extract_data(filename_in=filename_in, filename_gvas=filename_gvas_1)
        data_compressed = compress_data(filename_gvas=filename_gvas_1, filename_out=filename_out)
        data_check = extract_data(filename_in=filename_out, filename_gvas=filename_gvas_2)
        status = "Passed" if data_gvas == data_check else "Failed"
        print()
        print(f"{status}: Tested decompress-compress-decompress")

I was curious to see Astroneer's uncompressed saved game file (especially given how simply loading a saved game can subtly change the world you're in)... it seems that the format is more of a general Unreal Engine (UE4) thing.

The save file has two main sections: a 16-byte header followed by zlib-compressed data.

The first 12 bytes of the header appear to be constant (but it may vary in the future):
BE 40 37 4A EE 0B 74 A3 01 00 00 00

The remaining 4 bytes of the header are the size of the decompressed save data
(e.g.: 42362955 bytes = 0x0286684b = 4B 68 86 02)

Finally, the zlib-compressed data block has magic data 48 89 - default compression level with a 4K window size. You can read RFC 1950 for details on zlib's format, but note that UE4 evidently does NOT like other settings here! (Even though zlib will readily handle it, oddly the game will not). The zlib data block ends with the usual Adler32 checksum of the uncompressed data.

48 89 # Magic:
# CINFO = 4 == 2^(4+8) = 4K window size
# CM = 8 == "deflate"
# FLG = 0x89 == {FLEVEL=10b (==2=default), FDICT = 0b, FCHECK = 11001b (25)}

After decompression, we have the raw UE4 save data (with magic signature GVAS). I have no suggestions on how to edit it or such, but it's very interesting how much seems to change just by loading and then immediately saving the game.

Note: Recompressing a saved game may not result in byte-identical compressed data (certainly doesn't for me), but the decompressed UE4 game save data remains identical and it seems to work fine for Astroneer, provided the correct settings are used as above. (Some zlib implementations differ in how well they compress for given settings.)

I just sent you this reddit message, figured maybe it's seen here earlier:

---

Hello,

I just finished a good round of Astroneer and went to check on the subreddit when I found out I messed up my base, because I packed all 6 possible extensions with buildings not knowing one can extend the points further for larger bases.
Long story short, I searched for someone who tried digging into the savegame files, and your post on reddit was the only thing I could find.
Since you were able to decompress them, do you think there is a way to change your buildings as in remove one? I put some hours into my save and would hate to give it up just because I messed up the base design..
I'll look into your gist now, I'm a CS major almost finished with university, so I have some knowledge.

Cheers

Simply searching the savegame for 'Condenser' results in 5 results near the start of the file.

1st occurrence:
43 6F 6E 64 65 6E 73 65 72 5F 4C 61 72 67 65 5F 43 5F 35 00 14 00 00 00
Condenser_Large_C_5.....

2nd and 3rd:
2F 47 61 6D 65 2F 43 6F 6D 70 6F 6E 65 6E 74 73 5F 4C 61 72 67 65 2F 43 6F 6E 64 65 6E 73 65 72 5F 4C 61 72 67 65 2E 43 6F 6E 64 65 6E 73 65 72 5F 4C 61 72 67 65 5F 43 00 23 00 00 00
/Game/Components_Large/Condenser_Large.Condenser_Large_C.#...

4th and 5th:
2F 47 61 6D 65 2F 43 6F 6D 70 6F 6E 65 6E 74 73 5F 4C 61 72 67 65 2F 43 6F 6E 64 65 6E 73 65 72 5F 4C 61 72 67 65 2E 43 6F 6E 64 65 6E 73 65 72 5F 4C 61 72 67 65 5F 43 00 B1 00 00 00 08 00 00 00 02 08 00 00 00 86 2D 00 00 00 00 00 00 23 00 00 00
/Game/Components_Large/Condenser_Large.Condenser_Large_C......#

Now I have one Fuel Condenser in my world, And I checked with an empty world without one, no occurrences there, very weird.

A long list of Tether Posts, with an interesting bit of information:

00 10 00 00 00 54 65 74 68 65 72 50 6F 73 74 5F 43 5F 39 38 00 11 00 00 00 54 65 74 68 65 72 50 6F 73 74 5F 43 5F 31 30 38 00 11 00 00 00 54 65 74 68 65 72 50 6F 73 74 5F 43 5F 31 30 30
.....TetherPost_C_98.....TetherPost_C_108.....TetherPost_C_100

There is no information associated with the Components, no coordinates, just 5 bytes with the second one seemingly addressing the length of the name? When the C_ number goes into 3 digits the 10 changes to 11

The first occurrence is preceded by the entry .....Scene....., which occurs only once in the save

.....StagingSkybox..... seems to finish the Scene section

This script has been very useful in investigating Astroneer save files - great work!

I haven't reverse-engineered too much of the format yet, but https://github.com/oberien/gvas-rs looks like a good starting point to figure out how the decompressed file is encoded.

I thought nobody noticed this... I'm grateful that it came in handy! I later found an in-game trainer that was fun to dabble with (I've not played in a while though.)

As for why I didn't know anyone commented, I've learned that gist has a major old bug wherein you don't get notifications for comments to gists.

If you leave a comment, it would be appreciated if you ping me elsewhere so I know to find it!

Im new to this sorta stuff how do i use it

@MartyMacGyver

How do I recompress the file? I can decompress them but what is the command to recompress?

Is this project still active?

I haven't done anything with this in years. I'm not sure it works with current versions of Astroneeer.

Its still decompresses the file just fine. When I edit the -raw and rerun the program it overwrites the -raw instead of recompressing it.

I am not super familiar with python, so I was wondering if you could point me in the right direction. If not, that is fine too!

@ChunkySpaceman, @unnamedDE, et al...

This was meant to be a proof of concept, not a final utility.

Given a savegame file (e.g., abc123.savegame), this will open and extract it to the "raw" uncompressed file (abc123.savegame-raw), then it immediately re-compresses it to abc123.savegame-z. It checks along the way to make sure that, from the perspective of the raw data within the input and output compressed files, it's all the same data inside.

But I figured why not just make it a proper utility, with arguments and everything? So I did... it should also be easier to read what each block of code is doing now.

See the top of the file for usage instructions. Python >=3.6 is best (I used 3.8 to develop this).

@MartyMacGyver, if you have any interest in save editing Astroneer still, I run a Discord server where we've made some pretty good strides, thanks to your code. Add me on Discord if you're interested @Spyci#0001.

how can we edit the .gvas code ? because i can't compile this : https://github.com/oberien/gvas-rs

Heya, @MartyMacGyver , sorry to bother you with such an old topic.

I have used your code to decompile a savegame, and i'm trying to change anything that indicates that i used creative mode, so that the missions are available again. I turned on creative without thinking too much about it, just to take a look around, and didnt realize that missions would be locked from then on.

I don't even know where to begin searching for something to edit. The decompiler worked, but i dont know...

Thanks for the code, anyway!

@ReDJstone I haven't worked on this in years.... there's a comment above describing an active effort regarding save editing that might be useful.

I'm new to all of this coding and things, where I have to put the namefile in?, I can't figure it out

I'm new to all of this coding and things, where I have to put the namefile in?, I can't figure it out

pls guys... U_U

I haven't worked on this in over 5 years - the instructions are in the code comments, but I don't think this works anymore and is now obsolete.