Skip to content

Instantly share code, notes, and snippets.

@molind

molind/start_realm_sync.md

Last active October 7, 2024 12:48
Show Gist options
  • Save molind/c92cec1987ff1aaf015f02c4349b04a3 to your computer and use it in GitHub Desktop.
Save molind/c92cec1987ff1aaf015f02c4349b04a3 to your computer and use it in GitHub Desktop.
Download ZIP
Self-hosted realm sync
Raw
start_realm_sync.md

Recently, MongoDB announced that they are discontinuing support for Atlas Sync, and developers will be looking for alternatives. I thought it would be a good idea to write an article on how we host the Realm server ourselves.

In the realm-core repository, the source code for the synchronization server is located. We will need to add some wrappers and launch the server. Launching is trivial. Build realm-core from here: https://github.com/molind/realm-core/tree/auth_check (There are minimal changes to allow authorization with our tokens. You can check the diff). Then, in your server.cpp, write approximately the following:

#include <stdio.h>
#include <unistd.h>

#include <realm/sync/noinst/server/server.hpp>

#ifndef APP_VERSION
#define APP_VERSION "0.0.0"
#endif

int main(int argc, const char **argv)
{
    const char *address = std::getenv("REALM_ADDRESS");
    if (!address)
        address = "127.0.0.1";
    const char *port = std::getenv("REALM_PORT");
    if (!port)
        port = "9600";
    const char *key = std::getenv("REALM_KEY");
    if (!key)
        key = "key.pem";
    const char *path = std::getenv("REALM_PATH");
    if (!path)
        path = "realm_data";

    fprintf(stderr, "realm (%s) %s:%s, %s, %s\n", APP_VERSION, address, port, key, path);

    // Read hostname
    char hostname[50];
    gethostname(hostname, sizeof(hostname));

    auto config = realm::sync::Server::Config();
    config.id = hostname;
    config.listen_address = address;
    config.listen_port = port;

    // see FIXME: in server.h
    config.http_request_timeout = 100 * 60 * 1000;
    config.http_response_timeout = 100 * 60 * 1000;
    config.connection_reaper_timeout = 3 * 60 * 60 * 1000;
    // config.connection_reaper_interval = default value

    auto pkey = realm::sync::PKey::load_public(key);
    auto server = realm::sync::Server(path, std::move(pkey), config);
    server.start();
    server.run();
    return 0;
}

Then, build it.

# export CC=clang
# export CXX=clang++

SRCS = server.cpp

OBJS := $(addsuffix .o,$(basename $(SRCS)))
DEPS := $(OBJS:.o=.d)

APP=server

UNAME_S := $(shell uname -s)
ifeq ($(UNAME_S),Linux)
CXXFLAGS=-O3 -g -std=c++17 -Irealm-core/src/ -Irealm-core/build/src/
LDFLAGS= -Wl,-Bstatic \
    -Lrealm-core/build/src/realm/sync/noinst/server/ -lrealm-server \
    -Lrealm-core/build/src/realm/sync/ -lrealm-sync \
    -Lrealm-core/build/src/realm/ -lrealm \
    -lssl -lcrypto -lz \
    -Wl,-Bdynamic -lpthread -lc -ldl
endif
ifeq ($(UNAME_S),Darwin)
CXXFLAGS=-std=c++17 -Irealm-core/src/ -Irealm-core/build/src/ -g -O0
LDFLAGS= -g -Lrealm-core/build/src/realm/sync/noinst/server/ -lrealm-server-dbg \
    -Lrealm-core/build/src/realm/sync/ -lrealm-sync-dbg \
    -Lrealm-core/build/src/realm/ -lrealm-dbg \
    -lz -lcompression \
    -framework Foundation -framework CoreFoundation -framework Security
endif

# Check APP_VERSION env variable and pass it to CC
ifneq ($(APP_VERSION),)
    CXXFLAGS += -DAPP_VERSION=\"$(APP_VERSION)\"
endif

.SUFFIXES:              # Delete the default suffixes
.SUFFIXES: .cpp .o   # Define our suffix list

all: $(APP)

.cpp.o:
    ${CXX} ${CPPFLAGS} ${CXXFLAGS} -c $< -o $@

$(APP): $(OBJS)
    $(CXX) $(CPPFLAGS) -o $(APP) $(OBJS) $(LDFLAGS)

.PHONY: clean

clean:
    rm -f $(OBJS) $(APP) $(DEPS)

-include $(DEPS)

MKDIR_P ?= mkdir -p

Connecting to the synchronization server occurs in several stages, and you will need to implement several API methods.

    realmApi := dic.GetRealmAPI()
    if realmApi != nil {
        realm := apiGroup.Group("/client/v2.0")
        realm.POST("/app/<realm_app_id>/auth/providers/custom-token/login", realmApi.Login)
        realm.POST("/auth/session", realmApi.NewSession)
        realm.DELETE("/auth/session", realmApi.DeleteSession)
        realm.GET("/auth/profile", realmApi.Profile)
        realm.GET("/app/<realm_app_id>/location", realmApi.Location) // returns the addresses of the authorization and synchronization servers
    }

To obtain a RealmUser, you need to log in with a token from our backend. The authorization methods are also implemented on the backend, so it shouldn't be difficult. Verify the token and return the following response. I will copy pieces of my backend in Go because it's faster. :)

type AuthParams struct {
    Token    *string `json:"token"`
    DeviceID *string `json:"device"`
}

func (r *Realm) Login(c *gin.Context) {
    var auth AuthParams
    err := c.ShouldBindJSON(&auth)
    if err != nil || auth.Token == nil {
        c.String(http.StatusBadRequest, "token is required")
        return
    }
    claims, err := CheckUserToken(r.db, *auth.Token)
    if err != nil {
        c.String(http.StatusUnauthorized, "invalid token")
        return
    }

    userID := claims["id"].(string)
    var deviceID string
    device, ok := claims["device"].(string)
    if ok {
        deviceID = device
    } else {
        deviceID = uuid.Nil.String()
    }

    response := gin.H{
        "user_id":       userID,
        "device_id":     deviceID,
        "refresh_token": r.getRefreshToken(userID, deviceID),
        "access_token":  r.getAccessToken(userID, deviceID),
    }
    c.JSON(http.StatusOK, response)
}

func (r *Realm) getRefreshToken(userID, deviceID string) string {
    claims := jwt.MapClaims{
        "app_id":   "<realm_app_id>",
        "identity": userID,
        "device":   deviceID,
        "access":   []string{"refresh"},
        "salt":     rand.Uint64(),
        "iat":      time.Now().Unix(),
        "exp":      time.Now().Add(time.Hour * 24 * 365 * 10).Unix(),
    }
    token := jwt.NewWithClaims(jwt.SigningMethodRS256, claims)
    str, _ := token.SignedString(common.Settings.AuthRealmKey)
    return str
}

func (r *Realm) getAccessToken(userID, deviceID string) string {
    iat := time.Now()
    exp := iat.Add(time.Hour)

    claims := jwt.MapClaims{
        "app_id":   "<realm_app_id>",
        "identity": userID,
        "device":   deviceID,
        "path":     "/" + userID + "/db_name",
        "access":   []string{"download", "upload", "manage"},
        "salt":     rand.Uint64(),
        "iat":      iat.Unix(),
        "exp":      exp.Unix(),
    }
    token := jwt.NewWithClaims(jwt.SigningMethodRS256, claims)
    str, _ := token.SignedString(common.Settings.AuthRealmKey)
    return str
}

// Server addresses
func (r *Realm) Location(c *gin.Context) {
    response := gin.H{
        "app_id":           "<realm_app_id>",
        "deployment_model": "GLOBAL",
        "location":         "release",
        "hostname":         common.Settings.UserServerURL,
        "ws_hostname":      common.Settings.SyncServerURL,
    }
    c.JSON(http.StatusOK, response)
}

// User information
func (r *Realm) Profile(c *gin.Context) {
    userID, deviceID, err := r.checkRealmAuth(c)
    if err != nil {
        common.ReportError(c, common.AuthError(err))
        return
    }
    response := gin.H{
        "identities": []gin.H{
            {
                "id":            userID,
                "provider_type": "custom-token",
            },
        },
        "data": gin.H{},
    }
    if *deviceID == uuid.Nil.String() {
        response["type"] = "browser"
    }
    c.JSON(http.StatusOK, response)
}

// Start session
func (r *Realm) NewSession(c *gin.Context) {
    userID, deviceID, err := r.checkRealmAuth(c)
    if err != nil {
        common.ReportError(c, common.AuthError(err))
        return
    }
    response := gin.H{
        "access_token": r.getAccessToken(*userID, *deviceID),
    }
    c.JSON(http.StatusOK, response)
}

// End session
func (r *Realm) DeleteSession(c *gin.Context) {
    _, _, err := r.checkRealmAuth(c)
    if err != nil {
        common.ReportError(c, common.AuthError(err))
        return
    }
    c.Status(http.StatusOK)
}

I will leave working with JWT as homework.

A couple more things. The client connects via WebSocket, and we will need Nginx to verify the HTTPS certificate and forward the socket to our Realm server. You can use anything else as well.

daemon off;

events {
    worker_connections  128;
}

http {
    server_tokens off;
#    include       mime.types;
    charset       utf-8;


    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    upstream realm {
        server localhost:9601;
    }

    server {
        access_log    /dev/stdout;
        server_name   server_name;
        listen        443 ssl;
        ssl_certificate certs/server_name.cert;
        ssl_certificate_key certs/server_name.key;
        ssl_session_timeout 10m;

        location /api/client/v2.0/app/<realm-app-id>/ {
            proxy_pass http://realm/;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $host;
        }
    }

    server {
        access_log /dev/stdout;
        listen     9600;

        location /api/client/v2.0/app/<realm-app-id>/ {
            proxy_pass http://realm/;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $host;
        }
    }
}

Then, start it with:

/opt/homebrew/opt/nginx/bin/nginx -c $PWD/nginx.conf

You can then start testing.

The client accesses your backend, receives a JWT, and sends it to Realm for authorization. Realm returns its access keys. The client receives the address of the realm-api server and the synchronization server address (api.server_name and sync.server_name). API is what we wrote in Go. sync.server_name is where it will connect via WebSocket.

The Realm server will verify the authorization token and its signature using the public key we provided through the environment variables.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment