The Dragon's Trap - ProgrammerGuide (redacted).txt

Wonder Boy: The Dragon's Trap
-----------------------------

Quick Guide for programmers
Last updated October 2018
Contact: Omar Cornut <XXXXXX

===============================================
 INDEX
===============================================

 1. CHECKOUT SOURCES/DATA
 2. FILES/DIRECTORY INDEX
 3. BUILDING/PACKAGING
 4. GAME/EDITOR USAGE
 5. LOCALIZATION NOTES


===============================================
 1. CHECKOUT
===============================================

 Command-line:
 # mkdir my_work_folder
 # cd my_work_folder
 # git clone --recursive -j 8 https://XXXXXX
 # git clone --recursive -j 8 -b DragonsTrap2 https://XXXXXXX
 # svn checkout https://XXXXXX/trunk/WonderBoy/trunk game

 code/     
   The Git repository to check-out is: https://XXXXXX
   Branch: 'master'
   Web access: https://XXXXXX
   You will be granted access to this repository via your github account.
 
 engine/
   The Git repository to check-out is: https://XXXXXX
   Branch: 'DragonsTrap2'
   Web access: https://XXXXXX
   You will be granted access to this repository via your github account.   
 
 game/  
   The SVN repository to check-out is: https://XXXXXX/trunk/WonderBoy/trunk
   You will be granted access to this repository upon asking Omar.
   Contents creators (artists, designers) can check-out this repository only.
   We are publishing pre-built Windows binaries here for their use.
 

===============================================
 2. FILES/DIRECTORIES INDEX
===============================================

(A)

  code/ ................... GAME CODE (on GIT)

  code/misc ............... misc stuff
      /misc/docs/               this documentation
      /misc/mw2_backup/         original game binaries extracted from sources .hex files
      /misc/premake/            premake5 exe to create .sln (or other build tools) files
      /misc/sym2h/              simple tool to create symbols from WLA-DX/WLAZ80 into a C++ .h file
      /misc/tsv2h/              simple tooo to create localization key hash from a .tsv file

  code/libs ............... external libraries for project
      /libs/soloud/             simple open-source audio library (http://sol.gfxile.net/soloud)
 
  code/original ........... original game Z80 code. converted from an old assembler format to a format assemblable with WLA-DX.
      /original/1-vanilla/      unmodified 1989 SMS + 1991 GG versions
      /original/2-edited/       hand-modified for changes that were not easy to automatize
      /original/3-converted/    converted to WLA-DX, buildable
      /original/SourceConvert/  helper tool to convert 2-edited/ to 3-converted/, not used anymore
  
  code/packaging .......... packaging helpers/scripts
      /packaging/linux/
      /packaging/macosx/
      /packaging/ps4/
      /packaging/switch/
      /packaging/windows/
      /packaging/windows/steam/
      /packaging/windows/gog/
      /packaging/xb1/
      
  code/project ............ project files for e.g. VisualStudio, XCode APART from main wb.sln (FIXME: this is a mess.. wb.sln should be here too)
  
  code/srcs ............... game sources (small ~50 files, 1.1 MB of sources)
      /srcs/anim.cpp            animation code, tools 
      /srcs/app.cpp             application misc: command-line, screen resolution, steam/gog wrappers, trophies, settings saving, screen/video capture
      /srcs/audio.cpp           high-level audio code, tools
      /srcs/bonus.cpp           code related to the game bonus levels (most of it disabled as we changed the bonus design later on)
      /srcs/builder.cpp         output files/folder for packages, build texture atlas
      /srcs/config.cpp          configuration (end-user + developer)
      /srcs/drawing.cpp         code draw game scenes
      /srcs/editor.cpp          editor code + various tools
      /srcs/emu.cpp             master system emulator
      /srcs/game.cpp            various game code, handling effects, hooks with emulator, main game flow
      /srcs/gif.cpp             gif loader/player for gallery
      /srcs/hacks.cpp           hacks to emulated ROM and emulated virtual machine
      /srcs/hints.cpp           hints/tutorials
      /srcs/inputs.cpp          inputs code, mapping
      /srcs/imgui_ext.cpp       minor extensions to dear imgui
      /srcs/localize.cpp        localization code
      /srcs/main.cpp            application init, main loop, main drawing code (some of it should be moved to drawing.cpp...)
      /srcs/misc.cpp            minor stuff
      /srcs/particle.cpp        simple particle system
      /srcs/password.cpp        retro password decoding/encoding
      /srcs/renderer.cpp        high-level rendering functions
      /srcs/rom_symbols.h       symbols from original z80 code translated into C defines
      /srcs/savegame.cpp        end-user savegame code/data
      /srcs/soak_tester.cpp     helper tool to load all locations/animations and see how the game behave
      /srcs/symbols.cpp         symbols loading code
      /srcs/texture.cpp         high-level texture handling code
      /srcs/timeline.cpp        simple class to handle the intro timeline
      /srcs/ui.cpp              game ui, inventory, gallery, error dialogs, etc.
      /srcs/ui_options.cpp      options menus
      /srcs/ui_richtext.cpp     rich-text parsing/drawing helper
      /srcs/ui_title.cpp        title screen/save menu code
      /srcs/ui_tools.cpp        helpers for game ui: menus etc.
      /srcs/video.cpp           video player for gallery and boot logos
      /srcs/wb.cpp              data structures, code, helpers, related to the original game data structures (not the new game)
      /srcs/libs/soloud_ext/    soloud bits, ps4/switch adapter for soloud
      /srcs/libs/spine_wb/      wrapper for loading spine data (file/texture access)
      /srcs/sn76496/            sms sound chip emulator, vgm logger for tools
      /srcs/uwp/                uwp cruft for XB1 version
      /srcs/z80/                sms cpu (z80) emulator, from mame (BSD)  

(B)
      
  engine/ ................. ENGINE CODE ("LINGUINI") (on GIT)
                            Lots of stuff. Note that we don't use all of the engine.
                            - We use rendering for simple stuff, render to texture, shader compilation (all game rendering are piped through the ImGui library)
                            - We use inputs (InputMgr)
                            - We use the savedata code (PersistentMgr)
                            - We use the trophy code (AchievementMgr)
                            - We use various utilities, such as the serialization api (Reflect), string helpers
                            - We DO NOT use any of the Audio. See in wb_premake.lua for a list of the folders we are stripping from compilation.
                            - We DO NOT use most the high-level sprite, ui, roar stuff from the engine.
  engine/Classes/
  engine/Classes/engine_cpp/
  engine/Classes/engine_sdl/    sdl2-specific code (for mac+linux)
  engine/Classes/engine_ps4/    ps4-specific code
  engine/Classes/engine_uwp/    xb1-specific code (the engine isn't actually full UWP now, thanks god)
  engine/Classes/engine_XXXX/   switch-specific code
  engine/Classes/engine_win32/  windows-specific code
  engine/Classes/highlevel_cpp/
  engine/Classes/extern/
  engine/Classes/shared/
  engine/Classes/shared/opengl/ OpenGL rendering (windows, mac, linux, etc.)
  engine/Classes/shared/dx11/   DX11 rendering (windows, xbox)
  engine/Classes/shared/XXXX/   XXX rendering (switch)
  
(C)

  game/ ................... GAME DATA (on SVN)
  
  game/data/                game data
      /data/anims/              animation configuration file (.cfg lua)
      /data/atlas/              atlas output (packeged textures + indexes)
      /data/config/             user configuration files (not committed)
      /data/font/               font files
      /data/gallery/            gallery data and configuration files
      /data/locations/          level data, game settings, level mods, atlas config, fx mappings
      /data/music/              music files (.ogg) + lua settings
      /data/replays/            replay files (for debugging)
      /data/rom/                original rom file + symbols
      /data/rom/saves/          save files (for debugging)
      /data/sfx/                sfx files (.ogg) + .lua settings
      /data/shaders/            shaders (glsl+pssl)
      /data/sparse/             texture metadata information (stores tiling/opacity info), stored as 1 file per directory
      /data/spine/              spine files (barely used)
      /data/strings/            localization (.tsv exported from GoogleSheets with our tools)
      /data/tex/                source textures (.png)
      /data/video/              video files for boot logos (.ogv)

  game/bin_pc/            generated by the BUILDER when creating PC packages. roughly based on data/ with various differences (individual .png for animations turned into atlases, unused files stripped out, shaders for other platforms stripped out, etc.).
  game/bin_ps4/           generated by the BUILDER when creating PS4 packages. "
  game/bin_xb1/           generated by the BUILDER when creating XB1 packages. "
  game/bin_nx/            generated by the BUILDER when creating Switch packages. "
   
  game/exe32/             executable + dll (windows 32)
  game/exe64/             executable + dll (windows 64)
  game/tools/             development tools (ffmpeg for capture, script to fetch localization data from google sjeets)


===============================================
 3. BUILDING/PACKAGING
===============================================

Step 1: build code for the platform.
Step 2: build data for the platform.
Step 3: run packaging script for the platform.

The "Debug" project built both GAME and ENGINE in debug mode to maximize debugging capabilities.
Typically we use "DebugOpt" which builds the ENGINE in optimized mode and the GAME in Debug.
The game is quite lightweight in term of CPU and runs in debug.

The prebuilt solution for Windows and Consoles platform (code/wb.sln) is generated from the wb_premake.lua file using the premake.bat batch file.
You don't really/often need to regenerate it, we are committing the .sln to the repo for practical purpose.
The same premake files can be used to create Makefile for Mac/Linux or potentially e.g. xcode projects.
We are using Visual Studio 2015 to build for Windows version and with dedicated VSI integration for the consoles.
The Free Community Edition of VS2015 is ok.


3.1. BUILDING/PACKAGING FOR WINDOWS
-----------------------------------

Use code/wb.sln project, or command-line:
 # devenv.com /build "Release|x64" /project wb code\wb.sln 
 # devenv.com /build "Debug Optimized|x64" /project wb code\wb.sln 

To Execute, make "Working directory" points to the game/exe/ folder.

Both Windows x32 and x64 versions are functional.
The .exe is built under code/bin/ but also copied under game/exe32/ or game/exe64/ (svn) using a post-build event.

Internal team members (artists/etc.) can checkout the SVN and can work from the source files in data/ without the need 
the for bin_pc/ (exported) directory.
Programmers can use the script code/packaging/windows/build_commit_exe.bat to quickly build an EXE and commit it to SVN
for developers (this is what we did for the team during development, probably not needed any more).

Build data for packaging:
- Run PC version, press F to show Editor if hidden, open Builder tool.
- In 'Builder' window -> 'Sparser' tab -> click 'Build for all game'. This will create the files in the data/sparse/ directory. You only need to be do it once for all platforms (same files will be ruesed)
- In 'Builder' window -> 'Builder' tab -> click 'Build' button on BIN_PC/ repository.
  It will build/copy required files into the data_pc/ folder.

Shaders (if you modify them)
- Build modified DX11 shaders with data/shaders/00_BuildShadersWinDX11.bat (but they should be committed on SVN already)
- Commit modified shader binaries to SVN. Note that Windows and XB1 shaders are using same file names. We are committing the DX11 ones.

To package/release neutral:
- Run code/packaging/windows/package_win64.bat. Build .ZIP will be output in ./packages

STEAM WINDOWS PACKAGE

- Open wb_premake.lua, make sure PASTA_STEAM=true, PASTA_GOG=false
- Run premake.bat (this is already committed to repo)
- In code/skus.h make sure the default SKU is enabled. This is the default so normally you don't need to change it. (sorry for the misleading name!)
    #define WB_SKU_PS4_SCEA   1   // Default SKU (no extra logo) ok for Steam, XB1, etc.
- In Visual Studio Project Properties -> General -> Target Platform Version, change 8.1 to whatever 10.xx SDK you have installed .
  THIS IS IMPORTANT, the game will build without it, but with SDK 8.1 the code in DX11GraphicContext.cpp that enable DXGI_SWAP_EFFECT_FLIP_DISCARD/DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL won't run.
  Somehow could not get our Premake version to output project files with SDK 10
- Build both Win32 Final and x64 Final
- Create script in code/packaging/windows/steam/ based on the run_build_XXXX.bat with your username+password, don't commit.
- Run script in code/packaging/windows/steam/run_build_MYUSERNAME.bat
- Head to Steamworks:
    https://partner.steamgames.com/apps/builds/XXXX
- Add new build to a branch, publish etc.

GOG WINDOWS PACKAGE

- Open wb_premake.lua, make sure PASTA_STEAM=false, PASTA_GOG=true
- Run premake.bat
- In code/skus.h make sure the default SKU is enabled. This is the default so normally you don't need to change it. (sorry for the misleading name!)
    #define WB_SKU_PS4_SCEA   1   // Default SKU (no extra logo) ok for Steam, XB1, etc.
- In Visual Studio Project Properties -> General -> Target Platform Version, change 8.1 to whatever 10.xx SDK you have installed.
  THIS IS IMPORTANT, the game will build without it, but with SDK 8.1 the code in DX11GraphicContext.cpp that enable DXGI_SWAP_EFFECT_FLIP_DISCARD/DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL won't run.
  Somehow could not get our Premake version to output project files with SDK 10
- Build both Win32 Final and x64 Final
- Run script in code/packaging/windows/gog/run_build_gog.bat
- Head to GOG:
    https://devportal.gog.com/panel/games/XXXXXX/builds

DRM-FREE NEUTRAL WINDOWS PACKAGE (e.g. AMAZON, EPIC)

- Open wb_premake.lua, make sure PASTA_STEAM=false, PASTA_GOG=false
- For Amazon Luna/Solstice, set PASTA_SOLSTICE=true
- Run premake.bat
- [SAME STEPS ARE ABOVE.]
- Run script in code/packaging/windows/package_win64.bat THEN REMOVE TOOLS/ FOLDER FROM ZIP FILE.


3.2. BUILDING/PACKAGING FOR MAC, LINUX
--------------------------------------

Mac/Linux port by Ethan Lee <XXXXXX>
See Build.README files in packaging/linux/, packaging/macosx/ for building and packaging.

Build code:
 # buildSDL2.sh

Build data: 
- Same as Windows platform (note that the output bin_pc/ will be different because the shaders are OpenGL version).

Shaders (if you modify them):
- Build modified OpenGL shaders with data/shaders/00_BuildShadersGLSL.bat (but they should be committed on SVN already)


3.3. BUILDING/PACKAGING FOR PS4
----------------------------------

Use code/wb.sln project.
You need to build the Windows version first to use the 'BUILDER' tool.
You need to build the sparse/ directory once in the 'BUILDER' tool (see Windows section).

Run executable with '-localdata' parameter to use files from data/ on PS4 harddrive.
Run executable with '-remotedata' parameter to use files hosted from remote development PC.

Versions/SKUS:
- Make sure that 'code/skus.h' contains the right and only one SKU definition.
- You need the Windows version to be built with the same SKU settings prior to using the Builder.

Build data for debugging/running:
- Run data/shaders/00_BuildShadersPS4.bat if applicable (once of when changing shaders).
- Build run PC version (with skus.h edited), press F to show Editor if hidden, open Builder tool.
- Setup remote PS4 path to e.g. Q:\192.168.1.18\data (according to your own mapped drive letter and host nane/ip address) 
- In 'Builder' window -> 'Builder' tab -> click 'Build' button on BIN_PS4_REMOTE/ repository 
  It will build/copy required files into the data/ folder of the devkit.

Build PS4 package:
- Build run PC version (with skus.h edited), press F to show Editor if hidden, open Builder tool.
- In 'Builder' window -> 'Builder' tab -> click 'Build' button on BIN_PS4_LOCAL/ repository 
  It will build/copy required files into the data_ps4/ folder.
- Use suitable packaging/ps4/create_pkg*.bat script to create a PKG file from the Final executable + BIN_PS4_LOCAL/ folder.

Shaders (if you modify them):
- Build PS4 shaders with data/shaders/pssl/00_BuildShadersPS4.bat (but they should be committed on SVN already)
- You will need to build the repo again to copy shaders from data/shaders/ to bin_ps4/shaders/


3.4. BUILDING/PACKAGING FOR SWITCH
------------------------------------

- Use code/wb.sln project.
- You need to build the Windows version first to use the 'BUILDER' tool.
- You need to build the sparse/ directory once in the 'BUILDER' tool (see Windows section).

Versions/SKUS:
- Make sure that 'code/skus.h' contains the right and only one SKU definition.
- You need the Windows version to be built with the same SKU settings prior to using the Builder.

Build bin_nx/ folder:
- Run data/shaders/00_BuildShadersSwitch.bat if applicable (once of when changing shaders).
- Build run PC version (with skus.h edited), press F to show Editor if hidden, open Builder tool.
- In 'Builder' window -> 'Builder' tab -> click 'Build' button on BIN_NX/ repository 

Build Switch package:
- In VisualStudio, click the 'wb' project -> properties -> general
  Change 'Metadata Source' of the SWITCH64 Final project to point to the right .meta file according to the SKU.
- Build SWITCH64 FINAL version in VisualStudio. Output is .nsp file (which incorporate the bin_nx/ folder.)


3.5. BUILDING/PACKAGING FOR XBOX ONE
------------------------------------

(Procedure based on XDK October 2016)

Use code/wb.sln project.
You need to build the Windows version first to use the 'BUILDER' tool.
You need to build the sparse/ directory once in the 'BUILDER' tool (see Windows section).

Build the project once with Durango target to generate the Durango\Layout\Image\Loose directory

Build data for debugging/running:
- Run PC version, press F to show Editor if hidden, open Builder tool.
A. 
- Setup local XB1 path to e.g. C:\your_path_to_wb\code\Durango\Layout\Image\Loose\data
- In 'Builder' window -> 'Builder' tab -> click 'Build' button on BIN_XB1_REMOTE/ repository.
  It will build/copy required files to your local copy, and then Visual Studio will push them to your XB1 when running the code.
  All files located in the Loose directory will be pushed to the Xbox One.
or B.
- Setup remote XB1 path to e.g. X:\your_path_to_wb\data (according to your own mapped drive letter and host nane/ip address) 
- In 'Builder' window -> 'Builder' tab -> click 'Build' button on BIN_XB1_REMOTE/ repository.
  It will build/copy required files directly into the data/ folder of the devkit to have faster sync when building code.
  Make sure to deactivate file deletion during sync!

Shaders (if you modify them):
- Build XBOX shaders with data/shaders/pssl/00_BuildShadersXB1.bat (nb: ones commmited on SVN are Windows one commmited with DX10-feature level!)
- If you want to debug shader using PIX (step by step debug of shader's source code) you can build shaders with data/shaders/00_BuildShadersXB1Debug.bat.
  Debug shaders are significantly slower and must not be used in a master package.
- You will need to build the repo again to copy shaders from data/shaders/ to bin_xb1/shaders/

Packaging:
- Step 1: Build Durango executable in Final, with #define WB_EDITOR 0 in stdafx.h
- Step 2: Build data/ folder to your local Layout\Image\Loose directory.
- Step 3: Run code/packaging/xb1/StreamingInstallManifestMaker.exe
  All directories should already be setup correctly.
  Click AddDirectories. \code\Durango\Layout\Image\Loose directory should be highlighted.
  Click Ok, then Generate Manifest. Save dialog should point to code/packaging/xb1/StreamingInstallManifest.xml.
  Click Save.
- If you want to generate a package for testing or to do an optional submission, use MakePackage.bat
  Package will be encrypted for testing and you will be able to install it on a dev kit.
  (a good way to verify this is to look at the end of the ekb file. If it ends with XXXXX, it is encrypted for testing)
- If you want to generate a master package for final submission, use MakeEncryptedPackage.bat
  (ekb file generated should NOT end with XXXXX)
  Fully encrypted packages cannot be installed on a dev kit. Check your build with MakePackage.bat first.
- Packages can be found in the code\packaging\xb1\Packages directory.

Tips:
- Ignore loading symbols for DLL so VisualStudio debugger stops taking years to boot anything.
  Options->Debugging->Symbols->Automatically load symbols for [X] All modules, 
  then click "Specify Excluded Modules" and add those DLL:

  XXXX.dll



===============================================
 4. GAME/EDITOR USAGE
===============================================

Run once, then go to game/data/config/Settings.cfg and set Editor=true in there.

Game (Keyboard)
   Arrow Left/Right: Run
   Arrow Down: Crouch (when available)
   Arrow Up: Open Door
   S/D : Attack / Jump
   W/E : Weapon / Inventory
   P : Debug pause (no menu)
   CTRL+P : Normal pause (menu)
   
Game (Gamepad)
   D-Pad Left/Right: Run
   D-Pad Down: Crouch (when available)
   D-Pad Up: Open Door
   Start : Normal pause (menu)
   
Tools:
   F : Show/Hide tools
   F2 : Toggle new/old visuals
   F3 : Toggle widescreen/4:3 (debug only)
   F4 : Toggle 30/60 hz (debug only)
   F5 : Disable Monsters
   F9 : Step Frame
   Center mouse-click to teleport.
   Use menus to discover features.
   
Recommended Tools:
- Windows->Log: Good old log.
- Windows->Browser: Savestates, tweak form/equipment/gold/life, visualize game objects
- Windows->Location: Mini-map of the current location, Center mouse click to teleport, change location, etc.
- Windows->Builder: To sync game data with PS4/XB1
   
Editor:
   SPACE : Toggle Editor
   CTRL+S : Save all
   etc..

Joypads:
   XInput controllers supported.
   Some drivers expose 2 controllers to the software, and we don't always detect the correct one.
   Select gamepad in "Options->Inputs->Gamepad No"

Tools on PS4:
- Click TouchPad to toggle imgui tools. Hold SQUARE to use imgui navigation (experimental)
- Use native mouse.
- Best: Install Synergy on your PC, add a virtual screen called "name", then run PS4 version with '-synergy <name> <ip>' to use PC mouse+keyboard on PS4.

Tools on XB1:
- Press Select to toggle imgui tools. Hold X to use imgui navigation (experimental).
- FIXME: Add support for native Mouse (easy)
- FIXME: Add support for Synergy

Tools on Switch:
- Use XXXXXXXXXXXXXXX


===============================================
 5. LOCALIZATION NOTES
===============================================

The sheet is located at https://docs.google.com/spreadsheets/d/XXXXXXXXXXXXX/edit

Update data from the localization sheet using those batch files.
When editing localization data:
  - game/tools/update_localization.bat    Retrieve data from GoogleSheet, save in data/strings
For programmers:
  - code/misc/tsv2h/build.bat             Retrieve data from GoogleSheet, save in data/strings, update the localize_keys.h file (used by code)

When the boots in editor mode, it reads "main_unfiltered.csv", filters it according to current SKU and writes "main.csv".


===============================================
 end of file
===============================================