WARNING: OUTDATED TUTORIAL. I will get around to updating this at some point, but for now I don't recommend following this tutorial to build the apps.
Seen all the tutorials for building RSDKv5(U) on Mac that you can muster but still haven't gotten it up and running yet? Well, I hope you can handle one more, because this will finally be the one to fix it! I can guarantee it because I was in your shoes and had to figure all this stuff out myself. This is a text tutorial for learning how to decompile RSDKv5(U) and run Sonic Mania Plus or Sonic Origins on your Mac.
How to build RSDKv5:
Step 1: Install Xcode and the command tools (the command tools can be installed by running this command: xcode-select --install but only after installing Xcode). If you already have them, move on to step 2.
Step 2: You have to clone Sappharad's Sonic Mania GitHub repository, so run this command:
git clone --recursive https://github.com/Sappharad/Sonic-Mania-Decompilation
You should find it in your user folder (Macintosh HD/Users/yourusernamehere/).
Step 3: Open the folder you cloned, open the SonicMania.xcodeproj file and hit the triangle (that's the build button) in the top left corner. After a little bit you should se a window pop up that says "Build Succeeded". Click the arrow near the Products folder on the left hand side and you should see a white Game over two red words. That's the Game.dylib file and it's the game's "logic" (this will make more sense when we go to build v5U later). Right click it and hit "Show in Finder" to see the file in all its glory. Make sure to copy it and paste in the dependencies/RSDKv5/dependencies/mac/ folder.
Step 4: Open the RSDKv5.xcodeproj file in the dependencies/RSDKv5 folder and hit the same triangle build button in the top left hand corner. After the build succeeds, you will see the app runs, but the screen is black. That's where the next step comes in.
Step 5: You need a Data.rsdk file of Sonic Mania Plus. I will not tell you where to get it, or if you should or shouldn't pirate it in any way. Hey, I'm not going to question your morals. Once you get it, make a folder somewhere and place it in there. To follow along with building v5U later on, I also recommend you place the Data.rsdk file in this directory: Macintosh HD/Users/yourusernamehere/Library/Application Support/RSDKv5/.
Step 6: Find RSDKv5.app in the Products folder directories tree on the left hand side (you can find it by clicking the folder icon below the close button), copy it, and place it in the same directory as where you copied your Data.rsdk.
Then simply open the app! You are now playing Sonic Mania Plus on your Mac! But what if you want mods? There's an entire community making them on Gamebanana for Pete's sakes! Or what about Origins? Virtually the ONE PLATFORM that it isn't available on is Mac! Well, you need to go one step further and build v5U...
How to build RSDKv5U:
What's the difference between v5 and v5U? Well, U stands for Ultimate, and v5U is the same version of the Retro Engine that is used to run Sonic 3 and Knuckles in Sonic Origins. It features backwards compatibility with v4 (Sonic 1 and 2 for mobile) and v3 (Sonic CD for mobile), but not v2 or v1 because they were in unofficial projects like Sonic Nexus in 2008.
Follow steps 1 and 2 if you haven't already.
Step 3: Open the SonicMania.xcodeproj from earlier. Next to where it says "My Mac", you should see "SonicMania". Click it and change it to "SonicMania (v5U)". Build it and then copy the resulting dylib file (the one that says "SonicMania-v5U") to your desktop. We'll be needing that later...
Step 4: Open up the RSDKv5U.xcodeproj in the dependencies/RSDKv5 folder. Click the RSDKv5U on the left, click "Build Phases", and then the arrow near "Link Binary with Libraries", and then the plus icon that comes up. Type in "AppKit" and then hit "AppKit.framework".
Step 5: Go back to the /dependencies/mac/ folder and then create two new folders: "libogg" and "libtheora". Inside both of those new folders, create one new folder simply called "macosx". Then, copy the Ogg.framework folder from the Mac folder and then paste it into the macosx folder in the new libogg folder. Then, copy the Theora.framework folder from the Mac folder and then paste it into the macosx folder in the new libtheora folder.
Step 6: Hit the triangle button once again and the build will actually fail this time. Don't worry, this is what we want. The error with the red will contain <os_types.h>, so replace it from <os_types.h> to <Ogg/os_types.h>. Then build again and the build will succeed, the app will run, and the screen will just be white. Again, this is exactly what we want.
Step 7: Locate the RSDKv5U app you built, right click it and then hit "Show Package Contents". Make a new folder called "Resources" in the Contents folder and next to the MacOS folder, and then you need to place the Game.dylib file that you copied to your desktop into that Resources folder. Now drag the app into the same folder that you placed the Data.rsdk file when we built RSDKv5 earlier, and if you want mods, then make a new folder called "mods" and explore to your heart's content! Make sure you extract the compressed mod file into your "mods" folder.
Note: Macs cannot run DLL mods, so if you find a mod that doesn't have a Data folder and a mod.ini file, that mod will not work. If you want the equivalent of DLL mods on Mac, you have to use GameAPI mods.
Step 8 (this one is kind of an overall quality-of-life step): to make your app feel more complete, you probably should also put in a proper app icon! Go to the RSDKv5 Decompilation folder and depending on whether you built v5 or v5U, go to that corresponding folder. Find and note down the .ico file in that folder. Then go and find the RSDKv5/v5U app you made, right click it, and then hit "Get Info". Then, drag the .ico file to that shape in the top left hand corner, and your app now has a corresponding icon!
How to run Sonic Origins RSDK:
Step 1: Ensure that you follow EVERYTHING I JUST SAID FOR V5U except for Step 7 (yes, even follow Step 8 if you want an app icon): that needs to be filled in with this.
Step 2: Rip the Sonic Origins RSDK files from whatever version from Sonic Origins you have. There are plenty of tutorials on the REMS Server I link at the end.
Step 3: Grab the RSDK file corresponding to the game you want to play (Sonic 1u, Sonic CDu, Sonic 2u, or Sonic 3ku).
For Sonic 3, all we can do now is drag the file into the same folder that the RSDKv5U app is in and load, and even then, it doesn't even function properly. It just says a black screen with a SEGA logo and loading the dev menu makes it say "Project Cobalt", which was the project name for the Retro Engine version of S3K. The rest of these steps can only be followed if you want to run Sonic 1, CD, or 2.
Step 4: We have to unpack these RSDK files so that we can place Music and SFX inside of them. To do this, you will need two tools: Hex Fiend, which you can download from the App Store, and MainMemory's RSDKv5Extract and RSDKv5Pack tools, so go get them both and extract the 7z files of the latter tools.
Step 5: You will realise that the Extract and Pack tools are designed for Windows. Well, we can get around this by using a command line tool called Mono, so click the link to follow a tutorial for downloading.
Step 6: Drag in whichever Origins RSDK file you want to run and its corresponding mobile RSDK file into the RSDKv5Pack folder.
Step 7: Right click the mobile RSDK file and click "Open with Hex Fiend." Use Hex Fiend to change the RSDKvB value at the top of the file to RSDKv5, as this is necessary if we want to unpack.
Step 8: Open the Terminal, type in "mono", then drag in the Extract EXE and then also the Origins RSDK file and this will create a folder with the name of the RSDK file. Then do the exact same thing but with the mobile RSDK.
Step 9: After doing all of that, find the Music and Sound FX folders in the Mobile RSDK extraction folder and paste into the data folder that is inside the u folder you created. (Note: for Sonic 2, you have to convert all the sound effects in the Global folder from ogg to wav since v5U does not support ogg sound effects and if you try to load them in by jumping or whatever the game crashes. A free app on the App Store you can do this with is called Switch Sound File Converter. If you don't want to bother with that, you should use the decompiled cleanup scripts and put the Sonic 2 one into the same data folder as where you put the music and sound effect files.)
Step 11: Copy and paste your completed folder (Note: it MUST be named either Sonic 1u, Sonic CDu, Sonic 2u, or Sonic 3ku, and NOT DATA) into the RSDKv5Pack folder.
Step 12: Open the Terminal, type in "mono", then drag in the Pack EXE and the folder you pasted, and you should see a new .rsdk file.
Step 13: Move that file into the same folder you have your RSDKv5U app and run it once.
Step 14: Open the Settings.ini file and underneath "language=0" type "devMenu=y" is you want the dev menu and "gameType=1" if you want the TRUE Origins experience. (Note: if you add the latter option, you must be using version 1.04 Origins RSDKs as there is an operand in the scripts that is nonexistant in the v1.00 RSDKs and the game crashes.)
BOOM!!!!!!!!!!!!
I sincerely hope this helped you, and if it didn't, go speak with the Retro Engine Modding Server as they are filled with smarter people than me (or me if I catch your post) and will be able to help you. Alternatively, you could just make a comment on this post, but don't expect a timely response as I hardly check GitHub comments. Happy playing! :D
Excellent tutorial, thank you for making this!