Ansel is an NVidia technology which enables a photomode with shaders in pre-defined games. A game doesn't support Ansel out of the box, it has to be setup for that, which is usually done by using the Ansel SDK. The Ansel SDK is available to people who register for an account with NVidia, and I don't have such an account, so all I provide here is what I figured out myself from what NVidia provided online and what I've seen in game code.
Ansel works with call-backs, which means the game has to register a function with Ansel which is then called when Ansel needs
to, e.g. to update the view matrix or e.g. when to enable Ansel. Ansel runs in the background listening to raw keyboard input
and if the user presses the keyboard shortcut for the Ansel photomode, by default Alt-F2, it will call the function
a game has registered with Ansel to check whether the game finds it appropriate to enable Ansel. This function is
registered with Ansel by the game engine. If no function is registered Ansel will show a simple dialog that a supported application
has to be opened for Ansel to work. If the function is registered however, and the game denies enabling it, by returning 0,
Ansel will show a dialog that tells the user Ansel can't be enabled at this point.
We of course want to use Ansel whenever we want, e.g. during in-engine cut-scenes and other areas where the game thinks it's not useful for a user to enable Ansel, so what we need to do is find the function registered with Ansel which does this checking and 'simply' remove the checks.
There are several things we have to do:
- Find the location where in the Ansel dlls the game is called to do a check for enabling Ansel. This way we can determine where the game's function is located
- In the game's check function, find the checks which disable Ansel in areas where it shouldn't be enabled. We can't skip this function as it's likely the game does pre-setup work here.
We're going to use Cheat Engine and one of its more obscure, but nevertheless very cool, feature: Break & Trace.
Using X64Dbg, attaching it to the game exe (in this case Mass Effect Andromeda but that's not important), and searching for strings like enable / allowed etc. I ended up in the function below. What I then did was detaching the debugger and using another debugger, namely Cheat engine. Attaching cheat engine, opening memory viewer, going to the address of the function I found in X64Dbg, I scrolled a bit up and I placed a tracepoint. This is a neat feature of Cheat engine (and why I use it over X64Dbg which does have tracing too but I always end up not using it correctly so I keep coming back to Cheat engine for this) which allows you to track what code is called after a statement.
The portion where Ansel calls the game
NvCamera64.AnselGetFunctionTable+4A26 - 89 41 0C - mov [rcx+0C],eax
NvCamera64.AnselGetFunctionTable+4A29 - 48 8B 9E 98000000 - mov rbx,[rsi+00000098]
NvCamera64.AnselGetFunctionTable+4A30 - 48 8B 7B 60 - mov rdi,[rbx+60]
NvCamera64.AnselGetFunctionTable+4A34 - 48 8B CF - mov rcx,rdi
NvCamera64.AnselGetFunctionTable+4A37 - FF 15 83DE3300 - call qword ptr [NvCamera64.OnInstall+25E420]
NvCamera64.AnselGetFunctionTable+4A3D - 48 8B 53 50 - mov rdx,[rbx+50]
NvCamera64.AnselGetFunctionTable+4A41 - 48 8B 8E A8000000 - mov rcx,[rsi+000000A8]
NvCamera64.AnselGetFunctionTable+4A48 - FF D7 - call rdi // << Call of game check function
NvCamera64.AnselGetFunctionTable+4A4A - 3C 01 - cmp al,01
NvCamera64.AnselGetFunctionTable+4A4C - 0F85 CE030000 - jne NvCamera64.AnselGetFunctionTable+4E20
NvCamera64.AnselGetFunctionTable+4A52 - 88 45 C0 - mov [rbp-40],al
NvCamera64.AnselGetFunctionTable+4A55 - 4C 8D 4D E8 - lea r9,[rbp-18]
NvCamera64.AnselGetFunctionTable+4A59 - 4C 8D 45 F0 - lea r8,[rbp-10]
NvCamera64.AnselGetFunctionTable+4A5D - 48 8B 96 98000000 - mov rdx,[rsi+00000098]
NvCamera64.AnselGetFunctionTable+4A64 - 48 8D 4D D0 - lea rcx,[rbp-30]
NvCamera64.AnselGetFunctionTable+4A68 - E8 A3C90000 - call NvCamera64.AnselGetFunctionTable+11410Example, if you go to NvCamera64.AnselGetFunctionTable+4A26 and right-click it in Memory viewer/disassembler, you can then select 'Break and Trace'. Specify e.g. 5000 for the # of instructions to track and go back to the game. We now have to activate Ansel, using Alt-F2, preferably in an area where it's not allowed, e.g. in the menu.
After pressing Alt-F2 the trace window of Cheat engine will be filled with instructions in a tree form: it will nest calls as subnodes of the call instruction so you can easily see the structure of the code. We see that the Ansel code is located in the 7FF93xxxxx area and we can easily check the game is located at 140000000. So if the trace we just obtained calls an address which starts with 14 we've found the function which is checking for Ansel whether it can be enabled or not.
In the above snippet that function is called at the line NvCamera64.AnselGetFunctionTable+4A48 - FF D7 - call rdi. Installing a new driver will likely make this code move a bit, so it's not that reliable but until a better way to immediately pinpoint the location, we have to do it with this.
Now that we have a function, we need to remove the checks which disable Ansel in a spot we want to use Ansel. As we can see from the code snippet above, if it returns 0 in al (the byte portion of RAX) it's disabled, so the easiest is to simply open the game's own menu, and in cheat engine or other debugger, set a breakpoint on the first instruction of the function we found. After that we go back to the game (which is now in the menu) and press Alt-F2. The game should then break on the breakpoint set. If you're using Cheat engine, setting a breakpoint is done in the disassembly view of memory viewer using F5. We now have to step through the function and see where it does a check and jumps to the end if it fails.
Step through the instructions with F8, and be careful if you end up on a jump-relative instruction (e.g. jne, je, jnz etc.). if the code will take the relative jump, the debugger (Cheat engine in this case) will show you with an arrow. If it's to an address far away, you likely are on a check which failed. Make a note in notepad of this instruction (copy it) and step further. If the code indeed jumped to the end and makes sure al is set to 0, by using e.g. xor al, al, then next time you end up this function (by repeating the steps) you can nop the check you made a note off by right-clicking it and selecting 'Replace with code that does nothing'.
Some games have just a few checks, others have plenty. Though if you remove one after the other, in the end the function will no longer have failing checks and will return 1 as if everything is all right. Ansel will then be enabled.
To make this a more easier toggle, you can wrap the disabled checks in a cheat table script and disable/enable them on the fly.
Easy! :)
@dylanno, no that's not going to work. I wouldn't try it even if I were you: starting cheat engine and modifying code in the exe is something it will surely pick up and you'll be banned.