Do you really need an instant replay program for Linux? Something similar to Shadowplay? GPU Screen Recorder is just that! Here's my little guide for setting it up.
NOTE: This guide was made for KDE Plasma. If you're on a different desktop environment, you'll have to find out startup scripts and keybinds on your own.
If you're on Arch Linux, use the AUR package.