In anticipation of my shiny new Xbox One X showing up I began buying and ripping 4K UHD Bluray disks, so my Plex media server would have some content on it that could take advantage of my new hardware.
I was pretty dissapointed when I discovered none of it worked very well for me at all. After a lot of trial and error I think I've worked out the situation of 4K media on Xbox One, and how to get your library ready to handle it.
My problem is my televsion does not support 10 bit HDR video. I bought it at a very steep discount, and while its a great simple 4K UHD TV, it does not support 10 bit HDR video. The exact model is a Dish TV NZ 55" UHD TV.
I serve my files via Plex from a local server, over gigibit ethernet. On the Xbox I normally use the DLNA option in the Media Player app, but I have also experimented with using the official Plex Xbox One app. I decided I hate using this after some experimenting, and I'll explain why later.
Because my TV does not support HDR, a few different issues occurred. The first is that via the Media Player app, HDR content would result in a black screen, a green screen or a "Codec Not Supported" error. However in the Plex app content would play just fine. I later discovered this was due to Plex transcoding my content. Plex does a good job of making it unclear exactly what content you are getting on your TV. If it detects that your playback device does not support the content you are playing, it will transcode it. However, it only transcodes to h264 video. The issue with this is that the Xbox does not support 4K h264 video; 4K must be encoded in HEVC. The result is that Plex transcodes your content all the way down to 1080p, which at first made me think I'd wasted my money on all this expensive gear, because the video looked exactly the same as before. Meanwhile nothing in the client UI tells you this process is occuring, you have to check the UI on the server to see if its transcoding. And yup, I found out it was.
Another issue is that the list of content types Plex supports on Xbox One is really limited. If you have 7.1 DTS or Dolby HD audio, your entire video file gets transcoded down to 1080p. If you have subtitles embedded in your file as a stream, that gets transcoded down to 1080p.
The upshot of this is that I will never use the Plex app on any device. Transcoding is completely annoying to me. If I wanted to watch a 1080p version of my bluray rips, I'd encode them that way in the first place. Unfortunately it seems impossible to disable transcoding in Plex, the best you can do is watch from a DLNA device that Plex can't identify. Luckily, the Xbox One Media Player app does exactly this.
The solution was to re-encode the entire movie into a very simple format that Plex was happy to serve over DLNA to the Media Player app.
I used the command below:
ffmpeg -i <INPUT_FILE> -map 0:0 -map 0:1 -c:v hevc_nvenc -profile:v main -pix_fmt:v yuv420p -preset:v slow -rc:v vbr_hq -qmin 16 -qmax 20 -c:a:0 aac -b:a 384k -ac 2 -af "pan=stereo|FL=FC+0.30*FL+0.30*BL+0.30*SL|FR=FC+0.30*FR+0.30*BR+0.30*SR" -movflags +faststart <OUTPUT_FILE>
Let me walk you through each of these.
ffmpeg is the software we're going to use to re-encode this file.
-map 0:0 -map 0:1 tells ffmpeg to only select the video and first audio track from the file. This drops all of the incompatible subtitles and non-English language audio I'm not interested in.
-c:v hevc_nvenc tells ffmpeg to use the NVIDIA hardware HEVC encoder built into my graphics card (a GTX 1060). This speeds up the encode exponentially. You can use a software encoder if you want, but it will take a lot longer.
-profile:v main specifies that I want to use a standard HEVC Main profile. This profile only supports 8bit files, so it is fine for me. If you need 10bit for HDR, use main10 as your argument here.
-pix_fmt:v yuv420p sets the chroma sampling to 4:2:0. I chose to do this because I'm unsure if my TV supports 4:2:2 and there is no way it supports 4:4:4. The output video still looks great on my TV.
-preset:v slow -rc:v vbr_hq -qmin 16 -qmax 20 sets the encode preset to slow, so I get a small file at the end, and the other commands set my bitrate to be variable, with a minimum quality of 16 and a maximum of 20. I found this results in output that is indistinguishable from BluRay on my TV. You do not need to set qmin lower than 16, beyond that you can gain no visual improvement in quality.
The next section are options for the audio track.
-c:a:0 aac -b:a 384k selects AAC audio with a bitrate much higher than I need. This preserves a good chunk of the original quality while removing the potentially incompatible DTS HD audio codec.
-ac 2 -af "pan=stereo|FL=FC+0.30*FL+0.30*BL+0.30*SL|FR=FC+0.30*FR+0.30*BR+0.30*SR" is a little more optional. This specifies that I want two channels of output audio. I did this becuase I do not have a surround sound system yet, and I wanted to guarantee comaptibilty with my Xbox. Because I have a source file with 7.1 surround sound, I use an audio filter to downmix all of those seperate channels into stereo. If you want to do this and you have 5.1 surround in your source file, use -af "pan=stereo|FL=FC+0.30*FL+0.30*BL|FR=FC+0.30*FR+0.30*BR" instead. If you want to preserve your surround then drop everything from here, and increase the bitrate in the section above.
Finally, -movflags +faststart tells ffmpeg to put the moov atom at the start of the output file. This is something used for syncing the audio and video, so by putting it at the start the file can begin playing without you having to wait for the whole thing to download. In other words, this makes it more likely that streaming the file from a media server will work.
The result is a very simple file with nothing but a 4K HEVC video stream and a compressed stereo AAC audio track. Now the film plays just fine over DLNA in my Xbox Media Player app.