31 January 2019

Setting Up An Automated Media Encoding Server

Using Unraid, create a server that will automatically encode or transcode video using watch folders with Handbrake.

Spxlabs

Today's Guest Blog is by Stefano Partida of SPX Labs who has been using Unraid for the past 3 years for video editing and media storage.

Media Encoding Server

This all started because Final Cut Pro X does not (currently) support H.265 videos. So, thanks to Apple, we have now created a server that will automatically encode or transcode video using watch folders with Handbrake. The most important thing about this server, is having all work done automatically on its own. This way I can drop the videos into the watch folder and pop off for beer while Handbrake does all the work. Hmmmm, where to start…. prepare for word walls!

The Setup

Well, I guess it makes sense to talk about the big picture setup first. I feel like I should just diagram this….
 

Untitleddiagram1

As seen above, this is basically the workflow. I have to manually pull all video from the camera into the watch folder that is on my Unraid server. Handbrake see’s that new files have been placed into the folder and begins transcoding them into a 4K 60FPS MP4 file. When Handbrake is done, it puts the finished video into the export folder, from there I can easily grab those files and import than into Final Cut Pro X at my leisure. Then of course once all editing is done, I will just upload that to YouTube or where ever on the internet. So that is the 1000 Foot level overview. Let’s go deeper!

The Server

So I think it is most important to start with the server because this is truly the backbone of this entire build. Without the server, none of this would be possible; well it would but not the way that I have laid it out. Now, I have written in decent detail about what my Unraid server looks like and what it is comprised of here. However, in the spirit of keeping as much as possible in a single convenient location, I’m going to give you a list of all the components of this thing.

Total Cost: approximately $2,000

It’s safe to assume I virtually spared no expense on this thing. Now, I’m not saying you need a badass server like mine (although it helps) but you may want something…. anything that can sit separately from your workstation and can chug away on video transcodes/encodes while you work. Even if you don’t suffer from my issue, you can still take 4k or 8k footage and transcode it to H.265 automatically to get the file size down.

You can buy a used pre-built server for cheap off of eBay or from companies like savemyserver.com / orangecomputers.com / servermonkey.com. If you’re afraid to buy from them, then be sure to check out Dells website for slick deals on used equipment. It does not have to be a rack, you can also buy towers!

Working with a strict budget, try finding a Dell T30, as you can pick one of these up for under $400 on sale. Want a bit more power? Well, Dell has you covered. I highly recommend looking into the used market or buying an AMD system with as many cores as possible. Still scared or worried? Contact me and I’ll try my best to help you out.

What’s nice about having tons of cores working on a encode is the speed and quality of which you can throw at the video. Transcoding can take a very long time depending on how you have configured Handbrake (more on this later). For me, I have the transcode set to as close as possible to lossless. This type of work requires a great deal of system resources (mostly time), look at how much both CPU’s are working on this 1 minute 12 second 4k video:

Cpuspegged

That is 16 Cores and 32 Threads at a whopping 100% usage. Absolutely incredible! This setup will churn through tons of video footage while I work on other things.

Operating System

Anyway, if it wasn’t obvious by now, the host operating system for the server is Unraid, why? Well, many reasons. Here is a list of a few:

  • Unraid offers dual parity
  • Easy storage expansion
  • SSD cache (for quick transfer speeds)
  • WebUI accessible from anywhere on your LAN (local network).
  • Docker, setting up many containers like Handbrake
  • Great community support and vast knowledge base
  • Fairly inexpensive license fee (perpetual)
  • Very low maintenance required

More on why: In my opinion, Unraid is incredibly simple to use. They offer a USB creator for Windows/Mac/Linux that will literally make you a bootable operating system in a matter of a few minutes. Even Windows 10 Media Creation Tool can take more than 10 minutes and more often than not, it doesn’t even successfully create your bootable USB on the first go.

Arguably, the best thing about Unraid is Docker. Docker allows you install containers that have direct access to hardware, without the need for installing the applications inside a VM and the container has the ability to be “outside” of the hosting operating system. Very cool technology. Unraid’s Webui allows you to easily install containers for just about anything. In this case, we can access the Handbrake container either through direct IP Address or through the Unraid webUI. Although, once Handbrake is setup, you pretty much never need to access it again.

Since Unraid is a NAS (Network Area Storage), it’s important to us that we have resilient storage. The use of Parity in Unraid assures us that we can lose 1 disk and still have access to any data that was on that disk. I’d highly recommend reading about Parity from Unraid themselves. Now, because of the amount of math that needs to be done when using any storage array with parity, Unraid has given us the ability to have a cache. Typically, you will always want your fastest drives to be your cache disks. This way you can transfer data to your NAS very quickly and then Unraid will store it into the array at a later time on your behalf. This allows us to hit 1 gigabyte speeds. Yes, 1 gigabyte(!) not bits, when we transfer data to the NAS. I encourage you to see a video I made about 10Gb/s networking to Unraid:

Handbrake

You may be asking yourself, why Handbrake instead of Adobe Media Encoder or Apple’s Compressor? Well, reason 1: Handbrake is free, for life. Reason 2 is it works on PC/Mac/Linux, so you have the to ability to cater it to your needs. Let’s just be honest with ourselves, Mac’s are expensive and it is pretty easy to build or buy computer hardware to make very powerful systems for cheap. I bring this up because, if you are in the middle of editing a video, you don’t want to be using Compressor while you work. You also don’t want to have to wait for Compressor to finish encoding a ton of footage before you work. So, because of that, using Handbrake on a separate system can really save you some time and effort. Now sure, my server cost $1500 bucks, but it does more than just encoding media (more on this later). If you are looking for a cheap easy start, it is 100% possible for you to find cheap hardware and have a dedicated encoding machine.

But, I digress, back to Handbrake: As far as I know, Handbrake has the ability to convert any file format to any other file format. Also, it gets updated pretty regularly so if something new pops up, you can safely expect Handbrake to support it. Now, if you have done any research on watch folders, information seems to always point to Adobe Media Converter, which is fine if you have the money to pony up month to month. Unfortunately, I am an amateur so making a monthly investment in software isn’t something I want or need to do. Especially since I already have all the hardware. The other bit of information you will see are scripts that you can create yourself or use from others to use with Handbrake so it has a “watch” folder and automatically encodes video. While that is a possibility, the information can be very daunting for some people, including me. This is where, with the help of Unraid, we can use the power of Linux to help us automatically convert and encode video using Handbrake and watch folders.

It’s incredibly simple to do. So simple in fact, I made a video about using Handbrake to encode HEVC into H264 MP4 files for Final Cut Pro. There is also a bit of a tutorial setting up Handbrake with Unraid and configuring a 4K 60FPS preset so all future automatic transcodes will follow that same format. Check it out!
 

Why

Well, for me personally, I already had this hardware “laying” around. The server has gone from being a Media Server to the “Ultimate” do everything server. Every day that goes by, it does more and more. I’d really recommend this read about how the server has evolved over the course of a couple of years. This could really help you. Anyway, back to why: The first and foremost reason why is, I was forced to do something because Final Cut Pro X still doesn’t support HEVC / H265. This is why I initially started looking around for ways to re-encode my video footage to H264, something that Final Cut Pro X does support. Now, because I don’t own Compressor and was unwilling to pay a monthly fee to use Adobe Media Converter, I didn’t really have a lot of options available. Initially, I started out by having Quicktime export all video into a different format. Believe it or not, Quicktime does allow you to watch and export HEVC / H265 videos. However, while the video was exporting in Quicktime the Macbook Pro would under preform and was just not very useful until it was done. Also, this was a very manual process so it took a lot of time to export each individual video.

Initially, I was very close to getting some scripts to work on an Ubuntu Server for me on a totally different server. However, this was a bit cumbersome and not being the brightest bulb, I was unwilling to put in the effort to really decrypt how to do it. Now, I understand for MANY people this can be done very easily but I am not you nor that person. I’ve used Handbrake in the past and the software is absolutely amazing, however, this too would have been a manual process if I didn’t get Watch folders setup through using scripts. I almost dropped this project entirely until I saw a video showing off different Applications in Unraid. I saw the Handbrake icon and immediately Googled Handbrake and Unraid. While there were a couple of resources, those resources weren’t really trying to achieve what I wanted to accomplish. That’s how I got to where I am today.

Why would I recommend someone new to buy a used server/tower and Unraid? Well aside from the reasons I mentioned way earlier (about parity, containers, easy webUI etc), some other great reasons are listed below:

  1. You can setup Unraid to be your personal Time Machine for your Apple computers
  2. Unraid can also be used to be your Media Server using Plex
  3. Multiple people can access the server from a PC/Mac/Linux device, they can literally choose their favorite OS and still take advantage of what Unraid can offer
  4. All of the Applications (containers) in Unraid are free! The sky is really the limit here.
  5. You can setup a personal VPN within unraid (using OpenVPN) so you can reach your server anywhere in the world. About to board that plane? Upload that video to your server so it can start encoding videos before you get home!
  6. You can mix and match hard drives in the storage array. This is nice because you can just keep adding in newer bigger hard drives whenever you need.
  7. You can setup a Cloud Service so any files or pictures can be uploaded to it from anywhere (different than a VPN)
  8. You can have 7 Gamers and 2 CPU’s run with Unraid!!
  9. Also, Unraid uses Red Hat’s KVM. This is important because mega million dollar companies use Red Hat ( I don’t know if this actually counts as a reason…)
  10. It’s just easy to use. How do I know? Because if you have seen any content from me, you will find yourself asking, how the hell has this guy made it this far in life? Wait, he worked for NASA?!?!!!??!!!! (The NASA part isn’t true, but a man can dream!)

The opinions, product recommendations, procedures and views expressed here are those of the author and do not necessarily reflect those of Lime Technology, Inc.