In many ways one’s workflow is an entirely personal consideration. Not everyone approaches thinking about every problem in the same way. Moreover, not everyone has the same limitations or considerations that they have to work with and under.

I’ve been shooting video for YouTube now for around 6 months, and finding a way to efficiently get content uploaded has been a perpetual thorn in my side. I’ve finally figured out a workflow that works for me, and wanted to share the process in case it helps others.

In my case, I have to underlying considerations for uploading video content. First, I don’t have a ton of bandwidth available. My internet connection is currently only an ADSL line with about 1 Mbps of upload bandwidth.

Secondly, I don’t consider YouTube to be an archival storage solution. I need to keep a copy of my content under my own care. However, I also don’t have 100s of TB of storage at my disposal, and massive storage solutions are an expensive proposition. Never mind the cost and difficulty of effectively backing them up.

With these two constraints, efficiency in terms of file size is the name of the game. And one of the most effective video compression codecs for that currently is H.265 or HEVC. However, HEVC comes with a major disadvantage, it’s efficiency in saving space comes at the cost of computational complexity and encoding times.

To put some perspective on the efficiency I’m seeing with HEVC; for equal or better image quality, I’m seeing files that are 50-60% smaller than I was getting with AVC (h.264). Some videos I had uploaded in AVC weighted in at 1.2 GB, but when processed into HEVC files were only around 500 MB.

With the computers I currently have available for processing video (a 4 core/8 thread 3.8 GHz Xeon server, and a 6-core/12-thread 4.5 GHz i7 workstation) and the processing settings I’m using, I get compression rates somewhere in the 8-10 FPS range. Or, put another way, I compress video at about 1/3 of real time speed. So a 30 minute clip takes 100% of my CPU time for an hour and a half.

Suffice to say, if I have to tie up my workstation doing this kind of work, I can’t move on to the next project while its rendering. There’s simply not enough CPU to go around.

My solution was to move the computationally expensive compression off my workstation and onto my server. To start with, it sits idle most of the time anyway. So having it take up the slack frees up my workstation to do other things, and is stopping the server from doing anything it otherwise would have.

Secondly, it frees up a complex and tedious step for me when it comes to uploading the videos themselves.

In the past, I’ve used my Mac laptop to upload videos after they’ve been compressed. It’s quite, and low power, and so it can churn away at many hour long uploads without making a lot of noise or consuming a lot of power. To keep it from sleeping I’ve a program called Amphetamine (which you can find in the mac app store) to keep the computer from sleeping, and consequently breaking the upload.

However, try as I might, I’ve repeatedly had problems with the uploads from my Mac through my browser. Either I mis-configured Amphetamine, or something else interrupted the process and the upload would fail partially in to the process. Potentially not only losing the several hours it took to upload, but also since there wasn’t anyway for the computer to notify of the upload’s success or failure even more time when I didn’t realize that it has stopped.

So what makes this whole process work?

The answer is a chunk of custom python code running on my Linux server. This the glue that watches folders, and takes the appropriate actions when files are created or moved into the folder.

The video processing is handled by the extremely flexible FFMPEG package, and YouTube uploads are handled by youtubeuploader available on Github.

That said, as I said in the video, I’ve got this working well enough that I’ve been using it to upload a handful of videos for myself. However, I’m also working on cleaning up the python code so that I could potentially release it for others to use. That said, I have no idea if there’s any interest in something like this either. So if you’re interested in potentially making this part of your workflow, please leave a comment to let me know. The more interest, the more willing I am to prioritize getting things wrapped up for release.