Creating a better render farm monitor made for artists

In short

We use the AWS Thinkbox Deadline render farm manager at my school for handling all our heavy renders. Every student PC becomes part of the render farm whenever they’re not using it, which means the farm packs quite a punch during non-school hours. People often start a render job before they leave the school, which creates an issue: how do we check our render progress when we’re not at school? Before, checking our renders remotely went as follows:

1. Start up your own PC at home.

2. Connect to the school network via our school VPN.

3. Start a remote connection to your school PC.

4. Open deadline monitor on your school PC to see render progress.

A very time consuming process, yet crucial if you want to make sure your renders go smoothly. It has bothered me ever since I came to this school, so during the summer I came up with a new way of monitoring our render farm, something utilizing modern technologies and something that’s as easy as possible to use. Below I will be sharing my development journey, but let’s first have a look at the end result:

Students at my school can now monitor the render farm from anywhere, look at human readable crash report summaries by ChatGPT, and preview their renders within seconds. If you’re interested in the code, check out my GitHub! Also, if you want to see the monitor in action, check it out here.

The design

During the summer we don’t have access to our school PCs, which means I also couldn’t access the render farm for testing. So instead I followed this great React course on Udemy to up my web dev skills a bit.  I also got started working on a design for the website. I’m not a UI designer though, so I pretty much had no clue what I was doing. All I knew was that it had to be clean, simple, and not at all like Deadline’s own monitor, which looks like this:

I opened Adobe XD and started doing whatever. This is one of the first designs I came up with.

Clearly not a very flattering piece of work… Teachers at my school teach us to always look at references for VFX, so I took that advice and started looking at UI references too. I especially like the look and feel of modern apps, like the Revolut bank app. I took this as a source of inspiration and started working on some new designs after finding a nice color palette online. It took some time, but eventually I was happy with the final look:

Now all that was left to do was write the entire backend, write the entire frontend and voila.

The backend

The prequel: Crashy

I actually started working on an SMS system before I had the more obvious idea of a website. I thought it was a great idea to get texts from an AI bot called “Crashy”, which would update us on our render farm progress.

Crashy didn’t really survive, seeing as SMS APIs are quite expensive. Also, texting an AI bot about your render status is actually kinda weird. While working on it I did however figure out how to use the ChatGPT API, which is affordable enough to fit within our school’s budget. R.I.P. Crashy though.

Making the actual backend

Right, so it was time to write the actual backend. At first I wanted it to serve three main functions:

1. Send a list of render jobs information to the connected clients. I wanted three lists: Jobs that are currently rendering on the farm, jobs that have rendered on the farm in the last 48 hours, and jobs that are older than 48 hours but less than two weeks old. This seemed like a nice balance for how busy our farm was.

2. Send changed render farm data so the website updates while you’re looking at it. Just fetching the jobs would’ve been cool, but seeing the jobs update live on your screen is far cooler and much more useful.

3. Whenever a client looks at a specific job’s page, keep sending updates and also create a ChatGPT generated message based on the available render information.

Later during development a classmate suggested a fourth function:

4. Show image previews for frames that have been rendered on the farm.

Handling client connections would be the most important part of the system, so I needed a robust way to send data back and forth without losing information. This is where my favorite web technology comes in: websockets. Websockets are great. They are easy to use, can reliably send information and are supported in all modern browsers (sorry Nintendo 3DS). I’ve also used them a lot in the past for various projects, ranging from jackbox style games to live theater performances. I’ve never used them in Python though, as I typically wrote my websocket backends with NodeJS. But the Deadline API uses Python, so it was time to switch gears. It took a few tries to get a simple program going, but eventually I figured it out. I won’t be going in-depth on my code here, but if you’re interested you can have a look at my GitHub. I tried to keep it neat :)

To keep the backend speedy I decided to load all jobs data into memory, so connected clients quickly get the data they need. The Deadline Python API has a few function for fetching information, but it’s not events based so checking if data updates had to be done in my own code. After thinking about this for a while I decided that different data should update at different speeds. Active jobs should be refreshed quickly, once every three seconds. Recent jobs can go a bit slower, 30 seconds seemed like a good balance. Jobs older than 48 hour can update every hour, as that data doesn’t really change often. This way clients quickly get their fresh information, without overloading the Deadline API. Also, to minimize the amount of data to send, I decided to update the clients only with the changed information. This definitely lowered the amount of data sent, but also increased the amount of headaches I got while merging the data on the frontend.

I then implemented the ChatGPT integration, which just takes the crash reports from Deadline, tosses them through OpenAI’s magic and then streams the response to the connected client. After this I finished up the EXR to JPEG image previews, which was a bit cumbersome to figure out but luckily someone open sourced some conversion code which I slapped into my backend. This is a very rough summary of course, as creating this backend took well over a month of trial and error.

The frontend

I started developing the frontend after my backend was somewhat stable, and with that I mean not stable at all. I wish I took some more screenshots during this stage, as in the beginning everything was performing suboptimally. First I started writing the client websocket code, which I had done before so luckily I somewhat knew what I was doing. This was however my first project using React, and also my first project using a javascript framework in general. My god. I should’ve started using React a long time ago. I’m glad I took the time during the summer to follow that Udemy course because component based web development is certainly a much nicer experience than just doing whatever in vanilla javascript. Eventually data was funneling through and ending up in the right components, so now I had to do some styling. I’m also glad I took the time in the summer to make an actual decent design, because that massively sped this part up. I had quite a productive summer actually, now that I’m thinking about it, as I also picked up wet shaving, which has been great fun. Anyways. I have no clue how to write CSS. It wasn’t covered in my Udemy course and in the past I’ve barely gotten by with just copying some code here and there. That approach wasn’t going to work here, so I finally bit the bullet and avoided it altogether. Instead, I decided to use Tailwind CSS, a web technology that essentially allows you to write down what you want, and it then magically makes it work. This was also a great learning moment, as suddenly making a decent looking site wasn’t so scary anymore. I think the site now looks great, and the styling works well across a wide range of screen sizes. Also, yes, I did add a dark mode option. To finish up the frontend I added a bell icon users can click on to receive a desktop notification whenever their render finishes, which seemed like a nice finishing touch.

IT bureaucracy

The web app is done! It works great on the local network, but oh no, my school’s IT department won’t open ports to my webserver for “security reasons” or whatever. Something about my school being the IT hub for 8 different buildings, they all go down if there’s a virus, bla bla. I’m just kidding, from the start I knew IT wouldn’t allow any open ports, which meant I had to figure out some clever networking tricks. Here’s our issue: The Deadline Web Service runs on a VM on my school’s internal network. It can’t run anywhere else, as it needs to be connected to the render farm. The websocket backend that I wrote also has to run on the local network, as it has to connect to the Deadline Web Service. Clients however need to connect from the world wide web to the websocket backend, which is not possible if the ports to that server are not opened. So I came up with the following solution:

Yes, another websocket server and another websocket client! And don’t worry, I did tell my IT department about all this and luckily they agreed. External clients now connect to the proxy server, which sends data to the proxy client, which spoofs a connection and sends data to the websocket backend, which retrieves data from the Deadline Web Service. Some added complexity yes, but otherwise this whole thing wouldn’t work outside the local school network which kinda defeats the purpose of the whole thing. Also, the delay caused by the proxy is surprisingly small, so it all worked out in the end. The code for this is also open source on my GitHub.

Done!

Alright, two months, a bunch of code, even more deleted code, a whole load of bugfixes and a bunch of feedback from my fellow students later, it’s done! And most importantly, people are actually using it! Overall I learned a whole bunch about Deadline, web development and dealing with an IT department. Here are my key takeaways from this project:

- Make a design first, then start building your frontend. This way you know what you’re doing and you’re not wasting time figuring out two things at once.

- Component based web development is great!

- Be nice to your IT department if you want to get stuff done.

Are you a fellow pipeline TD student and happen to also use Deadline at your school? Installation instruction for this tool are on my GitHub, so maybe you could give it a try!