Jordan Westhoff

Jordan Westhoff's Blog


Leave a comment

SHOOTOUT! Data Usage Statistics!

Hey guys!

Over the weekend I spent a good deal of time looking at just how intensive a full RAW data workflow can be. I also wanted to compare the burden of 4K RAW vs Arri’s 2.8K RAW via a S.Two recording device and see which required the most data overhead to work with. This allows us to simply look at how much drive space is required and discount the physical CPU usage of the project since pretty much all of my machines were running at almost full bore whenever renders were required.

While storage is not really a problem for a lot of industry professionals, it can be quite the burden for independents or students. Not every student has several terabytes (a terabyte is 1,024 gigabytes) of unused, high speed storage. A lot of people wonder if they van get away with slower, basic desktop drives for data of this proportion but it really comes down to how long you want to wait. Slow drives serve information, well, slowly. Waiting for 300GB of renders to load can take ages and when deadlines are at stake, it really isn’t viable.

Below, I’ve compiled a good deal of raw statistics from our recent shootout project. Since I was in charge of managing the data, image processing and running the servers we worked off of, I have the entirety of the raw footage as well as a significant portion of the renders. This accounts for tons and tons of space, enough space in fact that I thought comprehensive statistics might be helpful to visualize where all of that information is going.

There are a couple foreword things to note though, before we get started. In some of the statistics, I pitted the total usage by camera which encompasses all of the information used, start to finish on each camera platform. In one or two other statistics, I broke up the information to reflect intermediate stages. For the Arri D-21, this required converting raw S.Two DPX files to .ARI files and then exporting them again from ARRI RAW Converter to DPX or .TIFF file sequences to color grade and then make a final export of. For the Sony there was simply taking the camera files from the onboard SD card and then grading and re-exporting. In 4K, however, there was significantly more to do. Dumping the card gave a nice, proprietary, MXF wrapper with all of the files which had to be opened with Sony RAW Viewer in order to convert them to 16-bit DPX files. These could then be graded and exported again to a DPX or TIFF sequence to be imported for analysis and editing. Each of these reflect storage as you can imagine, and it presents quite a trend in the statistics.

Screenshot

This accounts for all of the ‘mission critical’ information stored for the shootout.

Here, we can see just how much data there was overall. In total, the final aggregate size of all resources exceeded 1.6TB! This included all footage, start to finish, ARRI, Sony and Sony 4K as well as renders, CC passes, graphics for our final video and any other data in between. Keep in mind that for the actual camera footage (which comprised a significant portion of the overall data used, but more on that later) totaled only about 10 minutes per camera (and less for the Sony 4K). This is because most of the shots used in the shootout were of charts or color scenes – the longest scenes were barely over 50 seconds apiece. Therefore, shooting an entire film on any of these platforms would consume an incredible amount of data. Broke up above are four different categories and each are perhaps a bit vague so I’ll take a moment to explain them.

The first, and the largest is the Footage Archive. This is an aggregate gathering of just the base footage captured from each camera. This also incorporates some intermediate files in the case of the ARI – essentially all of the footage classified here was footage that was ready to go into editing minus any major color correction. The Shootout Archive contains all of the intermediates of the pick scenes and the color corrected scenes. This means that any footage that was observed and chosen to be good enough for analysis went on to continue the chain of picture processing. The files contained in this directory are renders from the S.Two and then processed in ARRI’s ARC as well as the Sony HD and 4K clips that were chosen – those also underwent their respective processing steps as well. Shootout MAIN is the working directory for all of the analysis, as well as the video production portion of the project. Here are all of the final renders, color correction finals, stock footage, B-roll, preliminary video screening renders and narration as well as all of the graphics that our team generated as well. Finally, there is a Web Access Point directory. This was a separate directory created on a network server in order to provide each member of the team with fast, reliable intermediary storage for their own assets in production. These could be screen captures, editing files, project files, you name it. This is the working miscellaneous that helped make the workflow so efficient – each member had a fast directory to work from and then contribute to the final project being assembled in real time.

Each day of shooting generated different amounts of storage requirement based on scene.

Each day of shooting generated different amounts of storage requirement based on scene.

Since the shootout was spread over three (technically four, when you look at 4K) days, it was useful to look at how usage varied by day. Some of the graph information was cut off but the four largest portions were indicative of the longest shots. Day 1 files came close to taking the lead in storage but our Day 3 files took the lead with 19.6% of total data usage – these stats merely incorporate the files coming from the ARRI and the Sony in HD video mode. The third largest, at 17% was from our fourth day of shooting and this comprises all of the Sony 4K raw shoot files. Each of the much smaller portions is broken up by shot – some scenes took many shots and some took far less.

final_data

This shows a better look of how production workflow can impact your data needs for each project.

Here, this is a final, final look at how much information from each step of production comprises the total. This specific figure ties directly into the final, cleaned up and organized storage stats of the shootout in its entirety. Of the approximate 1.6TB required, the most costly stage of production was generating all of the intermediate files. This was especially true of the 4K tests which equaled almost half of this information despite shooting for only about %20 as long as the ARRI and Sony HD tests. Both RAW tests required multiple intermediate steps which chewed through tons of space because of each’s respective resolution. We chose to work with DPX and TIFF’s since those are lossless formats and overall exhibited the best quality.

All in all, shooting RAW is a very exhausting process, both from a processing and storage perspective. Your storage needs will be dependent on the camera and the codec/format you choose to edit in but it’s always safe to budget one to two terabytes for shooting a short and always, always remember to BACKUP your information! All of the statistics here leave out the backups that were set in place to safeguard our information. At any one point, our information was backed up in two additional places – one in a hardware RAID attached to a workstation on another end of campus and a full minute-to-minute backup stored on a NAS. This NAS also pulled all of the web assets from each member in order to keep their assets online and safe at the same time. Feel free to contact me if you have questions as well!

In the future I’ll be making a post dedicated to the labyrinth of storage and why different types are better than others, as well as a look into what I’m using to manage all of this information! Thanks for reading!

Advertisements


Leave a comment

FreeNAS, woo!

Hello all,

The last couple weeks have been busy but I’ve made significant progress on a lot of the projects I posted about last and figured I’d share some of the gory details from the tech spehere.

First off, the FreeNAS server is finally up and running. FreeNAS is a freely distributed version of the FreeBSD operation built specifically for engineering a personal or enterprise level Network Attached Storage server (NAS). Since I have a solid array of computers and Dropbox is very limiting, I figured that with the insane bandwidth RIT offers student, it would be an infinitely valuable addition to my computing fleet.

FreeNAS Boot Screen

The welcome screen of FreeBSD based FreeNAS

I decided to repurpose a sever that I was running Windows Server 2012 Datacenter Edition on. Since FreeNAS is best run using the ZFS file system which requires 64 bit, I decided to utilize one of the blade servers from my rack – a Dell CS24-SC Cloud Server header. Dual Intel Xeon quad core processors will handle all of the work, and it has enough RAM (16 gigabytes) to handle all of the serving very capably. Installed are 6 Terabytes of information (2x2tb drives and 2x1tb drives) which should be plenty of storage to host all of my common files.

Since I am using all three major operating systems to access these files (OS X, Windows and Linux) I decided to also set up a variety of shares from within the freeNAS setup. This includes a Samba/CIFS share for Windows machines to access from (one of my personal laptops and two of my other servers), an AFP (Apple File Protocol) share for my personal workstation MacBook Pro and all of the school lab machines and a couple office machines, as well as a NFS share for Linux access.

On top of this, I decided to enable SSH and FTP/SFTP access so access from pretty much any device is guaranteed! A week or so ago I migrated all of my common data over to the server once it was running and I tested the validity of the drives. Let me tell you, moving 5.5 terabytes of information takes a loooooong time! Most of the slowdown was the result of moving files over LAN rather than something like FireWire, Thunderbolt or USB 3.0, however it still clocked along at a very good pace since my apartment is very well connected – RIT gives each of us several gigabit (10/100/1000) access jacks in each apartment and the network is full fiber! This makes accessing the files remotely very, very handy since the speeds support even the most intensive tasks (even streaming Blu-Ray content from computer to computer on campus!).

My final goal for the project is to open up a drive on the server to use for various Apple Time Machine backups, I’ll keep you all posted, thanks for stopping by!


Leave a comment

FreeNAS Adventure!

After a full day of work with RIT and a client of mine yesterday, I sat down in the evening and began work on one of the projects I’ve been planning for a while. 

Contrary to my initial goals, I decided to convert one of my Dell CS rack servers to a FreeNAS server instead of using the NZXT case that I had. This is because once racked, the unit has far better network connections (quad gigabit NIC!) and around the clock uptime since they are hosted from a datacenter and live under redundant power and power supply backups!

FreeNAS is an awesome software distribution I’ve used pretty frequently in the past, but with the update to Version 8, a lot of new features are available making it one of the most versatile systems to date. 

Once burned to a USB drive in a live ISO form, I was able to smoothly progress through the install phase! I chose to ran the new version with the ZFS+ file system, since the Dell server unit I have utilizes dual quad core Xeon’s and has 16gb of RAM. 

After the install, a beautiful interface was presented. Pretty impressive is that the system idle’s at %99.96 idle CPU load. Combined with 8tb of storage and I’d say we have a solid serving platform, accessible from Apple OS X, Windows and UNIX!

Image

 

 


1 Comment

Upgrades, Upgrades, Upgrades

Hey All!

With my return to Rochester, New York to spend a couple weeks working before the next academic semester, I’ll be posting a lot of neat content relevant to the assortment of projects I have planned! Here’s a quick breakdown!

Moving some servers around! I’ll be converting a Windows 7 server to a CentOS server to host this website on! I’ve been using WordPress for a significant period of time and it’s been great! The problem is WordPress’s business plan and pay per add on web features when you host through their web service. After some work with CentOS, I realized I have plenty of power to be hosting else where! I’ll still be running WordPress, the new version 3.8 is pretty snappy! But I want more power with hosting and SEO and back end programming, so hosting locally it is! As a note, if this site goes offline for a couple of days, don’t be alarmed!!

Image

Building another NAS! I have the parts, the case, (beautiful NZXT 210 Elite in sleek black) and the drives, but now it’s time for the build and setup! It’s going to be a real cool setup on the inside so I’ll be sure to post photos! I’ve been hosting somewhat of a local datacenter for my operations, clients and products but now it’s time to make a bit more of a contiguous storage pool, especially given RIT’s internet speeds here! The power supply is coming in in two days and then building will go forward pretty quickly! This should comfortably keep my storage ceiling around 20tb, even with redundancy and network shares!

Image

Image

On top of this, I will also be setting up my final server as a general use server, with a file system and user access panel. Currently experimenting with Windows Server 2012 datacenter Edition, but also looking into Ubuntu Server and the possibility of adding in another CentOS machine! With all of these installs, I’ve tapped out all of our networking opportunities so I’ll be installing a mid sized rack switch and firewalls. Time to take this setup to a full blown cluster now that I have the capabilities for web serving, file serving and administration and distributed rendering and computational engines. Hopefully the setup starts to look a bit more professional as well, as compared to the humble beginnings of the project, shown below!

Image

The humble beginnings of my growing server setup.