|
|
|
Categories
Reference & Information
|
Video Storage Elements of VIDEO STORAGE by Brian Adams
Your video storage and editing array needs to be simple! The old KISS principle, "Keep It Simple Stupid", truly applies here. If you add complexity you also add avenues of failure. Instead of using complex methods, your video data files should be in nice big volumes with a simple and automatic backup. The basic elements of a video storage system are PERFORMANCE (speed), VOLUME (storage size) and SECURITY (backup). Our goal here will be to cut to the chase, point out some simple solutions that work. It is our belief that the simpler the better, so we will cut out a lot of the geek speak and focus on a solution that you can put in place easily and quickly. The simple steps to conjure excellent video storage are:
PERFORMANCE The sheer size and I⁄O (Data throughput speed) requirements of todays HD video generally necessitate a RAID array. The means of achieving the needed speed is by RAIDing (Redundant Array of Independent Disks) several individual drives together into a single volume that combines the speed and size of individual members. The array must be assembled with the number of individual drives required to attain the needed performance for the format of video desired. More drives = more speed. For every format of video there is a performance threshold or minimum performance that must be maintained to avoid problems when editing the footage. SD has very low performance requirements and can usually be accomplished with just a pair of drives in a simple RAID 0. HD, especially editing uncompressed has much higher performance requirements. For example, 1080i uncompressed will require about 250 MB⁄s minimum and 2K starts at about 300 MB⁄s. Fortunately with todays high performance drives it requires less hardware to accomplish those speeds. To determine the performance requirements of your format refer to the table below. Choose your preferred format to determine the I⁄O speed requirements. Your editing array must have some headroom to avoid dropping frames. Let’s take the most common uncompressed format, 1080i 10-bit at 30 frames per second. This format requires slightly less than 250 MB⁄s. If you add on about 50 MB⁄s (20%) to avoid dropping frames you get a requirement of 300 MB⁄s for your editing volume. Simple enough? Read on.
NOTE: The data rates and storage volumes listed in the table above are approximate due to the many variables involved.
One other limitation for hard drives is that they shouldn’t be loaded completely full. When storing any data on hard drives a certain amount of directory index is written to the drives. If the drives become too full that index runs the risk of being overwritten. This can cause data corruption and damage or destroy your video. We recommend that you always keep the stored video data less than 80% of the total capacity of the array. This will leave ample room for that index to reside on the drive without damage. Individual drive performance varies between the manufacturers so we’ll pick our favorite, the 7200.11 generation Seagate, available in 1TB and the new 1.5TB drive. The 1 and 1.5TB are capable of the 124 MB⁄s performance range and maintain almost 75 MB⁄s at 80% or capacity. Now as of February 2009 we’re shipping the Seagate 7200.12 model 500 and 750 GB drives that are capable of nearly 160 MB⁄s. What is the benefit of using fast drives you ask? The faster the drives the fewer drives you need to attain the required speed. These new Seagate drives provide the best performance and storage volume for the dollar spent. Now to the nitty gritty. How many drives do you need for the your editing array to stream, edit and write video? Go back to that 300 MB⁄s we came up with earlier for 1080i. Considering drives at their loaded data rate of 120 MB⁄s you would need a minimum of 3 drives to achieve the required speed without dropped frames. Due to the number of drives needed to reach the I⁄O speed requirements, Port Multiplication is the leading hardware type for video editors. This great technology allows a storage enclosure to attach up to 5 drives with a single data cable, allowing up to 20 drives on a low cost 4 port PM capable Host Card. By combining multiple drives nearly any speed can be reached. The drives may be RAIDed across multiple enclosures allowing for 8-10 drive arrays while leaving room on the card for backup drives to be mounted. Port Multiplication is the most expandable and least expensive storage solution currently on the market. The Burly 6 Bay Dual PM Enclosure is an excellent way to extend your storage dollar. This little enclosure uses dual PM Boards to double the performance providing a maintained 400 MB⁄s with only 6 drives. With any hardware type, Port Multiplication does have it’s limitations. The board that combines the data signal of the drives within the chassis is limited to transferring about 220 MB⁄s. This means that a 4 or 5 Bay enclosure will provide approximately 220 MB⁄s where a 6 or 8 Bay Enclosure with two PM Boards will provide over 400 MB⁄s. With todays high performance drives it only takes three drives to flood a Port Multiplier Board with data. Adding more than 3 drives on a PM Board will only provide a larger storage volume, not more speed. To understand where the storage RAID array get it’s speed there are a couple of realities that must be understood. The performance requirements dictate the number of drives that must be used. Each drive included in a RAID be it either a RAID 0 or a RAID 5 adds it’s performance value to the RAID. Today’s retail desktop drives are capable of 100 - 160 MB⁄s when empty. If you RAID together multiple drives, the number of drives included dictates the maximum performance available. Sometimes there are other limits to performance such as bandwidth or computer limitations.
STORAGE VOLUME The volume of storage required is determined by the format of video multiplied by the hours of video that must be stored (MB⁄s X 3600 sec). This includes the RAW footage as well as the post processed final product, anything you can’t afford to reshoot. With the huge 1500 GB drives available now this is easier and less expensive than ever. When you choose what drives to RAID together to attain the performance required for your project you will also have to choose the drive capacity. By considering the volume losses and your project’s space requirements you should be able to decide what size drive is best for your needs. Keep in mind that you never want to fill a hard drive to it’s capacity, always stop at about 80% when storing video. If the RAID you plan on using is a RAID 0 all but about 7% is available for storage. That 7% is lost to the formatting process. If you choose RAID 5 you will lose the volume of one of the drives as well as that 7%. For example if you build an 8 drive RAID 5 using 1TB drives (8TB total) you will lose 1TB for the parity and the 7% format loss leaving you with approximately 5.2 TB of usable space at 80% of capacity.
SECURITY Contrary to popular belief, a RAID 5 does not offer much more data security over a simple RAID 0. What it does offer is failure tolerance, a RAID 5 can fail a single drive and maintain it’s data where a RAID 0 will lose all data if a member is lost. Parity RAID such as a RAID 5 or RAID 6 is a server technology designed primarily to maintain database uptime. It does this by storing parity data across all of the drives along with your video data, if a drive fails the RAID Controller can calculate what was on the missing drive. This makes for a very complicated algorithm which requires the magic of an advanced RAID Controller. The real magic comes into play when something goes wrong. Parity RAIDs are intolerant of data corruption, a little corruption introduced can make the RAID unreadable. Another issue is the complexity of the RAID itself. Parity RAIDs are complicated to maintain especially when a problem arises. Should a drive failure occur and the operator accidently remove the wrong drive the controller sees the missing drive as a failed drive, meaning that the array has lost two members and is now irreparably lost. We’ve seen this happen. Parity RAIDs have their own avenues of failure and for this reason can’t be used without a backup copy of your data. Even a RAID 5 can fail and lose your valuable, irreplaceable data. The best plan that exists for protecting what you can’t afford to lose is to have multiple copies. If you manage a backup copy of your data correctly it can protect you from events that the most robust RAID 5 can’t. If a failure or corruption should occur, the backup copy can be used to recover or keep working. The backup should be automatically updated by a software application which keeps the backup version up to date with the primary. The one caveat to this method is should you experience a database problem or corruption you must stop the automated backup from occurring until the problem is rectified, otherwise that corruption will be carried over to the backup rendering it useless. Anytime an Operating System, Security Update or new software is installed you should disable your backup so that in the event of a problem it will be in pristine condition and available.
HARDWARE Most of our customers are processing their video on RAID 0 Port Multiplier systems with a second array for backup. This requires a PM capable Host Card installed inside the computer to provide external eSATA ports. Each of these ports is capable of up to 5 drives when using the Burly Port Multiplier Enclosures allowing enough drives to be mounted for the performance requirements of today’s video formats. These RAID 0 systems are the most cost effective and expandable storage available. One of our best sellers is the Burly Dual PM which uses a pair of Port Multiplier Boards and 6 drives in a small chassis with excellent cooling to provide 400 MB⁄s performance. That’s fast enough to edit 2K or you can stripe 3 drives and back up to the other three in the same enclosure for 200 MB⁄s+ performance. Depends on your performance needs. For a true Hardware Based RAID 5 check out the Burly VR RAID Systems. This Enclosure comes with everything you will need already formatted into a RAID to your specifications. RAID 5 and RAID 6 are becoming popular for video and provide failure tolerance although a backup is always required. These systems have the dedicated CPU and RAM on a top level Areca card to manage the RAID leaving your computer’s resources untaxed. The high performance CPU provides excellent performance in the 700 MB⁄s range with 8 drives. Here we have to split you up into two groups, PC’s and Macs. The Mac is by and far the most common video editing platform but the PC users have more choices in hardware to make things happen. If you’ve chosen to edit on a RAID 0 you’re in good company. For either the Mac or the PC I would suggest installing a Port Multiplier capable Host Card into your computer and attaching a Burly Port Multiplier Enclosure or two for your editing array. These can be striped together to combine their speed. The host card chosen will depend on what computer you’re on. If you need help choosing the correct card check out our Host Card Guide or contact us directly. We’re always happy to help. For a Mac based editor planning a RAID 5 or RAID 6 solution there are only a couple of options. The new Areca cards we offer will support these RAID structures using Burly Mini-SAS Enclosures or you may choose the Burly VR as a ready made system which will be pre-tested and ready for you to build your RAID. These systems are also available for use with PC’s as well as our high performance 3ware RAID Controller products. Keep in mind that even with the best Hardware Systems you’ll still have to plan for a backup of any data you can’t afford to lose.
BACKUP ROUTINE No matter what solution you decide on for your editing array you must keep that data backed up. Working on your only copy of data is a great recipe for disaster and there’s no excuse for it. One of the simplest, safest and most cost effective backup solution we have found is to simply have a working volume that’s automatically backed up to a backup volume on a time schedule. Should the working copy become corrupted or damaged the backup copy hasn’t been damaged and can be used for recovery. The backup can be scheduled to run as often as needed to provide the level of protection preferred. In most cases a nightly backup is enough. Keep in mind that if the backup is allowed to run after any damage has occurred that damage will be transferred to the backup rendering it unusable for recovery.
BACKUP APPLICATIONS Most applications used for PCs actually make a disk image of your data drive. Acronis True Image and Genie Backup Manager are two of the best that we've worked with. There are numerous backup software applications available for PCs highly recommended among them are Norton Ghost and Retrospect. Do some research and find one that works the way you want it to. Most offer a free trial version to test drive. The application we use here on Macs is SynchronizePro X. We have been using this one for many years and have recommended it to countless satisfied customers. SyncProX is capable of backing up literally any file, folder, group of files or complete hard drive - to anywhere else. The backup can reside internally, externally or off your computer over the network. We use combinations of both, each computer backing itself up to drive(s) on that computer and then essential data backed up to a backup server as well. There are several other quality applications out there for Macs like ChronoSync, Backup and DejaVu as well as others. Our personal experience is mostly with SyncProX and we have less capability to comment on the use of those other applications. Many of them have a large following of happy users. Pick the one with a user interface you like. Most backup software has downloadable trial versions so you can check ’em out for free. We use the software applications to make backups of our data drives to backup drives. We also backup our OS drive on a weekly basis so if something gets corrupted we can be right back up and running with a reboot (perfect when that OS upgrade breaks something important... but nah, that never happens, right?). And we use the application to make daily backup of our RAID volumes to identical size RAID volumes.
CONCLUSION Figure out how much speed and capacity you require. Select an enclosure and the correct host card. Installation is very simple, requiring a philips screwdriver for the host card and for any drives requiring installation in trays. It is recommended that you zero drives before RAIDing them which burns the drive in and allows it to map the sectors on the platter. Stick with retail versions drives and avoid OEM versions, you’ll be glad you did. Check out our Initial Drive Preparation Guide.
If you have specific questions don’t hesitate to call (208) 983-2555 or email us. We’re always happy to help you configure a system that will best suite your needs.
Last updated March 2009 - Brian Adams
|
When considering performance you must consider the physical capabilities and limitations of the drives themselves. Drives use a spinning group of "Platters" to store your data. Unlike a CD, hard drives write from the outside in. This means that the first data stored on the drive is the fastest. The more data stored on a drive the slower that data I⁄O will be due to the shorter circumference per revolution. As the data tracks move closer to the inside of the platter, less data can be accessed per a given time. The throughput speed of a full drive is approximately 60% of it’s empty speed. Why does this matter for video? If you don’t plan for this slowdown, the array that worked great when you first set it up, will drop frames as you begin working on the inner tracks of the drive.|
207 Mill Creek Lane Grangeville, Idaho 83530 United States Telephone: 1+208-983-2555 Fax: © 2010 BurlyStorage.com. All rights reserved worldwide.
|
 
|
