"Worthy but Dull" time on Backup and Archive

I seemed to have been hearing or reading a lot about media data storage in the blogs, podcasts and courses I have been doing in recent times.  I have written before about my semi-paranoid approach to data on the road.  So this time I am going to concentrate on what happens when you get that all home.  I am not a leading authority on any of these technologies, I just have been looking for a decent solution for a long time and I have some experience which translates from the workplace.  

So lets get things in perspective first.  I am an enthusiast and my work has sentimental but no financial aspect bar the hardware it sits on.  This is a domestic environment and I only have ADSL access to the internet (5Mbps down 500Kbps up on a good day).  I have a Mac Pro which has 2 drives, one for Media and Apps and the other for data including my editing projects, music and photographs.  These are both 2Tb drives.  I have another 2Tb on a Firewire attached G-Tech RAID drive (RAID 0).  In addition to this I have a Windows Home Server (its a long story) with about a net 3Tb.  Lastly, I have a 2Tb Time Capsule and this is the only device not situated in my office. 

I have also have numerous external drives ranging from bus powered firewire drives mainly for travel to bare drives and a USB “Toaster”.  I have both DVD and Blu-Ray optical drives capable of writing and I have tried some off-line options.  I have Dropbox and the soon-to-die MobileMe.  I have tried Carbonite and CrashPlan offline services.

So I am no media facility but my levels of storage are probably into the top end of prosumer levels.  My set-up is no paragon and I am constantly in a state of angst about what to do about it.

The basis to my approach is the same process I would go through as an IT Professional for more commercial systems.  I have included a little mindmap that I used to bring things together that you might find handy. You have to look at risk and cost.  These are balancing factors and pretty much sit in opposition.  Reducing risk costs money and that cost is not linear.  The closer you try and get the risk to zero, the more the cost will spiral.  I also have a table of performance and costs.  This was compiled for my own use and should be taken as general guidance only.  Costs may vary over time and where you live and these are in good old pounds sterling.  The performance figures were based on multiple online reviews and I tried to stick to benchmarks which reflected the larger file sizes typical for AV users.

First of all we have to distinguish between backup and archive as these terms are often used interchangeably but they do have a separate purpose.  Backup is the one that is directly related to risk.  Your backup strategy is there to reduce risk.  It is primarily concerned with work that is in progress or highly likely to get accessed.  In a commercial environment this is often tied into a Disaster Recovery Plan or a Business Continuity Plan.  These differ mainly in scale.  Business Continuity looks at how can I get working again as soon as possible or keep working without your full resources for a short period.  DRP is more about recovering from vary large scale interruptions or total loss.  The business terms may sound over the top for a prosumer but the principle of risk analysis are just as valid outside big business.

You need to decide:

• How much work you are prepared to lose
• How long you are prepared to wait to get it back
• How much time you would be prepared to put into reloading things like software

Archiving is different in that it is an operational decision.  It is about treating material which you expect to use infrequently differently to that which is more likely to be needed.  Even in my simple set-up I have some storage that is speedier or more expensive or more difficult to extend than others.  I really don’t want to waste that by filling it full of stuff I do not expect to access.  I also don’t want to have to such through all that static data looking for what I am currently working on.  This is not stuff I will never look at.  The proper place for that is the trash can.  Original source files are a good example.  Its best practice to keep them but access will be rare.  You don’t need them handy but you do need to be able to get to them if you need to.  

Once you have placed data in an Archive this does not mean that you don’t also need to think about it in your Backup Strategy.  However, the approach you take will be probably be different to that in your working data.

Risks

Storage Failure

This is the most common form of data loss.  Hard disks are mechanical devices working on very fine tolerances.  They are vulnerable to shock and environmental issues as well as wear and tear.  Flash memory devices are more durable but have their own eccentricities which can affect longevity.  Optical disks are vulnerable to poor handling and decay.  Tape based media have a good reputation for durability but they are still a magnetic medium and they can wear or snag whilst being read.

The mitigation for storage failure is duplication.  If your data is on two drives the the mean time between failure (MTBF) is doubled.  That good but the real key is that the chances that both fail at exactly the same time become tiny.  You can duplicate using backup software to a different device, use folder synchronisation or Time Machine on Apple.  The most pure form of duplication is mirroring (also known as RAID 1).  This is either software or hardware controlled exact and real-time duplication of two drives.  I will come back to this in a bit more detail.  Mirroring is built into OSX at the software level and many PC’s have it at the BIOS level as a feature of their motherboard.

Data Corruption

This is not the failure of the data being read on the drive, this is an application error which writes bad data on to your drive.  Final Cut Pro 7 and Premiere Pro CS5 have both given me this issue in the past.  Both have an auto-save feature but it doesn’t always save you.

The mitigation for data corruption is versioning.  This is what auto-save is trying to do and is now built-in to OSX Lion.  It is also inherent in Time Machine and most backup software options.  However, unlike for storage failure, mirroring is not a mitigation, it will have two identically corrupt files.

Theft and Disaster

It seems an inevitable feature of the nature of the criminals that they are not going to leave that shiny drive array sat next to your Mac Pro when they are ransacking your place out of the goodness of their hearts.  Given time they are going to take anything plugged into your surge protectors and then they are going to nick those too!  This is the same with fire, flood

and other acts of God.  A wireless device like a Time Capsule may escape notice but you cant put it anywhere where the wireless signal will be poor.  A proper fire safe is probably beyond the reach of the prosumer.  The only option therefore is to get the data somewhere else.  This can be a physical or a virtual (cloud) location.

Virus & Security

Here the threat is to the data itself not the hardware it sits on.  Malicious code can brick your machine or force you to quarantine it from your own and public networks.  Hackers may seek to steal your data from your primary location or you backups.  If you have material of high sensitivity this may limit your backup options.  The mitigation here is to prevent the illegal access ever happening.  Once that firewall is breached then the mitigations are similar to Data Corruption

Running Out of Space

Is this really a risk?  Probably not literally but it tends to cause all kind of other issues.  The first problem is that things tend to break when the space runs out:  files get corrupted, background processes fail to run etc.  This may be something that is not immediately notice.  Secondly things tend to slow to a crawl as the system struggles to find a good place to put things.  There is also a tendency for you to start putting things where you know they shouldn’t be because there is no room in the right location - or is that just me.  One thing my long years in the IT game have taught me is that what seems a ludicrous amount of storage today will seem merely adequate in just a few years.

Devices

Internal Options

This is just about feasible on a Mac Pro but not the other Macs and MacBooks due to limited access and expansion capability.  Internally attached drives tend to be quicker than external single drives.  Anything even vaguely modern will be using SATA drives and this interface has been evolving through 1.5Gbps, 3Gbps and now 6Gbps.  However, you still only have 4 drive slots and your eggs very much in the one basket.  I would recommend concentrating more on performance on your PC and doing the backups elsewhere.

External Hard Drives

Absolutely massive choice of devices here so we need to split them into smaller categories.  They come in a variety of interfaces and may be pre-formatted for OSX or Windows although this is usually changeable.  They also can come with a variety of interfaces in order of speed USB2, FW400,FW800, USB3, eSATA, MiniSAS & Thunderbolt.

Bus powered

These are normally 2.5’ SATA drives in a protective case.  They are very portable and are typically in the 250MB to 1TB range.  USB2 is by far the most common but you also see USB3 & Firewire at a premium.

Western Digital My Passport Studio for Mac

Desktop

These are normally a 3.5” SATA drive in a housing.  The need plugging in somewhere and come in the 500MB to 3TB range.  Pretty much every interface type is available but faster speeds and multi-interface is at a premium.

Seagate FreeAgent GoFlex

Toaster

This is a drive interface and a SATA connection with a disk loading slot. Effectively the skeleton of Desktop External drive.  Most fit either size of SATA drive but for backup larger drives are the obvious choice.  It allows you to work with drives like you might have with tapes but remember tapes were born to be handled.  Bare drives are naked and vulnerable.

NewerTech Voyager

Flash Drives

These are stick like drives which are solid state.  Small and very portable they are limited to about 32GB currently.  USB2 is again by far the most common but a few USB3 sticks are beginning to turn up.

This is the entry level of backup solutions.  Bus powered and Flash Drives have limited capacity and are best seen as travel options and as good partners for Laptops where the primary storage itself is of limited capacity and probably slow.  For the same money a Desktop drive will be faster, bigger and with a better chance of a high speed interface.  This goes for Toasters too which offer access to excellent cost per MB figures providing you are a gentle disposition

Network Drives

There are a few single drive devices which are not direct attached.  In the Mac world the most obvious one is Apple’s Time Capsule.  This is actually a network router with a hard disk in one Mac Mini sized box.  Its a single drive of up to 3Tb capacity.  This is designed to integrate with the Time Machine service in OSX.  Most other similar drives are now designed with multimedia delivery in mind although they might be usable in this role with software or as a Time Machine Target.  Some modern routers have a USB port which allows a normal external drive to be attached.  This helps with sharing and with placement options but performance will take a bit hit compared to attaching that same drive direct.

Apple Time Capsule

Drive Arrays

I am calling anything with more than one drive an array but the breadth of devices that covers is pretty massive in terms of capacity, cost and implementation even if you rule out the massive SAS based storage arrays

If we look at them in terms of architecture first.  

Direct Attached

The simplest kind is a direct attached device.  In this format the drives attach just like a single drive external drive.  Intelligence in the device is limited.  Normally it is just adding a RAID controller card and some drive bays to an external drive.  Prosumer versions tend to look like stretched versions of a makers external drive and have just 2 drives.  Usually only RAID 1 (mirroring) and RAID 0 (striping) are offered.  RAID 0 gives you the full capacity of the dual drives and improved performance over a single drive but doubles the risk.  As such it is a poor choice for backup.  RAID 1 halves the risk but also the capacity so a maximum of 3TB.  If you can fit your backups in this size these drives in RAID 1 are safe pair of hands and good value but inflexible.  Slightly more expensive are versions which have removable drive caddies.  This opens up the world of drive rotation and they are a lot easier to get back online if you do have a drive go down inside.  Some of the manufacturers tie you into proprietary drives though which adds to the potential expense.  

Once you get beyond 2 drives new options become available.  Depending on just how many drives you have you can start to play RAID bingo.  Their are bewildering numbers of variants of RAID available each with its own subtle characteristics in terms of safety, performance in capacity.  Few devices support every possible option so its best to do your research on RAID levels, find the one which best matches your needs, then look for devices that support that RAID level.  To give an illustration I am going to look at the most commonly implemented on which gives good capacity and middling safety and performance - RAID 5.  In a RAID 5 array you lose one drives worth of capacity from the array and you need at least 3 drives.  So a 4 x 1TB RAID 5 array gives around 3TB of space.  The more drives you have the better the ratio of what you get to use.  For larger installations getting past the maximum physical drive size is essential.  Remember, backups may contain multiple versions of a file so you may need much more space for backup than you are using for your primary data.

Most multiple drive arrays have a choice of interfaces.  Its important to realise that this may be the deciding factor when it comes to performance.  It doesn’t matter how fast your drives are or which RAID level you picked, the data rate can’t exceed the bandwidth of the interface you use.

One thing to be aware of is that not all enclosures designed to hold arrays have the interface controllers built-in.  If you see an array described as a JBOD (Just a bunch of disks) then they will require a controller card (typically eSATA or MiniSAS) to be added to your PC.  For Mac users this means you will need a Mac Pro or a MacBook Pro with a PCI Express slot.  For any other Mac you will need one with the controller built in.

One last point about arrays - they can be touchy blighters!  All arrays have some hard maths going on in the background to make sure they OK.  If anything goes wrong then they have to either rebuild or go through diagnostics.  Now this is way better than losing a whole load of data but it will at the very least reduce performance while it happens or even take the device off-line.  This can take hours or even days.  I had a Windows NT machine with a bios based RAID controller.  One of the things that would upset it was the PC crashing or needing a forced reboot.  I said this was a Windows NT PC right?

G-Tech G-Speed QPromise Pegasus R6 (first Thunderbolt Array available)

Storage Robots

This term comes from Drobo and really the only thing that comes close in principle is my old Windows Home Server.  The main difference to the other device types is that Drobo and Microsoft did not go down the RAID route to achieve redundancy.  RAID is very fussy about disks.  Ideally RAID likes matched sets of the same type and capacity.  It will tolerate  a smaller capacity drive but at a big cost in overall capacity as everything works of multiple of the smallest drive leaving capacity unusable. Drobo & WHS will take anything you have handy and present it as a single drive but still retaining fault tolerance should a drive fail.  Using all your bays and need more space?  Take out a small drive and add a bigger one - the device does the rest.  This is impossible with a RAID array without changing all the drives and massive amounts of rebuilding and no small risk.  With RAID go as big as you can justify from the outset.  This approach focusses on easy of use and flexibility rather than performance.

Microsoft seems to have gone cold on WHS and the latest version has lost this feature so Drobo is now unique.  Its worth noting Drobo now offers NAS and SAN options as well as direct.  

Drobo S

Network Attached Storage (NAS)

A NAS is really a small server optimised for sharing data.  The RAID arrays are the same in principle as a direct RAID device but a NAS is seen as a completely separate device on your network where as a direct attached device appears as a local disk on your PC.  The benefit is that multiple computers can use the device as a backup target but network speeds rarely match those of direct interfaces in the real world as that bandwidth is shared with other traffic.  These are servers, so while they tend to look superficially the same, performance can vary widely due to CPU, memory, network and disk controllers.  There is also the quality of the interface (normally browser-based) to consider.  Strong competition in this sector keeps prices down so they offer good value if your priority is capacity and consolidation not performance.

Promise NS4600

Storage Area Network (SAN)

This is a bit of a specialised device.  Its really designed for big server rooms where multiple diskless servers are connected by a high speed fibre interface to a big box stuffed full of disks.  Each server is allocated a proportion of the disk and thinks it is local.  When you have a lot of disks the high cost of the interface hardware is offset by not having to provide a high level of redundancy and case room for each individual server.  There is a lower cost option called iSCSI that uses a network connection instead of the fibre.  But its still more a way of providing primary storage rather than a backup tool.

Optical Storage

We know Steve Jobs would like to kill it off but most computers still have some kind of Optical writing capability.  Most likely this is good old DVD.  Compared to magnetic and flash media DVDs are as cheap as chips.  They are also pretty easy to store and should last a while if you don’t go too cheap and handle with care.  But when a single memory card from your camera might have 32GB on it then 4.7GB is started to look a bit titchy.   Dual Layer disks help but are 10 times more expensive.  Blu-ray ups the capacity again to 25 or 50GB but again the cost rises dramatically and writers have never gained the kind of popularity which will drive the prices down to DVD levels and its doubtful they ever will.  Capacity is not the only issue - optical media is very slow to write and not very good for iterative saves.

Cloud Storage

This is a an internet service provision where your data is stored in a data centre operated by a third party.  This storage will be enterprise class, high redundancy and high availability.  All you need to do is set-up an account and run a client utility on your PC.  Most companies will offer different subscription models to suit users who have different size requirements or multiple devices.  They are split into two main types of service, synchronisation and backup.  

Dropbox is the best known of the synchronisers.  Rather than back up your whole disk you place data in a specified virtual folder structure.  The Dropbox client then replicates this in the cloud.  Any device you have Dropbox on, including mobile devices like smart phones, can access the data or you can go through a web client.  You can even elect to share folders with other people you nominate.  This is really designed for people who want to collaborate with others or tend to use multiple devices.  Its not really meant as a backup solution.  Its great for ad-hoc backups of relatively small amounts of data though.

The second type of server is aimed at the backup market.  Providers include people like Carbonite, Backblaze and CrashPlan.  My own experience of on-line backup as not been great but if I wanted a better one I would have to move.  The upstream speed on my ADSL is a paltry 400Kbps maximum.  If its at maximum it really seems to interfere with the downstream rate too.  With Carbonite in its low priority mode I only managed about 30Gb of backup over a month.  So you have to make sure that you link speed is going to make online viable for the size of data you need to backup.  Some vendors do offer a hard disk data take-on option.  This means that you only have to worry then about new data or changes being backed up.  They may also offer a hard disk restore option if the shipping time would be quicker than restoring over the net.

Arguably the biggest issue with a Cloud service is that you lose a degree of control over your data.  All the companies publish their policies on security and encryption but once it is in the Cloud you can never be truly be sure where it is or who can access it.  Unless you are a specialist it should be the case that they are better at security than you are but they are also a more obvious target.  Hackers or disgruntled staff are not the only risk - what if the company goes out of business?  If this is just a backup of your holiday movies from Majorca than its probably not an issue. If it is confidential work for clients then you need to take this into account. 

Other People Disks

This is a service that CrashPlan introduced me to.  Their client allows backups to other devices running CrashPlan.  This can be on a private network or via the internet.  The expectation is that this will be a “buddy” system where you agree to provide a reciprocal arrangement with someone you trust.  The performance limitation will be primarily the link you choose.  A private network should be much faster than an internet link.  A hard disk starter backup will probably be advisable.  It’s unlikely that your mate is going to have the same level of kit as a Cloud based service but at least you “know where they live”.

Tape

It may be old-school but tape is still a very effective option for both archiving and backup.  The standard that is best supported for big data volumes is called LTO.  If you do a bit of research you will see numbers after that and these are significant.  It refers to the generation of the standard.  Each generation has grown in tape capacity and data rate.  Backwards and forwards compatibility are enshrined in the standard as are the expected life and re-uses a tape should manage.  This is an enterprise class solution which means heavy-duty, fast, reliable and expensive.  Most of that cost though is up front, the tapes themselves are less expensive than other mediums.  You are going to need to have a lot of data though before you break even with disk based alternatives.

Quantum LTO5 Drives

Archiving

Archiving is “cut and paste” to Backup’s “Copy and Paste”.  If the data you are archiving still exists in its original place when you started then you didn’t really archive it, you just backed it up.

Why archive?

So if we haven’t improved our data security by archiving it why even bother?  Well there are several reasons why it is beneficial to do so.

Freeing up scarce resources

Perhaps you edit on a laptop in the field and to maximise your performance you have a fast but small SSD drive.  You would soon run out of space on that SSD if you leave old projects on there.  Some NLE’s have media manager functions that can reduce a project to just the assets used when they consolidate and move a project.

Move to safer environment

Resources like laptops, portable drives, fast RAID 0 arrays have inherently more risk attached to them.  Moving data to a less risk environment is advisable.

Confidentiality

If you are working with multiple clients you may want to avoid one client seeing any assets or folder names which may reveal the identity of another client.

Allowing a different backup regime

An archive isn't a copy.  So archives need backing up like everything else.  However, they are going to have a different risk profile and data change rate to projects in progress.  It makes no sense to tie up resources backing up the same file every night when there is only a tiny chance it will change.  If you do get to a situation where an old project needs substantial rework then it should be moved out of the archive back into the work-in-progress storage.

Cataloguing

You have to assume that by the time you need something in the archive that you will have forgotten all about it.  At my age, this is almost certainly true.  So how do you find stuff when people ask?  Perhaps you are just interested in finding something to use as stock footage in another project and just want to look on spec.

Folder and naming conventions

This is the place to start.  A degree of of orderliness is essential if you want to be able to find things.  How far you have to go with this probably depends on your working environment.  There are defined practices for movie and broadcast that you can use but these could be overkill and bring in a new level of jargon.  Some products may generate or impose the structure for you.  If you are using multiple software solutions sometimes you may have some conflicts.  My own general principles has to be to try and keep any software bin structures and physical folder structures in sync as much as possible.  Try and be descriptive with the project folder names.  If you are going to use any coding schemes then you need to be consistent and you need to make sure the key or index is accessible.

Metadata

With FCPX Apple moved to a metadata focused model from a bin based one much to the consternation of the editing community.  To me this wasn't too much of a shock to adjust to.  Not because of iMovie but because this is the same model that Aperture uses.  What did shock me was the poor integration with Aperture from FCPX but that is a different rant.  As a database guy, I appreciate that tagging is able to encode the multiple attributes that a clip possesses in a way bins can’t without a lot of duplication.  However, you have to be disciplined about it.  Video is a bit sparse for metadata compared to stills with their well established EXIF standard.  However, I think this will change over time and more metadata will arrive with the files themselves.  One of the interesting features of FCPX was autotagging based on analysis.  For example, it attempts to tag clips as wide shots or close-ups or for shakiness.  Adobe has some interesting technologies for voice recognition which could also find their way into metadata.

Off-line Cataloguing

If you use tapes or a tape-like workflow with other media, then you are going to need to generate a catalogue.  This is because your data is sat in a cupboard somewhere and is not available for searching.  So you need to capture the data structure and metadata into a stand-alone database that remains on-line.  You also need a physical indexing system so that when you search the catalogue you can go straight to the right disk or tape in the cupboard.

Retrieval

Whether its an archive tape or a backup disk its utterly useless if you can’t read it.  Some LTO tapes are guaranteed for 30 years.  30 years ago, in 1981, myself and three school friends won our school its first computer (a BBC Micro) in an inter-schools quiz on TV.  I think I would struggle to read files a BBC Micro saved onto cassette tape these days.  To be honest it was a bit hit and miss in 1981.  I have never really re-used my DV tapes and they acted as an archive once I had transcoded the footage onto to disk.  However, my aging DV camcorders are getting a bit flakey.  When the last one dies I lose my archive or will be scouring Ebay.

To ensure retrieval, you often need to transfer the data to a newer format prior to the old format being obsolete.  However, you need to be mindful of retaining as much quality as possible.  In the heyday of DV and DVD, burning to a DVD may have seemed a decent archive solution.  However, the transcode to MPEG2 within the DVD’s storage limit may have degraded the image.  However, a data DVD backup of AVI files created in the DV codec should be a true copy of what is on the tape.  But how long will that codec or file format be readable?  Always be on your guard for technical obsolescence whether it be hardware or software and be planning your transfer solution.

Summary and Recommendations

So my set-up is a paragon of virtue?

Err...not so much.  Part of the reason I am informed on this topic is I am perpetually in search of that ideal solution.  I am more of an experimenter than a completer finisher so I am always wrestling with finding a method and not quite as good at implementing it.  However, imperfect as it maybe, I do have some of the risks mitigated.  Lets go back to my basic questions and I will answer them for myself:

• How much work you are prepared to lose
  • Nothing, its not important to anyone else but I would hate to lose it.  My inner squirrel can’t cope with that.
• How long you are prepared to wait to get it back
  • No rush, as long as I know I can get it back time is not a big issue
• How much time you would be prepared to put into reloading things like software
  • I have a low tolerance for repeating things I have already done.  I have loads of software & plugins and the like.  I am prepared to lay out some cash to avoid that pain.

So how am I doing in mitigation of risks then?

Storage Failure

My main system drive is covered by Time Machine and my original media archive is on the Windows Home Server so they are OK.  However, my project drive is too big to fit on the same Time Capsule so I am having to do periodic back ups to an external drive.  This is not ideal.  I also have a few chunky things that have found their way onto my fast external firewire RAID 0 drive that I know shouldn’t be there but won’t fit anywhere else.

Data Corruption

Time Machine is a versioning solution so is good protection and the stuff on WHS is static so low risk.  I do have 2 generations of backup on the WHS too.  Again I am vulnerable on my other drive and the RAID.  My adhoc backups are overwrites so give less protection.

Theft and Disaster

I now use a Toaster to backup to bare drives which I transport in inexpensive plastic boxes.  I have 2 which I use alternatively and give to my parents who live 30 miles away.  I should do this a bit more often than I do but its a manual process and time flies.  I tried online backup but my rural ADSL is the limiting factor.

Virus and Security

I am a Mac user so the threat has traditionally been lower.  However, I have my doubts this will continue given Apple’s rise in market share and No1 company status.  I keep up-to-date with software updates on all devices and I have anti-virus and firewalls running on my PC’s and Mac Pro.  I don’t have significant Cloud based assets to be under threat.

Archiving

I don't have sufficient data or operational driver to make this necessary yet.

So if you know better, whats holding you back?

Well there is cash for a start.  Its money well spent but this is a hobby and there are so many, many things on my wish list... damn you Internet.

Secondly, I am that uncomfortable stage where 2TB drives are a bit under pressure but 3TB drives are still a hefty premium.  Once you scale up your primary storage everything needs to move up with it.  Also I am excited by the potential of Thunderbolt but we are still short on peripherals, including anyway to get it on my Mac Pro.

My non-media data is fine how it is.  I am not struggling for space and a Time Capsule is a good solution due to its risk and change profile.  With a bit of careful targeting I could probably even make the Cloud work for me whether this is through iCloud or another service we will have to see when Apple’s full plans are more evident.

My original media store is fine where it is.  My little WHS might be a technical dead-end but it does the job and has some growing room still.

My media drive I am hoping to replace with a large Thunderbolt array.  Thunderbolt is a fast enough interface that some of the old data organisation best practice designed to stop you hitting bandwidth bottlenecks just don’t apply.  This makes data organisation less complex.  I want to have 2 x 3TB drives in the array configured as RAID1 for backup.  Periodically I would swap one of the drives and keep my sneakernet arrangement with my Dad.  Fingers crossed they find a way of squeezing 4TB on a drive before 3Tb gets too small.

I could get the job done with something like a MiniSAS array now but I feel, given the level of investment, I should hang on a bit.  Without an option to retrofit Thunderbolt to my Mac Pro, I may not get the option and have to go that way.

Some helpful links:

Chris Fenwicks archive on a budget

16x9 Convergence podcast

A Tale of Two Softwares

I was there at NAB when Apple unveiled FCPX.  I was downloading it the second the first tweet broke and I have been dabbling with it now and again since.  But I am not an editor, I am just someone who edits.  In fact I am slap bang in Mr Jobs’ target market if the initial functionality is anything to go by.  I can afford a certain degree of detachment because its not my day job.

LAFCPUG Supermeet NAB 2011

My day job is as a Business Intelligence Consultant.  However, it just so happens that the main software I rely on is also going through a generational change.  In its way it is the same level of paradigm shift that FCP is enduring.  The software in question is called Business Objects, now part of SAP.  In truth, its really a vast suite of interconnected products but, for short, it has become known as BI4.

The outgoing version is called XIR3 and has been around about 3 years.  It was a mature product but one that was carrying a lot of baggage due to acquisitions.  The basic idea in the centre had been around since the early 90s and had been very successful but was showing its age.  Some of the acquired modules were based on a different premise and functionality overlapped between them.  SAP made the difficult call to rewrite that central idea within the product.  Is any of this sounding familiar?  One thing BI4 does not share with FCPX is that it is not consumer oriented - one of the organisations that I work with has 11000 licences.

SAP ASUG Sapphire Conference Orlando 2011

So, like FCPX, BI4 is 64bit.  One of its most popular tools has been canned despite there still being a huge legacy of projects using it.  The central way of working has been completely changed and existing developers will have a lot to learn but all the remaining tools now integrate much better.

So how does SAP deal with its customers and its release compared to Apple?

Roadmap

A software roadmap is an expectation for Enterprise software.  Big businesses are like supertankers - it takes a huge amount of energy to change their course.  If you need to make a course correction you are going to need to know in good time. Most enterprise software consists of multiple platform layers and each of these has its own life cycle.  SAP share their roadmap with customers and major customers may be involved at a very early stage in the development.  SAP like to talk to there customers and you will be bombarded with material once they know your name.

Legacy Support

SAP will normally try and support about 2 versions back for migration at a platform level.  Where things won’t migrate, they provide conversion tools.  However, its not unknown for some functionality to go entirely.  Support for old releases dies off over time, the first stage is that updates and patches will stop followed later by telephone support.  However, this is years after the release of the new version.  If SAP think it may be controversial they will normally extend this period.

Soft Release

This has come in only during SAP ownership of Business Objects.  I have been going to launch events since February but technically BI4 is not for sale.  It is currently in “Ramp Up”.  This is like a limbo between beta and release.  Only selected customers have the software, the rest of us are waiting for SAP to declare general availability.  This date has been creeping backwards.

Completeness

Despite “Ramp up” the software will be missing some functionality at general availability.  Some functionality is promised for BI4.1 and the roadmap puts that before the end of the year.

So in summary:

• SAP likes to engage its customers and keep them informed of new developments whereas Apple has a cult of secrecy.  
• Apple is used to having customers who are not significant in their overall volume where as SAP customers tend to be a more significant proportion of revenue.  SAP has maintenance whereas Apple has upgrades.  If you are paying for upgrades up front you would feel more entitled to know what you were going to get.
• SAP gives customers plenty of help to migrate where Apple has made hardly any concessions
• Apple is pretty good at hitting dates once it divulges them, SAP not so much.
• Both Companies are prepared to make the tough calls when they think architectural changes are required.
• SAP are prepared to put out software that is functionally incomplete but they limit who it is available to whereas Apple have fully released FCPX.  There are several orders of magnitude difference in price though.
• Despite mutterings about the lateness of BI4 to general availability, I have not heard the term "debacle" used at all about BI4.

So whilst there are some similarities in the situation surrounding FCPX and BI4, the vendors could hardly be more different in approach to their launch. I don’t want to just Apple-bash because I am a huge fan.  I love their clarity of vision but I think they need to temper their own corporate culture when they are dealing with people livelihoods rather than just their lifestyles. 

Aperture and FCPX

One of the things which struck me as odd is that despite the media browser section having access to Aperture’s library, there was no sign of any video clips in the browser.  This seemed to me like a massive missed opportunity from Apple but it seems I may have been a bit harsh.  This looks like it is a bug not a missing feature as video is mentioned in the FCPX help.  As such, I hope it gets fixed soon.

You can find the knowledge base article here 

The help is a little unclear so here is what my experiments have shown:

• You have to drag the image, multi-selection or Aperture project and drop it on the FCPX’s Event title in the Event Library not into the panel that where the assets or import options are.  You should see a + or a file count where it is valid.
• If you have done a version copy in Aperture it will see that as a different  the file in Aperture then FCPX will only see the trimmed portion.
• It won’t let you use drag in smart albums but you can still use them to preselect what you want to copy.
• Can’t see any evidence of Aperture key words being carried over.

... and your point is?

If you are someone who shoots mixed stills and video on DSLRs like I do then Aperture is often your first point of call for capture.  I have already written about my workflow here.  If you are using Aperture then both stills and video are ingested and you are already making decisions about organisation.  Whilst Aperture can’t currently modify video images the way it can with stills, you still have the ability to trim, grade (stars and flags) and key word.  It also has a robust library with the vault backup option.  With no log and capture anymore, Aperture can fill some of the gap.

Even when Apple fixes the bugs, I still think they could go further with the integration.  It would be nice not to have to duplicate the primary assets and it seems a shame to lose any effort you may have put into key wording in Aperture.

Initial take on Final Cut Pro X

Its been a crazy time since the launch of FCPX last week - a regular supernova in the twitterverse.  I hit go on the App Store the moment it popped up in the UK.  I was there for the debut at NAB, I remember the buzz by the end of that fateful Supermeet.  The very first impression was not a functional one but a financial one.  The days of cut $ and replace £ seemed to have ended with the App Store.  So the price that got the Supermeet crowd on their feet was still present for a UK audience.  As expected the price wasn't quite like for like with FCP7 due to the unbundling, but the ancillary apps were less than I expected both in terms of number and cost.

Unfortunately, the excitement of the launch soon turned to howls of protests.  Have the iMovie Pro protagonists been vindicated?  Have the Pro community been left out in the cold?  I haven’t deep dived into the software myself.  I did enough playing to decide that I needed to pull back and have a good look at this new paradigm and decide how best to engage in it or whether to engage at all!

I have read a lot of the comments and listened to the arguments for and against and I think I am finally in a position to have a draft opinion.  Not a final one, because that is going to need me to have a lot more direct use of using the software myself.

Is it Pro?

“No, it isn’t” is the most obvious answer but you have to be very careful how you define “Pro”.  FCPX is a Non Linear Editing tool and so the profession or craft you would associate that with is Editors - be that for films or television.  As a collaborative tool it is severely limited.  A situation compounded by Apple both failing to provide a transitional route and killing off FCP7 with almost indecent haste.  Although I don’t fit in this user category myself, I understand their frustration.

However, within the admittedly high walls that limit its functionality, the software is certainly capable of producing professional results.  The ground-up rebuild does mean that the software is capable of professional output in a shorter space of time with less recourse to additional programs or plugins than its predecessor.  With so few plug-ins available thats just as well but that was always going to be the case when 64bit arrived.  I’ll concede that some of that new functionality maybe a bit too “black box” for the purist but there are some exciting things both in FCPX and in Motion 5.

Is it the NLE for the rest of us?

Well, Apple will tell you that is iMovie still.  FCPX is still a step up from there but an easier step than FCP7 used to be.  For someone like myself who is an enthusiast who aspires to professional results then its a very good fit functionally and financially.  I can also see it suiting creative professionals who are not dedicated editors.  

The demand for video content is growing almost exponentially but much of this growth lies outside the realms of traditional post houses.  I think this is the market Apple is gunning for and, if it exceeds, could be very lucrative.  Would Apple forsake the high-end customers for this market.  I think they would...I think they have.  I think they will gamble that the high-end will move or someone else will fill the gap even if they don’t.  I do expect to see FCPX change quite quickly though.  Its the App Store way of doing things.

Choosing a different path

Why bother learning a new paradigm, isn’t it easier just to jump ship and go to Premiere?  I have been using Premiere more over the last year than FCP7.  There have been a couple of reasons for this.  The first is I have been shooting nearly everything on DSLR and Premiere handles DSLR footage much better than FCP7.  I have a Mac Pro with 8 cores, 12GB RAM and a Quadro 4000 and FCP7 runs pretty much the same as it does on the older iMac it replaced.  Only in Premiere did my Mac Pro really fly.  I have been trying to learn After Effects and the integration via Dynamic Link is really cool.  If you are a big user of Photoshop and After Effects then its sort a no brainer.  The major downside is cost.  Most people will buy the Production Premium pack and that is a big chunk of change.  The full version is an order of magnitude more expensive and even the upgrade from 5.0 to 5.5 is dearer.  Even the month-by-month subscription cost is close to the outright cost of FCPX.

My First and Last Shuttle Launch

I am a child of the space age.  I arrived in the world just days before the first Saturn rocket tests.  At the tender age of five I was woken and brought downstairs in the middle of the night to watch Neil Armstrong set foot on the moon.  As a teenager in 1978, I visited the United States for the first time and went to Kennedy Space centre.  The mighty Apollo Saturn V still lived outside back then and NASA was busy trying to get the shuttle program operational.  As if that wasn’t enough to turn me into a space nerd then I had the combined efforts of Messrs Lucas, Spielberg, Roddenberry and Anderson to help me along.

Despite many trips to Florida since then, I had never been during a launch.  With just the final few Shuttle launches to go, I never expected that opportunity to arrive.  A few weeks back I was jealously watching tweets from Stu Maschwitz and Trey Ratcliff as they waited for the take-off of STS-134.  I was looking at the mission specs and wondering whether I might be in Florida in time for the landing.  Unfortunately for Stu and Trey, a combination of weather and technical issues delayed the launch a couple of weeks.  This meant I would be in the right place at the right time for the launch - a rare combination for me.  So rare in fact, that I was pretty sure that something would come along and send my plans pear-shaped.

The first small problem was that the launch happened to coincide with the key note at SapphireNow, the conference I was actually in Orlando to attend.  To be honest, I didn’t struggle with that dilemma anywhere near as much as I should have done.  My father was going to come along for the trip.  He had tried to get to a launch many years ago and got stuck on the toll road.  So an early start was in order to travel the 50 miles or so to Titusville.

No vantage point is wasted 

Launch day dawned and the NASA website still said it was a go for launch.  We stuck our selected viewing site into the sat nav and headed off.  This trip was not designed to be a photo-oriented trip so I was pretty lightly armed camera-wise.  I had only taken my Olympus XZ-1 with me which maxes out at 112mm.  So I had borrowed my Dad’s Lumix TZ7 for a bit of extra reach whilst he had his Panny palmcorder.  The worries over traffic turned out to be unfounded and we reached Titusville in good time.  Parking was another matter.  Most of the public parking was already taken, which left enterprising souls renting out spots on private ground for $20-30.  Whether they had any title to that land is a moot point but after 3 laps of the one way system we were ready to take that chance.

Space View Park

Space View Park was already pretty crowded as we walked in.  Whilst the weather was no issue for take-off, it was not brilliant for spectating.  There was a layer of low cloud which meant the viewing opportunities were going to be short.  This was not helped by us not being exactly sure where on the peninsula opposite we were supposed to be looking.  We could see the VAB through the haze but not the shuttle itself.  There didn’t seem to be a consensus of direction amongst those with more elaborate gear on which way to point.

The Vehicle Assembly Building

As the countdown continued the crowd’s anticipation grew.  On the jetty opposite there was a relayed PA from the Space Centre but we were to far from it to hear details.  It was more felt through the increasing buzz from the crowd radiating out.  To get a clear view above the crowds my Dad was using a convenient lamppost as a support with his display angled down.  I was using his shoulder to get the TZ7 high enough and could see virtually nothing on its fixed display.  Suddenly there was a roar from the crowd.  It turned out the launch position was slightly behind a palm bush from our position.  I struggled to find it on the Lumix.  I squeezed off a few speculative shots and then decided to just watch the final couple of seconds as the giant torch supported by a thick column of steam speared its way through that inconveniently low cloud barrier.  One of the problems with being of a photographic persuasion is that you often only see events secondhand.  Sometimes you have to overcome your urge to record and just leave it to your eyes and your brain. 

One of my speculative shots paid off

Just as we thought the show was over the noise arrived.  The source of the aural assault that surrounded us had already departed from view.  The only trace of its progress being the stripe across the clouds, the shadow cast by its vertical ascent across the top of the cloud layer.

The plume shadowNASA Footage of the Launch, see more on NASA TV