An Argument Against Future Proofing RAW Files

I see this question come up with some frequency on various discussion forums. What format should I use to future proof my digital images. The answers of course vary from JPEG, to DNG, to TIFF or PNG. Very rarely do I see someone who responds with don’t do anything, which is exactly what I recommend.

This may sound shocking, but let me explain. There are two broad aspects as to why I don’t recommend messing with your digital negatives now for future proofing reasons. On one hand, there’s the technological and technical side of digital storage and computers; on the other, there simply the time aspect.

Let me start with the time aspect. Doing anything that involves converting RAWs now is a waste of time for two reasons.

First, is that it’s simply not a foregone conclusion that the manufacturer RAW files will become unreadable. A lot of DNG proponents like to play up the fact that DNG is an open standard and the proprietary RAW formats aren’t. They argue that because of that a DNG can be expected to be readable at some arbitrary point in the future. While that’s true, it ignores the reality that by far the vast majority of manufacturer RAW processing implementations were written by reverse engineering the RAW files themselves and not by using some manufacturer provided library. So long as people still understand graphics processing and statistics, there will be people who can understand converting a Bayer pattern blob into a usable bitmap image.

Moreover, even if it does become necessary to convert to another RAW format at some point in the future, the realities of the progression of technology suggest that doing so at that time will be faster than doing so now.

The second aspect is relates to the realities of digital storage media and the progression of technology. It seems to me that many of the people talking about future proofing their RAWs, are really looking for the digital equivalent to being able to stick slides in an archival container in an archival vault and not think about them for 50 years.

The simple realities of digital storage are that the media degrades and the devices to read it become obsolete and disappear. Long-term digital storage is more about moving data to new media in a controlled way than leaving it alone in a vault somewhere to degrade into uselessness.

Also, it’s important not to forget the management of the physical media involved. Newer media has higher storage densities, meaning you need fewer disks or tapes to store the same amount of data. The counter point is that with more data in fewer containers the loss of a single container means losing more data. That said, if you can consolidate 4 disks or tapes of data into 1, you could keep 2 duplicate copies and still have fewer disks or tapes to keep track of while having more redundancy in the system.

Given the realities of progress, it’s not hard to envision a scenario where the files on an archived disk are still readable in current software, but the disk itself can no longer be read.

The final point I want to make is that simply future proofing the RAW files doesn’t future proof the rendering of the image. What I mean by this is that the conversion from RAW data to a usable image that we can see as intended is as much a product of the software used to convert the image, as it is the data in the file.

We’ve already seen such changes in rendering in software like Lightroom. The 2010 process introduced a much more advanced rendering engine, producing better fine detail with better noise reduction, and different controls. Files updated from the 2003 process to the 2010 process not only looked different, but in some cases, couldn’t even be made to reflect the output of the 2003 process.

If there is a concern that the image might change, then it’s not enough to simply be able to read a RAW file in the future, but as much concern has to be given to it being processed under the same engine in the same way as it was in the past.

Instead of worrying about future proofing RAW files, I find it’s much more important to worry about insuring that you have the files to work with in the future at all. This means worrying about backups and data integrity. Both of which are topics that warrant a post or more discussing.

Offline Tools

Mobile Tools

I started writing my JavaScript tools to provide me with solutions to problems I had. My depth of field calculator started out as a quick way to get 1/8-depth of field values that I needed for building an autofocus target. It grew from there in to a way to compare lenses across platforms with an eye towards maintaining a fixed composition, i.e. the same angle of view and depth of field. One thing that’s always slightly annoyed me with these tools is that they only live online.

The trouble is, developing applications for mobile devices (phones and tablets) is quite a mess. The 3 major platform all use different environments, APIs, and programming languages; Microsoft uses .NET, Apple uses objective-c, Android uses “Java”. Of the three options I have the most experience with .NET, but as a user I have Apple’s iOS based devices. Even then there are associated costs with bring an app to market on any of the respective platforms. Never mind which one do I target? Apple’s iOS because it’s what I use, or Android because it’s the most popular? Or do I try and find a cross platform toolkit of some kind?

Thanks, but no thanks.

There is another option, HTML 5. All the mobile phones support HTML 5 at this point, at least in the browser, if not though some mechanism that frames the “app” as something more “native”. And HTML5 supports an offline caching mode that can store all the relevant files on the device so the app works when there’s no internet connection. Okay sure, the experience isn’t quite as good as a native app, but right now that’s a compromise I’m willing to make for the sake of getting things done. Moreover, since my tools are already entirely based on JavaScript and HTML 5, it’s relatively painless for me to port the existing tools and roll out new ones as I find things I need.

Which brings me to why I’m posting this. I’ve rolled out the first early release of my offline capable HTML5 tools. The first tool is a direct port of my depth of field and equivalent lens calculator. It uses the same underlying code as the existing tool, it’s just packaged up so you can book mark it (or if you’re an iOS user, add it to your home screen), and use it anywhere any time you need it.

I still consider this to be a beta class product, as I’m still rolling out the documentation and refining the functionality. It’s currently been tested in iOS 7.1 and should support running as a full screen app. Simply visiting the site should download and cache the content required to run the application offline. Help and other documentation is still missing but is on the to do list.

Click the button below to go to the mobile tools landing page.

Mobile Tools

Script to find and Delete JPEGs form RAW+JPEG Pairs

Documentation

The other day, a friend of mine trying to free up some space on their computer asked me if I knew how to delete the JPEGs from a RAW+JPEG pair in Lightroom. After scratching my head on that for a while, I threw together a Powershell script to do the job, and so I’m sharing it here in the event it might be helpful for someone else.

WARNING: Advanced computer stuff ahead.

This script is for Windows and requires Microsoft’s Powershell environment to execute. I’ve tested it in Powershell version 3 on Windows 7. If you don’t have Powershell installed, the latest version can be downloaded from Microsoft.

To run the script, place it in the top-level directory of the directory tree where you want it to scan; right-click on it and choose “Run in Powershell.” The script will prompt you before running whether you want to just build a list of files to remove or delete them. I recommend not deleting files on the first pass, just to be safe as the deletions done by this script aren’t reversible.

The script considers a JPEG to be part of a RAW+JPEG pair when the files have the same name but a different extension (.nef, .cr2, or .dng for raw, and .jpg for JPEGs), and exist in the same directory. The script recursively scans through every child directory starting from the directory the script is executed from looking for RAW+JPEG pairs.

Results are always logged to raw&jpeg.log in the same directory as the script itself.

Due to the nature of the process, the script can take some time to run, and doesn’t display progress unless it finds matches.

I recommend running the script and verifying that the files it’s selected to delete are in fact RAW+JPEG pairs, before deleting them.

This tool is provided without warranty or support.

Download: Clean RAW+JPEG JPEG.zip

 

Radio Trigger Roundup

January 2014 Edition

When I first started with Radio Triggers you had basically two options, some cheap modules of questionable reliability that were marketed under a number of brands, and LPA Design’s Pocket Wizards. That was a scant 5 years ago, and since then the radio trigger marketplace has changed dramatically. Pocket Wizards, while still the Cadillac of radio triggers, have had their reliability and capabilities matched and in some cases beaten. In short, the question of which triggers are best has become considerably more complicated.

The simple reality is that there is now a massive number of competing radio trigger products on the market in every product space. The entry-level market, for example, is awash in similar products from various brands that have little if anything to differentiate them from each other. Almost all of the studio strobe makers have joined in the radio trigger market, producing triggers designed primarily to provide users of that manufactures strobes with remote power control and cordless triggering; though many of these systems also have generic modules available. TTL systems have become more commonplace, with a range of products from inexpensive designs to fully featured and integrated ones.

My intent here isn’t to recommend a product. What I use and what I need aren’t what you use or you need, and my solutions aren’t necessarily your solutions. Instead, what I’ve tried to do is cover most of the available systems and the pros and cons as I see them. You then can use this as a launching point to dig further into the solutions you find best fit your needs.

Continue on to page 2 for an overview of features and functionality to consider. Skip to page 3 for an overview of the systems. Skip to page 4 for data tables comparing the various systems.

Read the rest of the story »