First Thoughts on Lossy DNG

Whoever said storage is cheap, never had to buy and manage that storage.

I’ve been meaning to write in some depth about the challenges of storing digital photos for quite some time, unfortunately, this article isn’t it. However, with rising camera resolutions, and the respective increase in image file sizes, finding room to store, backup and properly manage a large volume of image files is becoming increasingly expensive and difficult.

Beyond practicing shutter discipline and being aggressive with our editing, there is one more aspect that is worth considering, and that’s compression.

Lossy vs. Lossless Compression vs. Photographers

Compression comes in two basic flavors, lossy and lossless. The difference between the two is that lossless compression produces a file that can exactly reproduce the original source material, while a lossy file throws “data” away to produce a more compressible file.

Lossy compression has always had something of a bad wrap in the photographic world, even when it didn’t actually throw away anything worth keeping. Strictly speaking, most photographers, and most of the people who will be most vocal about not using lossy formats don’t really understand what’s going on behind the scenes. That’s not to say we should all start shooting in JPEG now, because we shouldn’t. But we shouldn’t be so quick to write off “lossy” compression when it’s properly modeled, and fits the needs.

I don’t proprot to be an expert in compression, but without going into excruciating detail, it suffices to say, if the “lossy” algorithm is properly modeled to exploit the behavior of light, the distribution of data in an image, or the psychology and physiology of the human viewing the image it’s certainly possible to produce a “lossy” compression algorithm that is visually indistinguishable from a lossless one.

Lossy DNG

So what does lossy DNG actually do. Well I can tell you it’s not a DNG version of Nikon’s Lossy NEF. That is it doesn’t simply use a non-linear mapping function to convert 12 or 14 bits of data into 8 or 9. In fact, it’s considerably more complicated than that, on the flip side, it does produce some incredibly tiny files with no perceptible loss of image quality and still some headroom for post processing.

So how does it work?

According to an Adobe employee’s comments in this thread, it works something like this. The RAW file is demosaiced, to produce a RGB file similar to a TIFF or BMP. The color channels are then remapped using a 1D perceptual mapping algorithm, so that their values are spread across the full range of possible values. Some noise is added—dithering—in specific places as necessary to insure that the now remapped data doesn’t become blocky. The image is then compressed using the same compression algorithm that JPEG files normally uses.

The result is a file that’s often 1/3 or 1/4 the size of the source file, that retains more fidelity than a JPEG would and has sufficient data to allow the image to be processed to a limited degree but more extensively than a JPEG.

In addition to the processing and file size advantages, being stored in a DNG container also means that the file retains the DNG’s checksums to preserve image integrity.

Finally, Adobe did handle the two edge cases, converting a JPEG to a lossy DNG and re-compressing a lossy DNG as a lossy DNG. In both cases, the image data is copied bit-for-bit instead of being decompressed and recompressed.

Size v. IQ

Given that it’s lossy the real question is how much do I really lose versus how much do I really gain. Well lets start with a simple case, in the past when I wanted to save space with a snapshot kind of image, I would export the image as a high quality JPEG, to the same folder and replace the raw with the JPEG.

The RAW from my 10.1MP 1D mark 3, and this was a best case ISO 100 file, was 11.31MB. The JPEG produced by Lightroom that I would have used in the past, is 5.4MB. The lossy-DNG is only 3.83MB. But the real question is how do the images compare?

Sample images in RAW, lossy DNG, aRGB JPEG 100, aRGB JPEG 83. (Click image to see full size PNG).

Judging the comparative quality of images is a tad more difficult. To my eye, I can’t detect a difference in detail between a 5MB JPEG 100, a 2MB JPEG 83, a 3.3MB lossy DNG, and the origioal 14-bit RAW. There is a color shift in the two JPEGs due to the color space they’re rendered in (the lossy DNG is still in the same capture color space as the RAW).

Of course these images are unprocessed, and certainly not heavily processed, which isn’t something I intend to be doing with any regularity with how I plan on using lossy DNGs. However, how the lossy-DNGs handle processing is something that I want to look at eventually. More specifically, I’m interested to see how the noise added during the quantization process ultimately effects image noise—though I’m not really expecting it to.

For me at least, lossy DNGs are definitely a useful tool that wasn’t there in previous Lightroom Versions. So far they seem to be smaller and offer more processing headroom than a quality 100 JPEG, and preserve the checksum feature of regular DNGs. Of course the real utility is based in a large part on how much storage you have and how much of your library is less than “serious” business. Granted, it’s not like any appreciable amount of my library is going to end up being converted from lossless RAWs to lossy DNGs, but as it stands on average every 3 files that I compress, I get space for another uncompressed RAW. Which in my opinion is a better storage solution than just throwing the images away.

Reader Comments

  1. On

    Brandon Price said:

    I really appreciated this article. I have just made the switch to Lightroom after being very set in my ways using a Bridge/Photoshop workflow for the last four years or so.

    I love photography, but I am not a professional photographer. I love my job and I love taking pictures on the side, whether that’s for a friends’ senior portraits or simply to bring home great travel photos. However, the majority of the pictures I take would definitely be labeled as snapshots. These are shots from parties or random events where I simply want to remember the moment and no more. I always shoot in RAW on both my Rebel 550D and my S95 because I like being able to make up what the camera lacks in post (contrast range, digital noise, etc.).

    Up until Lightroom, I would save all these snapshots as JPEGs after I processed them and then delete the RAW file, and I did this simply to save space. (Contrary to what RAW purists tell me, there is absolutely no reason in the world I would need a terribly lit picture of my friends and I sitting around my living room playing Catchphrase in my future. It was about the moment, not the art.) Any pictures that did touch my artist’s side, I would save the RAW files so that I would have the originals if the need ever came up.

    However, this workflow is not at all streamlined in Lightroom. Lightroom is designed to be non-destructive and I’m trying to destroy. It’s hard enough for me to get my mind around not working through the file manager itself, but then we add to the fact that if I want to simply save a file as a JPEG and delete the RAW, I’ve got a whole lot more steps that really take the benefits out of the Lightroom workflow.

    I’ve spent the last few days trying to really learn Lightroom theory. I’m trying to get off on the right foot so that I don’t cost myself time and frustration in the future. This difficulty incorporating a JPEG step into my workflow has been almost depressing, and it wasn’t until last night that I stumbled onto the fact that Adobe incorporated a lossy DNG option into LR4.

    I began reading more about it and found your blog. Though many are shouting “Unclean! Unclean! It’s not *real* RAW!” I’m finally feeling at ease. It’s fantastic that I can convert my large RAW files into Lossy DNGs right in the library. There’s no re-importing to the LR catalog and then reassigning to a collection. The lossy DNGs are the same size (if not smaller) than my JPEGs usually are, and with Lossy DNG I still have wiggle room in the future if I need it (cropping is really the only regret I’ve had about saving in JPEG). Because of this one new feature alone, I’m glad I waited till now to switch to Lightroom.

    So thanks for the article. This was a big help in me making peace with Lightroom as I try to step into this new photo managing world.

Leave a Reply

Comments are moderated. Abusive, inflamatory, and/or "troll" posts will not be poasted. If your comment doesn't show up, it may have been blocked by the spam filters.

Follow me on twitter for updates on when new comments and articles are posted.

(required)

(required, will not be revealed, sold, or distributed)