Sony A7 RAW file weirdness

For some time I've eyed the Sony A7 as a full frame camera (on which to use some of my Olympus OM 35mm film lenses for a start) which is not as much of a "Behemoth" as the 5D and similar SLR cameras are.

However its come to light over the years that the A7 RAW format is somehow not the "right stuff" and as its also my preference to avoid Out Of Camera JPGs I'm sort of "stuck" on dropping a thousand bucks (Australian < > US Dollar sucks) on even a used one to sus it out.

Fortunately I have a friend on the Net (thanks Yu-Lin) who does have one (for more or less the same motivations) and he's kindly sent me some files for analysis.

preamble

Firstly I suggest the reader toddle over to this site where the issues of posterization are shown and explained and also have a look over here at the examples that were perhaps the first reportings of this.

In this post I want to present some findings which I've made with a few tools like dcraw and rawhistogram (downloaded from here).

Using this pair one can actually make a spreadsheet of a summary the data captured in the RAW file and see what's there. I think its actually pretty much what DxO do when they rate the sensors of cameras. However there are clearly differences because DxO rates the sensor output at 24.8 bits and the EOS 6D at 23.8 bits ... so I remain out to sea there.

I am not entirely certain that what I'm seeing isn't some sort of artifact, but if it is, then its an artifact that will encumber most RAW processing engines.

Lastly I recommend reader to have a quick look at my older post on RAW and White Mans Magic over here. If nothing else its a good starter into the analysis method used.

Note however that the output of the Analog to Digital Converter (ADC) is just the assigning of analog numbers to digital numbers and there is no intrinsic "scale" in that. Meaning that the output does not represent the count of photons only the number assigned to a count of photons (recalling that a sensor pixel analog value translates more or less to photon count). So a longer range just means more accuracy in the quantification of that output signal not that the sensor has a higher threshold for capturing light.

findings

So, firstly Yu-Lin sent me this image:

which he exposed to avoid clipping ... my initial question was to see how well the shadows held up and what could be pulled out of the RAW file. So he sent it to me and I processed it into this:

which I'm sure you'll agree shows a remarkable rescue of the shadows (no discusson on if that's better, I just wanted to see). In particular I used Photomatix to do the tone mapping, and its normally a good test of what horrors lay in the lower levels to reveal themselves when pushed out. For instance this is a more detailed look at what I got from the shadows:

which I'm sure you'll agree is quite impressive noise wise.

So far so good ... its all looking fantastic to my eyes (and I'm not actually worried about anything that follows here really)

But when I tossed the RAW file into my processing system I found the following interesting points. To be clear the processing uses dcraw with the following switches  -D -4 -t 0 -o 0 -v
So there is no "stretching" of the data. Presented below are essentially the histogram counts for each quanta. As its not "demosiaced" we have Bayer R G G and B channels.

where does it start?

The data starts recording quite late in the binary buckets ... meaning that quite a few bits are wasted in the low end. In fact the data starts showing only at quanta 128

fair enough ... so they decided to get rid of "floor noise" ... My G1 (from that earlier blog post) started recording a bit earlier at about 36, but not as evenly representing in the channels ... meaning that there was data in Green before Red or Blue (in that order).

where does it end?

So this led me to then look at "where does the data set finish", which was where I found interesting things. The data sort of changed recording at quanta 1523, to recap 16 Bit can record up to quanta 65535 if it was presented with that level from the sensor. Often it is not (fully 16  bit, and here is where we get to the possible method DxO uses) and ends somewhere else much lower (in quanta number, using less of the available significant bits).

For instance in the case of my G1 it ended in a clump around 3989 and with my 10D  around 4007 ... which is a long way short of 1523

However what came to my attention as I was paging down the data, was that after position 800 the Sony started skipping data points, writing data at every second quanta not every quanta.

Interested I aske Yu-Lin for a file that contained over exposed (clipped) data for analysis and found things got even more interesting.

Like that file it started "skipping" every 2nd data point, but then went to recording at every 4th data quanta when it got down far enough.

... then at every 6th

then every 8th ... which went on till the data just stopped at quanta 4157

like what the fuck?

So while at first glance it would seem that the Sony A7 records data from quanta 128 through to 4157 (which seems a really good showing) the reality is that Sony have not got much data in there at all. Probably over half the quanta are missing (I'm too slack to count).

This may go towards explaining why DxO rates the sensor as having more bit range (or data range) than the Canon when the Canons I've seen do not employ such tricks to "compress" the RAW files (by having them more or less empty).

Why this may not be the end of the universe...

Anyone who's done any work processing images should know that there is a non-linear compression applied (or curve) which has a toe and a shoulder. This results in the lower end of the data range being streched (putting a challenge onto the quantisation of it) with the top end being compressed (or squeezed together). Let me recap this (from another source):



So perhaps Sony has had a go at "optimising" its data knowing what's going to ultimately happen to it to try to get "more into less".

In my history of working with Sony products (from Audio days onwards through MiniDisk and into Computers and digital audio recorders) I would suspect that Sony is doing something like this.

I know that with my Sony PCM-M10 recorder they employed a super scruffy MP3 encoding which resulted in high end truncation .. but one could ask was it "obvious" listening or only something one can pick up in analysis.

Well a bit of both. If you happen to post process images (as I also post process sound) it did show up as being faintly less clear, but only if post processing the MP3 ...

Is this a "valid" technique on the part of Sony? Well that's your call ... what I'd like them to do is at least make it more transparent. Because as it is they seem to fudge greater data depth on sensor reviews than perhaps they deserve ...