Sony A7 RAW file weirdness

For some time I've eyed the Sony A7 as a full frame camera (on which to use some of my Olympus OM 35mm film lenses for a start) which is not as much of a "Behemoth" as the 5D and similar SLR cameras are.

However its come to light over the years that the A7 RAW format is somehow not the "right stuff" and as its also my preference to avoid Out Of Camera JPGs I'm sort of "stuck" on dropping a thousand bucks (Australian < > US Dollar sucks) on even a used one to sus it out.

Fortunately I have a friend on the Net (thanks Yu-Lin) who does have one (for more or less the same motivations) and he's kindly sent me some files for analysis.

preamble

Firstly I suggest the reader toddle over to this site where the issues of posterization are shown and explained and also have a look over here at the examples that were perhaps the first reportings of this.

In this post I want to present some findings which I've made with a few tools like dcraw and rawhistogram (downloaded from here).

Using this pair one can actually make a spreadsheet of a summary the data captured in the RAW file and see what's there. I think its actually pretty much what DxO do when they rate the sensors of cameras. However there are clearly differences because DxO rates the sensor output at 24.8 bits and the EOS 6D at 23.8 bits ... so I remain out to sea there.

I am not entirely certain that what I'm seeing isn't some sort of artifact, but if it is, then its an artifact that will encumber most RAW processing engines.

Lastly I recommend reader to have a quick look at my older post on RAW and White Mans Magic over here. If nothing else its a good starter into the analysis method used.

Note however that the output of the Analog to Digital Converter (ADC) is just the assigning of analog numbers to digital numbers and there is no intrinsic "scale" in that. Meaning that the output does not represent the count of photons only the number assigned to a count of photons (recalling that a sensor pixel analog value translates more or less to photon count). So a longer range just means more accuracy in the quantification of that output signal not that the sensor has a higher threshold for capturing light.

findings

So, firstly Yu-Lin sent me this image:

which he exposed to avoid clipping ... my initial question was to see how well the shadows held up and what could be pulled out of the RAW file. So he sent it to me and I processed it into this:

which I'm sure you'll agree shows a remarkable rescue of the shadows (no discusson on if that's better, I just wanted to see). In particular I used Photomatix to do the tone mapping, and its normally a good test of what horrors lay in the lower levels to reveal themselves when pushed out. For instance this is a more detailed look at what I got from the shadows:

which I'm sure you'll agree is quite impressive noise wise.

So far so good ... its all looking fantastic to my eyes (and I'm not actually worried about anything that follows here really)

But when I tossed the RAW file into my processing system I found the following interesting points. To be clear the processing uses dcraw with the following switches  -D -4 -t 0 -o 0 -v
So there is no "stretching" of the data. Presented below are essentially the histogram counts for each quanta. As its not "demosiaced" we have Bayer R G G and B channels.

where does it start?

The data starts recording quite late in the binary buckets ... meaning that quite a few bits are wasted in the low end. In fact the data starts showing only at quanta 128

fair enough ... so they decided to get rid of "floor noise" ... My G1 (from that earlier blog post) started recording a bit earlier at about 36, but not as evenly representing in the channels ... meaning that there was data in Green before Red or Blue (in that order).

where does it end?

So this led me to then look at "where does the data set finish", which was where I found interesting things. The data sort of changed recording at quanta 1523, to recap 16 Bit can record up to quanta 65535 if it was presented with that level from the sensor. Often it is not (fully 16  bit, and here is where we get to the possible method DxO uses) and ends somewhere else much lower (in quanta number, using less of the available significant bits).

For instance in the case of my G1 it ended in a clump around 3989 and with my 10D  around 4007 ... which is a long way short of 1523

However what came to my attention as I was paging down the data, was that after position 800 the Sony started skipping data points, writing data at every second quanta not every quanta.

Interested I aske Yu-Lin for a file that contained over exposed (clipped) data for analysis and found things got even more interesting.

Like that file it started "skipping" every 2nd data point, but then went to recording at every 4th data quanta when it got down far enough.

... then at every 6th

then every 8th ... which went on till the data just stopped at quanta 4157

like what the fuck?

So while at first glance it would seem that the Sony A7 records data from quanta 128 through to 4157 (which seems a really good showing) the reality is that Sony have not got much data in there at all. Probably over half the quanta are missing (I'm too slack to count).

This may go towards explaining why DxO rates the sensor as having more bit range (or data range) than the Canon when the Canons I've seen do not employ such tricks to "compress" the RAW files (by having them more or less empty).

Why this may not be the end of the universe...

Anyone who's done any work processing images should know that there is a non-linear compression applied (or curve) which has a toe and a shoulder. This results in the lower end of the data range being streched (putting a challenge onto the quantisation of it) with the top end being compressed (or squeezed together). Let me recap this (from another source):



So perhaps Sony has had a go at "optimising" its data knowing what's going to ultimately happen to it to try to get "more into less".

In my history of working with Sony products (from Audio days onwards through MiniDisk and into Computers and digital audio recorders) I would suspect that Sony is doing something like this.

I know that with my Sony PCM-M10 recorder they employed a super scruffy MP3 encoding which resulted in high end truncation .. but one could ask was it "obvious" listening or only something one can pick up in analysis.

Well a bit of both. If you happen to post process images (as I also post process sound) it did show up as being faintly less clear, but only if post processing the MP3 ...

Is this a "valid" technique on the part of Sony? Well that's your call ... what I'd like them to do is at least make it more transparent. Because as it is they seem to fudge greater data depth on sensor reviews than perhaps they deserve ...

the 2010 shootout: a brief review of their review

Show me an artist who knows nothing about their materials and I'll show you a poor artist.

So when people wonder just why the hell do I waste my time understanding my media I don't mind, clearly they don't know much about how to work with anything.

As a photographer we rely on media which now more than ever is changing rapidly, as a photographer I want to be able to make the best "go" at capturing something fleeting I can. Knowing my materials is critical to this.

So from time to time I examine what I have and what's around and try to see if I would benefit from this or that and how.

One of my friends recently pointed out this comparison to me

http://www.zacuto.com/shootout

which is a more detailed analysis over a greater variety of digital SLR cameras and a couple of films.

These guys did a world class job, working with the worlds best and showed their results to top cinema photographers. I encourage you to watch that video ... its a little 'long winded' but worth 30 minutes of your time.

Some things I observed from it, which I think seem to coincide with many of my own observations over the last year of examining my Panasonic G1 and film.

Now the purpose of their analysis (being cinematographers) was on colour rendition and contrast response to lighting (shadows and highlights).

I don't want to regurgitate their findings here, but I'd like to point out some things which I noticed and was not clear among the points which they brought out

First, looking at the the two film stocks (which of course are their reference as cinematographers)




Quite an amount of detail is visible in the patterns of the glass bricks behind the bath tub (why these guys always choose chicks in tubs is beyond me), but look at that part of the digital images.






just blown out ... (although it appears nicely in the reflections on the water), maybe thats what people are used to accepting with digital, and maybe its just a detail ... after all what artist would be interested in the details....

Everyone in their audience also thought that the results were really good. Putting aside for a moment this issue; if you were a small budget producer you could buy a 5DMkII for less money than you could hire an ARRI camera for a 3 week shoot.

That has to be attractive for any director / producer.


Then they did another scene. The topic of discussion was mainly about the light bulb ...




clear an "normal" in both of the film shots ... but in the digital's it looked washed out and nasty.





Every digital camera tested behaved like this ...

This is exactly consistent with my previous findings of how well film handles high key items like this:



Film:


Digital


as well as other examples, even HDRI



All very interesting.

They discussed briefly in their "brain storming" discussion this concept and called it "blooming", I have my own theory of this. I feel that this is not only related to the sensor, but is related to the combination of anit-alias filters / IR filters / blah blah

I mean think about it for a moment.

The sensor on a digital camera has a number of layers over it for a variety of purposes (including protecting it).

Film is simply naked at the time of exposure, so there is nothing over it to flare up ...

More, film has this concept called an anti-halation layer which is specifically designed to prevent this happening at the deeper layer of the film as it's illuminated and perhaps glows internally.

I'd also suggest you look at the contrast of the images in the high key light of the naked bulb.

The digitals all show some sort of flare overall, which I'm suspicious is related to reflections from that shinny surface of the sensor and reflecting back to the back of the lens ... which is shinny too.

I feel there can be no way around this and its an inherent advantage of film, unless we can make sensors naked and matte in surface.

While your watching that video, I recommend that you keep an eye out for the way that the makeup on the model shows shinny cheeks more on the digitals than on the film and the ugly way that her face blows out as she approaches the bulb.


in comparison the film remained beautiful and faithful in the face of lighting adversity....


lastly I'll suggest you take some screen shapshots (as I did) and examine them yourself, because there are differences in the final balance that their colour guy got. The GH1 Panasonic was rated least hightly (although they were impressed) yet interestingly they managed to get its white whiter and a higher level than the other cameras tested.

Check the levels on the bulb in all the shots, only the film managed to make white and had the best control. Now this could be as a result of their WWW compression, or it could be something else (like effort needed to keep things under control in post production).

This was a very interesting test and there is much more information to be had that was directly presented. Perhaps they all yakked about it, but it would be nice if more of it was presented.

RAW and WMM: what's he building in there

Most photographers have by now heard of RAW, however WMM is a TLA for "White Mans Magic". Meaning something complex that you can't understand. Having just written about my explorations of RAW and Exposure I thought I would put up a little more information which I have been finding in peeling back what is in RAW files (using some tools and dcraw)

As I mentioned before, with film determining exposure was a reasonably straight forward affair (although still seems to have and continue to mystify many).

Essentially you just used a tool to measure your film density and you could understand from that if you were under exposing or not.

The graph here is the density for negative film.

Of course you needed to measure this, and before we had digital tools this may have been tricky. But since the mid 1990's this has been easier as Scanners as it happens make good and simple densitometers.

With Slides however its much easier, you just look at it. If it looks dark you got it under exposed and if its washed out then you over exposed it.

If it wasn't for the fact that slides were notoriously difficult to get prints from I reckon more and more people would have used slides. As an aside here I'll mention that while slides look like the sorts of thing that is projected at the movies, the cinema industry actually uses negative to shoot with and the "prints" are what you see on the projectors. Cunning.

An important point is that the quantization of analog data from the sensor is not the same as the absolute values that the sensor is producing. Unlike film, where density is determined by light levels (up to a saturation point) the Analog to Digital Converter (ADC) is tuned by the camera maker to match the sensor (pixel) output. So while we have numbers we don't really know what they correspond to without making known source measurements. More bits is better up to a point, but after that only if more bits means greater ranges of analog readings.

Well, anyway ... I know how digital cameras record their data on the sensor, which I covered in that previously mentioned article on exposure:

The RAW data is the Linear Distribution, mapping that to an image is where the Gamma Corrected Distribution comes in.


Of course you don't have to map it according to that, and many don't. Although maybe most cameras do, that's changing with makers addressing tone mapping concepts in camera.

Well, anyway before I get carried away, lets get back to analysis of my images.

Below is the data set from the image which looked like this


now, look at that histogram and there is no data in the very black and no data in the very white. Suggesting no blow outs. Now look at this data table below. Data seems to start from about the 27th level in the count...



and sort of run out at about step 3988. Remember this is greater than 8 bit data and as it happens these data levels fit within a 12 bit range (binary value 111110010101 if you wanted to know)

Looking carefully you can see that data representing high levels is trickling to a standstill as we approach 3988. Now, lets look at the next image in my bracketed sequence


which is just showing clipping, but interestingly still has a gap where nothing is really black. Looking at the data from that image we see that indeed low level data starts a little later ...



we are starting to see a little clustering happening around 3972 level ... The bunches of data show more and more levels are approaching the clipping point.

But its funny to me, its almost like its being scrunched together before it hits the mathematical limit. This starts to look like some sort of compression algorithm is being applied to it before the analog signal gets digitized. This sort of signal processing is actually common in the Audio industry, even before digital we used 8 to 1 or 10 to 1 compression leading into the more ugly infinity to 1 compression (limiting) to prevent tape saturation.

Lets look at the camera generated JPG image where we have really obvious clipping


and a slightly extended area of no data, looking at the data from that RAW image the data doesn't start appearing until about level 120 (although note that green is well under way there)



and we have the same bulb of numbers appearing at the end and a soft fall into a hard limit of 3989 again.

So to me it seems that the hard limit is 3989, and nothing goes over that (or reaches it) and that as data approaches it it gets scrunched up in a bundle rather than just clipped with a smack.

The data for this is available here for those who are interested to see the data.

Because I am not able (yet) to look at the actual raw data, I am employing dcraw (which is the most reliable source I know for converting RAW, and is used by dozens of software vendors to make their products work) to decode this and then using another tool to explore that, I can not be certain this is not an artifact of dcraw. What I do know however is that this is not confined to my Panasonic G1.

Here is a sample from my 10D, a much older camera, and one from the generation where people really do talk about "hard clipping".




we see a few things of interest in this dataset:

• the data starts much later, with nothing before 128 (a significant number in digital)
• all the data channels start much earlier
• the two green channels are not equal at their cut off point and the blue cuts off earlier(perhaps making ugly images at that point).
• it goes a wee bit higher in the range than the Panasonic

So, like all research this leaves as as many new questions open as it answers my initial one. I'd really like to know if

• what seems to be low level truncation of data in the 10D results in the reputation for "clean files" that the Canon has;
• if the ugly high lights of the Canon are related to the scaling of data;
• if the ugly noise seen in the Panasonic when tonemapping is related to the gentle trail in of data at the start
• if all this means that we can just use camera JPGs more on the newer cameras, not needing to rely on tricks and tools to get better images?
• are we loosing real data by not compressing more effectively, perhaps cameras doing log encoding of the data in the first place would get us away from some of the noise (you know, optics have flare and stuff ... we don't all do astrophotography) in the system and have better access to the high count (bright light) data.

Certainly with this last point I've found that the benefits of RAW processing in the case of the G1 reveals less than it did with my 10D ... seems like things might be getting easier.

... just as long as you get the capture right in the first place.

stay tuned for more as I find it out.

---

PS I thought that incase anyone had not thought about this, I'd take this time to demonstrate what a histogram is. If you remember this image from my previous post...


well if we take the data from that spread sheet and in Excel plot a graph averaging its values over the range it covers, we get this.



so now you perhaps see your histogram in a better light ;-)

exploring RAW and Exposure: getting the right mix

Its often said that digital cameras are like slide film, in that exposure needs to be controlled within 1/10th of a stop to make the best out of the media.

To be honest that's really just not possible with most gear.

Why do you need this control anyway? Well perhaps you don't, but if you do want to get the best out of a shot, avoiding ugly blow outs (that you may not want) and minimizing noise in your images (again, that you may not want, though some like noise) then getting your exposure 'bang on' is the goal.

As I wrote yesterday, this can be hard to achieve with the in camera tools such as the histogram and the light meter. So today I thought I'd take a 1/3 spaced EV series and pick the one that clips and then the one just before it (1/3 an EV) to demonstrate the differences.

This is the JPG from the camera (set to daylight white balance, and taken in daylight), note that its slightly clipping.

This is the one that's just 1/3 EV less.

and that histogram would not make you think you only had 1/3 of an EV to go ...

Looking at a linear decoded TIFF using dcraw we can indeed see that the red channel is clipping here.


but at 1/3 EV less we're well away from clipping.


to see just how far away I used the level tool in Photoshop with its ability to show the clipping graphically (hold down the alt key as you slide the white level to the left and you'll see)

I adjusted it to get the same amount of clipping in both images and as you can see (in the figure beside) I had to essentially strip off 76 steps or levels.

This is significant, as for people who advocate expose to the right it makes it very hard in practice to actually achieve this without bracketing in small steps to get this.

Due to the nature of the linear capture in digital cameras by avoiding over exposure you do run the risk of having far to little data to make the best image your camera can make. Bruce Fraser makes this point well in a white paper published by Adobe which (at the time of writing was still) is published here. Essentially the point is that applying the Gamma to make the image look as it should you stretch the shadows (and thus posterising them) further than you need to.



thus the less of your image that's down in the darker areas the better your image will look when its ready to view. That RAW image above looks murky and dull, but when you apply a little gamma to it it starts to shine more.


so with gamma applied (by me by hand to my taste in this case) we still don't get clipping and get a better image.

As Bruce suggests in that above article:

You may be tempted to underexpose images to avoid blowing out the highlights, but if you do, you’re wasting a lot of the bits the camera can capture, and you’re running a significant risk of introducing noise in the midtones and shadows. If you underexpose in an attempt to hold highlight detail, and then find that you have to open up the shadows in the raw conversion, you have to spread those 64 levels in the darkest stop over a wider tonal range, which exaggerates noise and invites posterization.

Correct exposure is at least as important with digital capture as it is with film, but in the digital realm, correct exposure means keeping the highlights as close as possible to blowing out, without actually doing so.

Since he wrote that (and as sadly he has passed on he will not be updating it) perhaps the cunning tricks that camera makers can do with their in camera JPG engines has increased, making picking that clipping point harder. In the case of my Panasonic G1 I can see that if it clips in the histogram then it clips in the RAW, but you'll need to experiment with that on your camera.

Tools such as ACR and LR have hilight recovery tools (so too does dcraw) which can make it easier to get better images from your raw files by perhaps blowing out the high lights a little and then setting up curves in LR to bring your images back to how you like.

I hope that by reading this it has helped you move a little closer to getting the best quality image you can out of your digital camera.

The RAW we've just worked through (saved as a JPG of course)


the JPG as it came out of the camera


which one do you prefer?
is it worth your effort?

now those are questions I can't help you with :-)

tools for the job: why don't we have them?

I was reading a post by a friend of mine who has written of what he's found in looking at the RAW files produced by his G1 via a tool which has no bling, and just does the job of converting the RAW data into an image. He focuses in that post on the extra pixels which are available around the edges of the image (normally 4000 x 3000) which can be had by using dcraw (however wrapped in funky GUI apps) as the underlying tool for conversion of RAW files into images.

NOTE: after a comment from Don below I thought I should add that I do not really recommend you turn to a tool like dcraw for your general RAW conversion, as tools such as LR and ACR are more suited to production processing and will be faster. However they do mask much of what is happening happening in the process, so for those who seek to explor and understand what is there in the RAW file I encourage you to play with dcraw.

This got me to thinking about the other issues I have with digital cameras, and that is exposure.

Exposure is something with digital which has driven me nuts, and comprehending it without access to RAW files is like comprehending your exposure on film looking only at prints. Thus this article is about determining exposure and is also a complaint about the tools we have at our disposal for determining a proper exposure.

Lets start with this image, I took it a few weeks back when we had lovely soft -20°C snow that softly lay in big flakes over the garden (some 30cm deep).



I've tweaked this image a little, so I thought I might present what I had started with. Below is the JPEG image as the camera made it into photoshop.

looking at this I did not really use expose to the right (which I am certain is the best strategy for minimizing noise on) but I was a little concerned with the possibility of blowing out a channel.

Out of interest I converted my RAW file using dcraw to make a 8 bit TIFF file with standard gamma. Now understanding gamma is important, nay critical to really getting digital imaging. If you come from a history of printing with enlargers onto paper you may begin to think of this as choosing a grade of paper ... not perfect but that'll do.

The result of the "standard" conversion to a TIFF gave me this ...

which looks quite different, and quite acceptable to me (close to my personally tweaked one). However it doesn't take much to notice that the blue channel is going to be showing some clipping in there looking at that channel (and I'm sure I can guess just where).

This of course got me wondering what was really lurking there in the RAW file because I didn't see anything like that in the JPEG ... so I converted it using a LINEAR gamma. This basically applies no corrections. Now keeping in mind that digital cameras produce numbers to represent levels (0 is totally black and 255 is totally white), gamma basically applies a maths function to the starting number and then writes a file with the changed number as the new value for the pixel.

Eg

• 1 might become 1
• 10 might become 15
• 100 might become 180
• 180 might become 255

stuff like that.

So, using a LINEAR gamma shows us what we had to start with ...


bloody hell ... I call that under exposed if you don't.

Its a reasonably well known (and bitterly complained about) fact that digital cameras histogram is based on the JPG data and not the RAW data. So while we have the cameras light meter for determining exposure it really is not sufficient to work out where clipping is (should you want to expose right).

Of course Film is a different creature to digital sensors, especially negative where you can bravely shoot with the sun in your face and still not get a blow out.

This is of course because film has a shoulder which you can see appearing in this spec sheet of the characteristic curve of neg ... the maker (Fuji in this case) cuts the response curve while its still mostly linear but you can see its starting to shoulder off over there to the right hand side.

It gets difficult in film because the linearity of the responses is uneven after this graph stops ... but that's not my topic today ...

So with that thought in mind I thought I'd attack this RAW file and put my sort of preferred gamma for this subject and this lighting. You can see that its really steep and rolls of gently with a taper at both ends (not in a hard angle).


This results in (what I think) is a pleasing rendering of the scene and in keeping more with what I saw with my eyes when I was there.

This of brings me to further support that you should never apply harsh machine like maths to a subtle thing like an image and certainly the light meter and histogram are inadequate tools for determining the right exposure or capture of the image in the first place.

If you download and examine that image (click to get a 1600 pixel version) you can see that noise isn't dreadful, but remember this was 100ISO ... it should be nearly perfectly noise free.

The "near enough is good enough" approach may suite some people but just as scene brightness ranges vary and need different gamma applications to obtain the best results, so too do we find histograms as an exposure determination tool inadequate because of the Gamma applied to them. Particularly as I've found that the greatest strength of digital is the subtle renderings it can yield in soft light, its sad that we can't maximize this with the tools on the camera.

I'm tempted to take my spot meter with me ... which if I'm going to do that I might as bloody well use my LF camera with all that hassle.