Thursday, 21 January 2010

JPG and RAW processing swings and roundabouts

Last weekend we had a large family gathering which I was asked to photograph (since I used to do events and other commercial type photography in a past life). I picked my G1 as my main camera and thought I'd toss in my film EOS for a couple of shots (which actually I really wanted to get). As mentioned in my previous blog post there are some situations where a larger capture format (or among digital folks "full frame") just works better. If I had a 5D or other full frame DSLR I would have used that for the event not the G1 for just those reasons.


However this is not the main reason for the post, this time its (mainly) about the processing pathways of RAW and JPG.

I know that a number of professional photographers just use JPG as their main capture and shun the RAW workflow. Others do use RAW capture and often they are also people who have a team who work with them in post processing that for them. After this weekend I really see why that is.

Now normally I use JPG on the camera, picking RAW for just a few shots here and there, so as a result I'm not normally processing 300 images from an event so much as hand tuning five or six (perhaps) at a time.

Previously my workflow (don't you love that buzzword) has been to open the RAW in Adobe Camera RAW within Photoshop and then operate on the file. This was clearly not going to work for this situation or I'd still be chained to the computer working on images (as if I had nothing else to do).

what I mean by quality photography

One of the things I expect in a series of images is consistency,from one shot to the next (in the same situation and lighting) I expect;
  • consistent colour
  • consistent exposure

... to me nothing looks worse than faces and suits changing colour casts, contrast or tone.

It just looks like crap and is (part of) what separates the dopes from those able to deliver a good product to the client.

Now back when I got my first film camera with Av on it (we called it aperture priority back in those days) I soon found that my exposures were all over the joint and I wondered
"what the hell is the benefit of this automation if it gets in the way more often than not"
It didn't take me long to be lamenting my previous "match needle" light meter built into my (manual) 35mm camera (yes, it was an OM-1) and the way of using that I had developed. Heck compared to a hand held light meter that system was really handy automation.

I quickly learned to plop the camera in manual and just take meter readings occasionally (as lighting situations or lenses changed).

The next issue was proofs ... if I took a roll of film to a commercial lab I'd get a strip of 6x4 inch proof prints along with my roll of film. These were consistently exposed and reflected the exposure of my film (you can guess I was using negative here cos you don't bother with this on chromes / slides / E-6), by which I mean that each print was made with the same exposure from the negative that the previous one had been ...

Now this is totally unlike what happens if you take your images to a typical mini-lab at the supermarket / chemist / corner store ... there every bloody print is made by a machine which knows nothing about the previous print and thus tries to "optimize" each print. No matter how consistent YOUR SHOTS are the prints go back to looking like grandma's patchwork quilt (lovely as it is) not anything you'd be proud to show to anyone (if you have any shred of professionalism).

the real question

So, how do I process my 300 odd RAW files to meet my criteria of quality consistent results.

Well you've essentially got two choices use a product such as Lightroom (ohh, don't you have that yet? well .. just whack another couple of hundred down and add another few hours of your time getting to know it too) or you can use a free product such as dcraw (which is a command-line utility that actually is what is used as the engine for many RAW converters anyway).

Now, as it happens, I have my camera set so that RAW files contain an embed medium res JPG image that would be what the camera would have produced if I didn't use RAW. That is to say all the settings are applied to it including stuff like white balance, the "curves" the camera prefers and any lens corrections which it might apply (some cameras are doing this for you now). Naturally dcraw allows me to extract these JPG's (which then load nicely into something like irfanview) to quickly evaluate everything.

So far so good.

Now, because I set my white balance manually for each shooting situation, I can then use this value to tell dcraw how to apply colour to the RAW when domosiac occurs (eg when making an image I can do something with).

So armed with a few parameters I set it loose to convert my RAW files into 16 bit TIFF's for tweaking in Photoshop.

This is where I hit my first hurdle .... you see, dcraw is also a simple binary, so it can't make use of multi-threading. This means that if you happen to have a dual or quad core machine then its all going to just hog the one processor. On my box (1.4Ghz per core) this was taking about 2 minutes per image.

Fark (said the crow) ... 300 x 2 minutes is TEN hours.

So I killed this and thought about it for a bit.

Its worth mentioning at this point that each image as a TIFF at 16 bit is about 70Meg ... so I hope you've got some spare space on your disk for this cos thats 21 Gig right there.

So I though Ok, maybe I don't need need 16 bit ... after all, I didn't think I'd be making huge edits to the files, so 8 bit would be likely to be enough. To cut it down even more, I decided to go through and weed out my stuff to process only the files I wanted to convert.

All in all this brought processing down to just 2.5 hours

Time for going and doing something else while my machine crunches all this for me.

Getting back to Lightroom for a minute, a fellow I know who uses it says that he can use it to manage which images he wants to render to JPG and apply similar parameters to "groups" which were in similar lighting conditions ... thus it doesn't matter if the images were shot with the right light balance as I used above ... he can leave it on Auto White Balance (shudder) and ignore those parameters (while still getting lens corrections, something that dcraw can't do).

Ok, now two and a half hours later I still need to open my images in Photoshop and make any changes such as sharpening or colour tweaks (because I already know that my sensor does not make reds and yellows just where I like them for skin tones and I have not used any system to profile my sensor ... more about that topic here for example).

I have enough experience with Photoshop to know to sort this out on one image, then create an action to apply that across a batch of files, so I again don't need to sit there like a monkey (thank God).

This took me about 15 minutes to sort out, and then another 2 hours for the machine to grind away applying this to my subset of converted images.

As it happens I have a few computers in the house (... don't ask) so while I was allowing the PC to do this, I used my other PC to scan some of the film images on my LS-4000.

Thank God this has ICE as my negatives naturally came back from the "lab" (no professional labs round this neck of the woods) with fingerprints, streaks and some fine scratches running the entire length of the roll .. and no, its not my EOS that did that!

A few things came from this process:
  1. image processing of my negatives was a bit longer than that of the RAW
  2. all of the film images looked great, had better snap and better colour straight off of the film (heh ... he says)
  3. I am so glad I am not locked into scanner hell anymore, cos doing this in strips of 6 really takes my time for attended operations.

This last point makes me think (at first)
"why the hell don't I have a roll film adaptor to scan 36 at a time, unattended"
... which then reminds me of the time I got all the film processed and scanned by a lab with a Noritsu scanner at the time of developing. Dam they were great!

Seriously I got very high quality 6Megapixel images from my film for $15.00 per roll (processing and burning to CD) at the time of processing. No dust, no scratches and I still got the Negs if I wish to use them for optical enlargements or other uses.

Sitting here now I am thinkin ... Man I'd pay $9 extra per roll to save all this time!

So now I'm thinking
"has digital let us down, or did we shoot ourselves in the foot with it?"

Aren't these things meant to make our lives easier?

OK ... the computer beeps to let me know the final batch is completed and so I give them a quick look over and then send them off (via FTP) to a service to have them printed and posted back to me (man, there is no way I'm going to pay hundreds for a good desktop printer to have that to fuck about with ... inks to go dry, calibrations to perform...).

So a full day (and some of the evening) of my attention was needed to produce this result, so that the proofs can be posted off to my grandmother, and the CD's burnt for other relatives to get their own prints made if they wish.

Sitting here on the train to another town I'm thinking about all this process and wondering about all this.

I can really see why many professionals prefer to use JPG and save themselves all this hassle, but the problem is that having looked at the results of processed RAW files, I definitely got better contrast and colour out of the RAW images than I'd have got with the JPG.

For example, this is one I took in the beginning of winter (part of my 56 series)

Camera JPG

RAW lightly processed

Its not immediately clear what's different about those two images, but if you open them up into separate tabs and look around you'll see (as I've identified below) that I have much better shadow details in the RAW sourced image, while actually getting better contrast in the stone area where there was low contrast to begin with.

and this wasn't even an optimized (carefully exposed right) image for RAW, I've easily got another 1/3 stop of headroom that I didn't make use of (cos I was in a hurry and when it clips it gets ugly fast). Heck I've even added some curves to the JPG to make it less contrasty (and chosen a camera profile which was less contrasty ... and this isn't even contrasty by Australian outdoor landscape photography standards!

options and choices

Now looking at my weekends images (you know, the usual drama black suites and white stuff on white stuff) the camera JPG's however had superior noise processing (with little discernable loss in quality), so the in-camera processing there is superior to what I can do here (or I could spend more money and try Noise Ninja or something).

Another advantage of the in-camera JPG processing is that it is done in parallel to the capture process (and not to mention done by dedicated signal processing hardware), so by the time I get finished shooting I have finished products (more or less) ready to roll (and they take up less storage space than the RAW too). So assuming that you are able to get just what you want from the JPG first go then you're better off with that in many ways.

The question is, "should I just go with the JPG's and by not seeing the results of the RAW and never know what I was missing be happy, or should I spend the money (oh, I love this "Digital is free after you buy the camera" shit) on Lightroom yet another product (ohh, and need to upgrade my OS and hardware probably) to get a better work flow and stay chained to the desk??

Hmmm ...

kirstiHauttalle2Well just maybe, but when I look at the results of the negative film workflow (which just happend to make the loveliest prints) I'm left wondering ... you see the Negs produced images which (in the exact same conditions) have better shadows and held hilights better(in snow for gods sake) than the JPGs (by a mile) and in some cases the RAW too *(mainly in situations where I let the camera meter and didn't do it manually resulting in missing head room on the RAW).

Images such as this one from the 35mm negative turned out to be the sweetest images in the group (and didn't really need much more effort than the digitals)

If I could reliably get my C-41 processed and turned into scans as I have had by the above mentioned Noritsu process I would probably never shoot another wedding or funeral in digital again (and I know a few wedding photographers who've said just that before too).

So with many professional labs going or gone under (leaving us with no choice but to go for a full digital process, not a hybrid one which has film capture then digital for editing and printing), I'm wondering if Digital is having some hand in killing jobs and contributing to skills losses in our community (I dare you to find a professional photo finisher or high quality master print maker for me in your area).

Actually there are quite a many professional wedding photographers who swear by Cameras such as the Fuji S3 and S5 digital bodies for exactly the reason that the sensors capture as much as negative does (but of course are still stuck in the digital post processing mud like me).

For me as a photographer I now need to know heaps more about the process than I ever did before. Once I just needed to make a well exposed negative (or chrome), now I need to do almost everything in the process with the actual printing being done by machine from the file I send it.

So sure I love my digital camera, I have actually developed myself as a photographer in using it to enable me to take way more experimental images than ever I would have with film.

Clearly I can't say I don't want digital, but perhaps the hidden costs are spread further than we thought.

I recently saw a discussion where wedding photographers were being hit with a "per gig price" of $500 ... hah! Go out on site for 8 hours, provide all the gear, go home (cos you won't be affording an office at those rates) and process your ass off. You can hear the clients now ...
"well you don't have any film costs anymore".
Its no wonder that photographers like this are taking over the industry.

Ahh I remember the good olde daze, I'd shoot 12 rolls of chromes for an event, wander down to the proLab in town drop them off ... go grab a bite to eat, pick them up one hour later and sit down with the gig director to sort out in 20 minutes on the light table what was going to be put into the projector that night. Digital has made it all so easy now ... as long as you don't want the quality to go with that side order of new software.


Charles Maclauchlan said...

I'm still chuckling.

It's funny how many digital photographers and how few film photographers I know. When I made my "Escape" from film I quickly saw that I needed more computing power and a program to keep track of the thousands of (mediocre) images I could produce "For free." Aperture, like Lightroom allows me to sort images and also quickly quasi-convert from RAW but it also encourages me to shoot even more...which points to the need for even more computing power.

I had intended to sell my film cameras and buy a D2X. Thank goodness I didn't. The money I've "wasted" on film saved me that expense and also the upgrade to D3 expense which would have quickly followed. Perhaps not a waste after all.

I have a couple of labs (mail) that do a fine job of scanning and return images from 6MP to 14MP dependent upon my needs and the prices are quite reasonable. Don't know about Finland or Australia though.

When I do shoot digital I use the RAW + JPG setting and find that 90+% of the time the JPGs are great. The other 10% are when I was too lazy or inept to properly set the white balance.

Noons said...

"For me as a photographer I now need to know heaps more about the process than I ever did before. Once I just needed to make a well exposed negative (or chrome), now I need to do almost everything in the process with the actual printing being done by machine from the file I send it."

That indeed sums it all up, doesn't it?

Go to any online photography forum, and I mean any. What do we find? Folks discussing the subtleties of sensels, mega-whatsits, dpi-vs-ppi-vs-lpi, colour profiling(!), blahblahblah.

None of them has ever had to rely on their photographic work for the next meal.

They are the same who used to populate "letter-to-the-editor" not so long ago, claiming such and such a lens could resolve so many lines per mm at 13.876mm from the centre...

Meanwhile real photographers were out there taking photos and sending film to labs.

Photographic content? Zippo!

Art content? Not much more...

Any wonder I keep feeding my "lab" with rolls and rolls every week? And that's just the colour and slide stuff: then there is the b&w developed at home...

Mind you: I'd rather be taking even more images than spending the time I do on the scanner/printer workflow.

But at least the last thing in my mind when going through that workflow is the number of angels one can fit on a megapixel...

Not to say I don't like digital. But I'm far from spending the time on such detail as the online forum folks!

Yes, we need to know a bit about all those things. But I'd rather spend my time learning how my dslr can best produce that balance of colour I'm seeing!

Nowadays, I'm sending more and more files to print at the lab and often asking them to scan&print my film rather than just develop: it gives me time to go play with the cameras, which is what I wanted in the first place!

If only we had more Noritsu boxes around...

Soundimageplus said...

It is a bind. All that processing and editing. Like Charles whenever possible I will use the jpg if its right in terms of exposure.
I have a scanner and 10's of thousands of transparencies but it sits dormant most of the time.
Mainly because after the scanning, which takes long enough, the whole process of editing begins again.
I haven't shot film since 2001 and don't really miss it, but the digital workload sometimes takes on monumental proportions.
I shouldn't really complain, it is my job after all and it can't be all fun all the time.

obakesan said...

I too certainly don't miss scanning film either (and I remind myself of this often enough by doing it still). I just wish I could hand this over to someone else to do ... like a lab?

I guess that I'm perhaps glossing over the problems I've had with some labs in the past too ... but I can't be sure as it was too long ago now to be entirely certain its not just my perception.

I guess that the only real choice is to use the latest tools (now that I know them well enough) shoot more carefully (something I think I do anyway from growing up in film) and be quite ruthless in the edit and selection process from the thumbnails.

I've got a mate who was a school and portrait photographer, he has an air conditioned room full of negs (lives in Brisbane where fungus on negs is an issue) for his "obligation holding time" for reprints. Sleeving and cataloging that is no less a drama than digital asset management.

so its still swings and roundabouts ;-)

When someone makes a CCD with the capture of the SuperCCD and gives a nice 16 bit range then I'm sure I'll not look back to Negative and miss much at all ... certainly not colour negative.

Noons said...

Almost missed the link to 27b/6!
Love that site: it's just so spot-on!