scantools - a few tools to help with film scanning

[Ted Baker]

I have been writing some software to help with film scanning to do the following:

1. Speed up flat bed scanning, so a full platen of negatives can be scanned in 1 pass. So for example I can scan 24 35mm frames in 10 minutes at 2400dpi, on my old Epson 4990.

2. Software to invert both B&W and colour negatives, using the same method that is used in Cineon/DPX. i.e. just about any movie you see that was shot on film in the last twenty years.

It is an open source project so free to use and extend etc. Some of colour code needs some revamping, but I am using it to good effect for my current B&W work.

For a B&W negative the result is roughly what would be printed in grade 00 paper, so I find it a good starting point.
A colour negative results in low contrast image, that should only require a contrast adjustment and some linear colour balancing.

It in its current form it probably only usable if you have some basic programming/development skills, you need a bash shell and Imagemagick. PM me if you need some help, but it is a start.

https://github.com/laurencelumi/scantools

[ericantonio]

This looks awesome. Been busy but I have a v700 hooked up to my ubuntu. I usally unplug and stick it onto my Win10 for some scans like 35mm batch scans which Vuescan can't do. Well, it can but so much easier on windows.

[Pali K]

Thank you for this Ted. Looking forward to trying it out and will let you know how it goes.

[Pali K]

Hi Ted,

Good news is that I got it to work under Ubuntu and have a few notes for you from my testing. I am using Gamma 1 scans from my drum scanners to test and here is what I am noticing. Large images fail to convert and I have noticed anythign over 10MB seem to fail for me all together. It creates at small 386kb file and never finishes. For files that do convert, I am getting very dark conversions even with DMAX set to 1.

Here is the negative I am using in case you want to test it yourself.
Download Original TIFF File (http://www.netsoft2k.com/Docs/Media/Pictures/Scans/Calibration/SM%20Neg%20Test%20-%2016%20Bit%20Tiff.tif)

http://www.netsoft2k.com/Docs/Media/Pictures/Scans/Calibration/SM%20Neg%20Test%20-%2016%20Bit%20Tiff.jpg

Hope this is helpful!

Pali

[SergeyT]

Hi Pali,

try it with "-dmax auto".

Hi Ted,

I have a copule of notes too (Mac OS X):
1. It is a good idea to have all the variables used in the script initialized at the beginning . Example "exiftool".
2. The script did convert a 128 Mb file for me but would not anything above that. It simply stops using CPU and does nothing.
3. The effect of conversion looks similar to that obtained in Photoshop with either "Subtract" or "Divide" of a layer filled with unexposed border color. The converted image blacks are at r=0, g=0, b=0 (clipped) and white point requiring significant tuning (with Auto Color (Enhance per Channel Contrast; No "Neutralize the mid-tones")). Visually and practically (math and algorithms aside), in my opinion, a simple combination of 2 layers (Invert) + (Curves with Enhance per Channel Contrast; No "Neutralize the mid-tones") provide for more flexibility in balancing the collors as neither Blacks nor Whites appear to be clipped.

Best Regards,
SergeyT

[Pere Casals]

Ted, this is an amazing set of tools...

Thanks for sharing it !

I'll read a bit the code...

[Ted Baker]

Hi Pali,

I am not sure why it does not work. If you send me a PM, with details of

#convert -version
and maybe a log etc.

The colour code at moment does does not have make any adjustments yet for printing density. i.e. Cineon/DPX values should represent how the paper sees the negative instead of the human eye.

I am focusing on using at the moment on about 50 odd rolls of HP5, I can scan a roll in under 20minutes on my flatbed which is big improvement. So I put the colour calibration code aside for a little bit...

Anway,

I did this

#invertscan -c1 -fb 0.914557,0.602957,0.480313 -dmax auto pali.tif
Expirmental code
Normalised values are:
max min
Red: 0.901625 0.311894
Green: 0.591104 0.179248
Blue: 0.447105 0.0554971
using calculated FB: 0.914557 0.602957 0.480313
log() values are:
base shadow DMax
Red: 0.0387892 0.00618483 0.467204
Green: 0.219714 0.00862244 0.526832
Blue: 0.318476 0.0311148 0.937254
TODO need to do check of all three maxes
using auto calculated dmax
dmax: 0.937254
scale: 17952.6
1 image files updated
inverting: pali-P.tif

which results in this:

182163

there is no attempt to balance anything at this stage but the intensity ratios should all be linear or a gamma of 1 except where the film the toe and shoulder compresses these ratios.

I use a curve like this to represent the paper.
182165

and balance the image using linear controls to get some thing like
182164

Eventually I like to incorporate the code directly into Rawtherepee etc.

For BW I have found it very effective.

[Ted Baker]

Hi Pali,
2. The script did convert a 128 Mb file for me but would not anything above that. It simply stops using CPU and does nothing.


Its possible your installation of imagemagick is resource constrained. Try

#identify -list resource

this will list any constraints





3. The effect of conversion looks similar to that obtained in Photoshop with either "Subtract" or "Divide" of a layer filled with unexposed border color. The converted image blacks are at r=0, g=0, b=0 (clipped) and white point requiring significant tuning (with Auto Color (Enhance per Channel Contrast; No "Neutralize the mid-tones")). Visually and practically (math and algorithms aside), in my opinion, a simple combination of 2 layers (Invert) + (Curves with Enhance per Channel Contrast; No "Neutralize the mid-tones") provide for more flexibility in balancing the collors as neither Blacks nor Whites appear to be clipped.


No clipping ever occurs. In the file the darkest information sits at 0, for convenience, and the brightest sits at the maximum, if you use dmax=auto. i.e it will adjust for the density of the negative (-dmax=auto) to use all of the 16bit values available, or you use a set max density value that you choose. For the straight line portion of the characteristic curve the intensity ratios should then be linear in relation to the original scene. i.e a value that is 10 times brighter than another, should just as same 10x ratio as in the original scene. This means the image will appear low contrast on a monitor, print etc.

I am reasonably pleased that it is useful for BW, the single pass scan of entire platten for roll film, speed up the laborious process of scanning a film of a flat bed, and results in a scan that I find easy to work with.

I am still working out a few things with the colour side of things, the main thing that is really outstanding is to calibrate the scanners spectral sensitivity, to match the spectral sensitivity of the print stock which is how the film was designed to work in the first place.

Plus take a few photos from time to time...

[Pali K]

Ted, that are some very impressive results with just simple commands. I'll spend more time with it this weekend and let you know how everything goes. I have decent programming skills as well and will try to study your code and ImageMagick commands to see if I can help out with this project.

Pali

[Pali K]

I think getting the film base values right makes a huge difference. Here is a quick conversion using Ted's commands above followed with simple curve and color adjustments.

http://www.netsoft2k.com/Docs/Media/Pictures/Scans/Scantools%20Ubuntu.jpg

[IanBarber]

How does this compare with ColorPerfect results for black and white

[Ted Baker]

How does this compare with ColorPerfect results for black and white

That was tool was what got me started, the maker published a few papers, which I used to implement a version of colorperfect. But I was not satisfied and I got a little carried away. :o For BW the end result will be be very similar, keeping in mind that your final destination is always at the control of the person developing the image. Colour film is far more complex probably 100 times so.

But from a work flow perspective if your using one of those miniature cameras, where stuff comes in rolls it has a lot of advantages from a time perspective. If for example you would prefer to use something like lightroom instead of photoshop etc. (neither of which I use BTW).


see if I can help out with this project.

Pali

Thanks for the encouragement. Yes I am pleased with the results, as soon as get a chance I will document it better, and explain how it works etc...

If anyone has a status-m densitometer, that they might be kind enough to measure some film for me I am keen to trial a calibration technique...

[SergeyT]

If I understand how the code works, the next step for the color conversion might be to calibrate the system by :
1. Making series of exposures in day light of either a white card or color checker on your favorite film with bracketing from -2 to +5 stops in .5 stops intervals.
2. Developing the film with normal development
3. Making scans with fixed scanner settings that would not clip the data on either end
4. For each scan calculating the DMax per each channel individually based on 2 values - one from unexposed border and the other one from the white patch. RGB values in each of patches (border and white) should be "averaged" before DMax calculations to avoid errors
5. Hopefully the calculated by your code individual Dmax values for each of R G B channels will allow to create some sort of a profile for a combination of film type, film exposure, scanner setup. maybe (if the Dmax values dont vary much based on exposure) the resulting DMax values could be approximated to make one profile that is "insensitive" to film exposure level.
Then the dmax values could be further translated into per channel scale and applied during real conversion to each channel calculations accordingly.

At the end of this excercise to validate the code (if the scaling was calculated correctly) the conversion of frames made in #1 should result in unexposed border being close to RGB (0,0,0) and white patch being the brightest with RGB values close to R=G=B.

If that approach works for one emulsion type it surely can be extended to other emulsions as well.

[Ted Baker]

If I understand how the code works

The inversion code is based on Cineon which is an analogue to analogue system, or film to print system, that you can still buy and is well documented. The system is closely based on how print film is designed in first place.

In summary the steps could be summarised as:

1. A raw linear scan of the negative, with the exposure set such that the film base is reasonably close to the maximum value the sensor can record. I use approx 90% and set this for the film stock.
2. A matrix should be applied to the sensor readings to better match the spectral sensitivity of the print stock. The spectral sensitivity of the sensor is designed such that it can be used to approximate (usually via a matrix or LUT) that of human vision which in turn is not the same as the print stock. Status-M measurements that are used to measure negatives which don't actually match sensitivity of the print stock, are a good proxy because they only measure a small part of the spectrum, and with the correct scaling should be close enough.
3. The values are then converted to density values of the negative
4. The density values are then stored as linear intensity values. The Dmax=auto just ensured that 16bit space is not wasted
5. A matrix is applied to convert the linear intensity values that are intended to printed by CMY print stock, to a known colourspace. (XYZ)

[Ted Baker]

see if I can help out with this project.

Pali

Pali,

I have been testing my calibration logic, with some success.

If you have a common negative film and scanner combination you like to use, I would be happy to "attempt" to build a custom calibration. I would need a shot take of macbeth/x-rite colour checker taken in daylight (or flash) exposed at your normal ISO etc, scanned in the same manner as the image you provided earlier.

Its won't be fast turn around, so a just a shot that you might fit in with your normal work flow.

184179

[Pere Casals]

I would be happy to "attempt" to build a custom calibration.

Ted, led me suggest a way to nail a client calibration.

From the same image you take the standard conversion and a conversion that likes the final customer. You feed 3D LUT Creator with the two images to obtain a 3D conversion LUT (that can be used in a Ps adjustment layer). Several LUTs can be made for different kinds os subjects, portrait, landscape...

Probably different films may require specific 3D LUTs.

IMHO 3D LUTs it's what address that kind of requirements. Just an idea. I'm in the course to explore well your software

[Steven Ruttenberg]

Interesting. Where can I read up on LUT? Not sure if I would ever implement it, but worth the read. I am currently scanning the rest of my 4x5 about 60 frames from the year and seeing my progression from photographer with all kinds of issues/mistakes to getting better. Both at scanning and taking images. Anyway, interesting program. I'm not a coder so, not interested in that, but rather the final product and how much control one has in using it. The more the better.

[Ted Baker]

I finished an update, that increases the reliability of the auto cropping of a full platen of film from a flatbed. So that only a single pass of a flatbed is required.

For example the following raw scan from my old Epson 4990, scanned so that there is gamma of 1

https://i.ibb.co/tq8NL0y/kodak400.jpg

Some example commands would be:

#scantool frames kodak400.tif output_directory
this results in 24 raw scans

#scantool both -fb 0.4196,0.2745,0.2039 kodak400.jpg output_dir

this result in 24 inverted scans that have a gamma of 1,

This is also works for reliably for roll film

[Steven Ruttenberg]

Nice. Good thing I only do 4x5 and up on flatbed. I use a Nikon Coolscan 4000 for 35mm.