Concept: Using rolling shutter for digital IS

I make no claim that this idea is original, but I wanted to write it down anyway.

Take an ordinary CMOS digital camera sensor with a rolling shutter and since it only costs a few cents, add accelerometers and gyros.

Now, instead of merely capturing each row from the sensor into a finished image, record each row from the sensor separately, along with its location on the sensor and the time the capture started and ended; record this and the accelerometer and gyroscope data.

When you've got enough processing power available (in the camera if you have it, on a general-purpose computer if you don't), use [the integral of] the accelerometer and gyro data to estimate the orientation and position of each row within a larger image, refine it by using any overlap between rows to align them. Once all rows have an initial placement, refine the placements repeatedly until the best fit is achieved. Paint all the rows onto a larger canvas, trim the result down and interpolate at any spots you didn't cover. ALE and hugin have some excellent algorithms for these steps, though there are probably some additional tricks and wrinkles when all the input images are 1d.

You can even incorporate multiple complete scans of the sensor in this way, adding information without increasing smearing due to motion of the camera. This information can either be used to reduce sensor noise or increase output resolution using ALE algorithms.

You could also do your best without having accelerometers and gyros: use any simple motion estimate (such as zero between the first and second rows, and the previous alignment-based estimate from row N-1..N for row N..N+1) as input to the alignment step.

Finally, an altered CMOS sensor that reads out in an interlaced fashion (or other "non-linear" fashion) would be even better, because the first few rows read out would be spread over the sensor, "pinning down" rows that are read later. For instance, if the initial stride is 16 rows on a 1024-row sensor and the full exposure time is 1/8s then the basic geometry of the scene is captured in just 2ms (relatively little motion being possible in this time) and subsequent rows read will have the opportunity for overlap with both rows "above" and "below" during the initial alignment phase.

Entry first conceived on 13 March 2012, 14:26 UTC, last modified on 14 January 2013, 15:03 UTC
Website Copyright © 2004-2024 Jeff Epler