Technical testing DSLR cameras
by Lorenzo Comolli

Hurried reader? Skip directly to the test results below!
Introduction
As many of you, I'd like to get the most from my cameras in low light circumstances, such as under a dark sky. In the past I've tested many of my cameras, and also of some of my friends. Here are the results.

Measurement protocol
To get reliable measurements, I've used the standard method for measuring gain, i.e. the linear interpolation of points in a signal versus variance graph. The protocol is well described in this page of Christian Buil, a reference for this kind of tests.
I've captured a series of image pairs of a flat field, with exposure times approximately between 6 s and 1/8000 s, using a standard lenses (i.e. 50 mm or 18-55 mm). I've used full resolution raw format. I've analyzed only the green channel. I've selected a 100x100 pixel crop near the part of image with minimum gradient due to uneven illumination or vignetting.
I've measured gain and read noise at each ISO full stop from the minimum to maximum setting of the camera (i.e. 100 to 3200 ISO), and -whenever possible- with the addition of 1000 ISO and 1250 ISO as a representative of 1/3 stops. I've tested also extended sensitivities, when available.
To get the measurements from the raw files, I've used a custom made Matlab script.

Main results
The main results I've obtained are:
  • Gain (inverse gain). Unit of measurement: electron/ADU. This value shows the way an electron in a pixel is converted to a number (ADU, Analog to Digital Unit). Please note that gain is in truth the inverse of the gain (inverse gain): common use has consolidated this error. So if you'd like to increase the sensitivity you'll have to lower the (inverse) gain.
  • ISO at unity gain. Unit of measurement: ISO. This is the ISO sensitivity with an unity gain. Using sensitivities below unity (or below 0.5, the limit is not sharp) should not gain better results, but this is not true because the read noise can decrease if the amplifier has a lower noise respect to the acquisition electronics. Higher values, while comparing cameras with the same number of bits, are synonymous of better results even if not always.
  • Read noise. Unit of measurement: electron. This quantity is fundamental for low light subjects. Increasing ISO sensitivity will usually decrease the read noise, and so this improves the results on faint objects. But if increasing the ISO sensitivity over a certain limit will not lower the read noise, then using this high sensitivities is absolutely not useful, and also detrimental because the dynamic range will be reduced.
  • Dynamic range. Unit of measurement: decibel.This is the ability to gather in the same image faint and bright objects, without respectively loosing them in noise or saturating. In these test the dynamic range is measured only approximately because this is not of fundamental importance in astrophotography. Usually the best dynamic range is obtained at the lower ISO setting, but similar values can be obtained also for higher sensitivities.
  • True sensitivities. A true sensitivity is obtained by changing the analogical gain in the sensor electronics. Another possibility is to "simulate" this sensitivity using a simple software that multiplies the measured pixel values. In example, a camera can obtain a 3200 ISO image by setting on the sensor an analogical gain half of the one at 1600 ISO, or by setting the gain at 1600 ISO and then doubling the measured values. Clearly the software manipulation will not get more information, and so this is of no use (except if you shot in JPG format, not the case for most astronomical imaging). Not only the software manipulation has no advantages, but it has disadvantages, as a cut in half of the dynamic range (equivalent to -6 dB).
Results and list of the tested cameras
Here are all the tested cameras (ordered by introducing year). Click for detailed results. Here are only a few main results
Camera
Year
Lowest read noise [e-]
Highest advised ISO
Best approx dynamic range [dB]
ISO at unity gain
Notes
Sony A7s
2014
0.85
25600-51200
82.5
4529
Impressive sensitivity and low noise; but raw data is not raw and maybe some denoise is applied for the highest sensitivities
Sony A6000
2014
2.16
3200-6400
75.9
865
An improved successor of NEX-6 mirrorless, better for night imaging
Sony NEX-6
2012
2.45
800
77.1
904
First tested mirrorless; not good for night imaging
Canon EOS 5Dmk3
2012
1.96
12800
65.9
500
Most sensitive camera (up to test date, together with 6D)
Canon EOS 6D 2012
1.81
6400
68.7
575
Most sensitive camera (up to test date, together with 5Dmk3)
Canon EOS 60D RAW
2010
1.99 1600-3200
64.7
205
Tested, as usual with all cameras, with full raw format
Canon EOS 60D sRAW
2010 4.04
1600
79.8***
846
Same test but with sRAW format
*** not true because of this format
Canon EOS 60D mRAW
2010 3.48
1600
78.9***
794
Same test but with mRAW format
*** not true because of this format
Pentax K-x2009
5.33*1600
69.0
887
Auto noise reduction above 1600 ISO
3200 and 6400 ISO are not true
*considering only true sensitivies
Canon EOS 5Dmk2 2008
3.15 1600
67.3
409
6400 ISO is not true
Canon EOS 450D (modified with a Baader filter) 2008
3.12
1600
62.5
235

Nikon D902008
3.61*1600
73.7
1298
Non linear quantization!!! At mid brightness only 10 bit are true.
Denoise on raw data at all sensitivities above ~15% of dynamic range.
Auto noise reduction above 1600 ISO
*excluding noise reduced ISOs
Canon EOS 20Da 2005
3.52 1600
63.7
1284

Canon EOS 5D (modified with a Baader filter) 2005
4.80 1600
63.9
1713

Canon EOS 350D (CHDK firmware) 2005
4.15 1600
62.6
1075
No real advantages of using CHDK
SBIG STL 11000 (an astronomical CCD)
~2005
9.59 -
75.3
-
Best true dynamyc range of any DSLR tested up to 2013.
...more coming... do you want your camera tested? Contact me!






Comparisons and notes
Comparing cameras of different epochs and sensor size shows the main trend in technology. While improvements are present, they are not so large as one can expect: I assume the reason is mainly the mega-pixel fever that seems not to stop. I'm confident that a newer camera with the same mega-pixel count of an older one would have improved much more, with much better low light behavior.
In example, the dynamic range has improved of about 2 dB from the 350D of 2005 to the 60D of 2010. And the APS-C sized 60D was able to reach nearly the same dynamic range of the full-frame 5D, a camera of 2005.
60D has improved very much the read noise respect to 350D, but the merit seems not to be due to the 14 bit analogue to digital converter (ADC) (350D has 12 bit) because in example the 5D, with a 12 bit ADC, has nearly the same dynamic range of 60D.
But due to the smaller pixel size, 60D is not so much more sensitive respect to 350D, as also the detailed test of 60Da showed.
The knee of the read noise curve is in nearly all the tested cameras at about 1600 ISO (except 6D and 5Dmk3), so any sensitivity increase above this value has nearly no advantages on the night time shots. So the large ISO values in 60D, such as 6400 and 12800 ISO are only lark-mirrors...
On the contrary, the test of 450D shows that maybe a higher sensitivity (maximum native is 1600 ISO) can improve the read noise, as the read noise graph let imagine (the trend seems to decrease).
An improvement in newer cameras is the increased number of true sensitivities. In 60D all the sensitivities are true, and also the extended 12800 ISO seems to be true (even if this is of no use). Also in 350D and 450D all the standard sensitivities (100-200-400-800-1600 ISO) are true; the eventual other sensitivities possible thanks to CHDK are not true. Also the 3200 ISO of 5D and 20Da are not true (however they are correctly named "extended" sensitivity by Canon).
A latest note about 6D and 5Dmk3, renowned among photographers because of their low light sensitivity. The test evidence indeed a very good behavior, with a knee in the read noise at 6400 and 12800 ISO respectively, that is much better respect e.g. the 1600 ISO of nearly all other cameras (including 5Dmk2).
The drawback of Canon cameras seems to be the dynamic range. Even at the lowest sensitivities the dynamic range is not very good. The best one is 6D with 68.7 dB (corresponding to 11.4 EV, or 11.4 bit). This let understand the usefulness of 14 bit converters (instead of 12 bit on older cameras).

Here are my comments about non-Canon cameras I've tested.
The first is Pentax K-x: looking at the results, I would never buy such a camera because of the many anomalies in the test results, such as 3200 and 6400 ISO sensitivities that are declared true, but they are not, and because of the read noise that does not decrease very much going from 200 ISO to 1600 ISO, giving nearly no advantage in terms of lowering noise when increasing ISO.
Another one is Nikon D90. I was really impressed (negatively) of this camera behavior. Even much much worser than the Pentax K-x. Many undesirable are present: the raw data is not raw and by many aspects. Tree main problems are the cause of all problems:
1. non linear quantization: raw data are stored by loosing quantization resolution in brighter pixels, leading e.g. to 10 bits instead of the declared 12 bit at mid brightness. This is also associated with some kind of noise reduction on bright pixels.
2. black level clipping: no offset is added before sensor data acquisition, as is known from the ABC of data acquisition. Nikon ignores deliberately this obvious rule imho for a deprecable reason: to fool the image benchmarks looking at the read noise in bias frames. This way nearly half of the noise is obtained. I've developed a method so that my read noise measurements are not fooled by this behavior.
3. noise reduction (NR): above a certain ISO, a noise reduction algorithm is applied to raw data! Even if it is turned off by the user. This started in the first Canon-CMOS vs Nikon-CCD era, when Nikon sensors have large dark signal (i.e. hot pixels); to hide the problem Nikon added a firmware algorithm to cancel the hot pixels. Unfortunately this removed also stars!!! Now that Nikon converted to CMOS, there is no reason to keep NR!
As a conclusion: NIKON RAW IS NOT RAW AT ALL!!! This camera (and I suspect all Nikons up to 2014) cannot be used for serious astroimaging. Nikon, please, give us true raw data.

Here are my comments about astronomical CCD cameras I've tested.
The only one I've tested is SBIG STL 11000. Since sensitivity is fixed, the only useful information is the dynamic range, quite large, and the linearity, good up to 40000 ADU.



HTML Editing and Publishing by Lorenzo Comolli. Email me at comolli@libero.it.
Back to Main Page