In The Mirrorless Conundrum, we discussed our initial thoughts on mirrorless interchangeable lens cameras -- especially Sony’s role in driving the MILC market segment, in no small part due to image sensor technology innovation.  We reached the conclusion that, based on our personal use cases, the time for us to adopt mirrorless technology had not yet arrived, and we therefore compared digital single lens reflex camera offerings from Canon and Nikon relative to our personal interests -- concluding that Canon still best met our needs, albeit by a much narrower margin than when we switched from Minolta to Canon in 1999.

But, now that Canon and Nikon have entered the mirrorless fray with the EOS R and the Z6 and Z7 respectively -- and granting that MILCs perform very well for landscape and other non-action applications -- it’s time to revisit the mirrorless question.  And, in the process we discuss the direction that digital camera sensor technology is evolving as well as the capabilities that are thus enabled.  We’ll also examine the fact that Canon, once a leader in digital camera sensors, has now fallen behind in some, although not all, areas -- and what that means to Canon DSLR and mirrorless users.

A Brief Sensor History

In September of 2002, Canon introduced the 11 megapixel EOS 1Ds, the first professional grade full frame CMOS DSLR on the market. Most prior DSLRs, by all manufacturers, were APS-C crop factor cameras and were largely based on CCD technology.  But, CMOS had technological advantages in power consumption, chip integration and fabrication costs,¹ and today CMOS sensors² are used for most traditional photographic applications.  Having been an early adopter, Canon pushed to the front of the DSLR market, a position that it has not relinquished from a sales perspective -- although it is a mixed story when it comes to leadership in sensor technological innovation.  Sony has moved ahead of Canon in dynamic range and sensor readout bandwidth with its Exmor line.  Nikon, through its partnership with Sony for sensor manufacturing, shares that lead.   A notable exception is Canon's dual pixel autofocus, a highly useful feature for video.

Source credit:  https://www.edmundoptics.com/resources/application-notes/imaging/understanding-camera-sensors-for-machine-vision-applications/

Image credit:  https://www.edmundoptics.com/contentassets/1e85009206814710980a5dea8810781f/fig-1-ucsmva-lg.jpg and https://www.edmundoptics.com/contentassets/1e85009206814710980a5dea8810781f/fig-4-ucs.gif

One of the key innovations of the Exmore line is on-chip analog-to-digital coversion.  On-chip ADCs increase dynamic range and reduce noise by removing the noise susceptible external analog path from sensor chip to external ADCs.  Canon has trailed in this design, to the detriment of its own sensor performance.  Undoubtedly the cost of a new fabrication line calls for a significant capital outlay, possibly an unpleasant prospect for a board of directors watching profits and market position soar from past investment successes.  In fairness, it must be added that Canon has the consensus largest, most varied and optically excellent lineup of lenses in the industry, undoubtedly a prime area for continued corporate research and development investment.  And, Canon did finally introduce on-chip ADCs with the 1DX MkII.

In addition to on-sensor ADCs, Sony's Exmor sensor technology also incorporated column parallel readouts.  The column parallel design greatly increases the readout speed possible from the light collecting elements of the sensor.  This unlocks all sorts of high performance enhancements, including faster frame rates, higher megapixel counts at those high frame rates, faster video rates at higher and uncropped resolutions, more capable sensor-based autofocus, blackout-free electronic viewfinders, etc.  A comparison of the two approaches appears in the Framos article, What is Sony/s Exmor Technology Anyway?

Source credit:  https://www.framos.com/en/news/what-is-sony-s-exmor-technology-anyway

Image credit:  https://www.framos.com/media/image/8b/61/4e/csm_exmor-figure1.jpg

The reasons why Canon has fallen behind are hotly debated on Internet forums.  A frequently offered explanation is that Canon deliberately restricts camera features to entice buyers to move upscale in the Canon line, at higher consumer costs and thus higher corporate profits.  While there might be an element of truth in this supposition, it does seem a bit overly simplistic.  One suspects most companies follow this practice to a greater or lesser degree.

Another possibility might arise from Canon's having long been a sales leader based on past technological leadership and the continuing superb optical quality of Canon lenses.  Many users may simply have such a large investment in Canon gear, particularly in high quality lenses, that they are reluctant to make the financial sacrifice (and to settle for a lesser lens selection) that changing brands would entail.  Add to this the fact that Canon gear performs quite well despite not always being at the top of the specification heap (there's an old engineering aphorism, "better is the enemy of good enough"), and the enticement to switch simply isn't compelling for many Canon users.  Absent a decline in sales Canon has little incentive to invest in better sensors.  Or, so goes the theory.

Perhaps a more skeptical view might suggest that Canon has allowed its sensor technology to stagnate as it has exploited its position of initial leadership and market dominance (Canon’s current ILC market share is over 40% worldwide, twice its nearest competitor) to rake in immoderate profits, to the detriment of investment in continued sensor technology evolution.  A possible example is the fact that Canon was very late in its introduction of sub-500nm circuit design.  According to one source³, Canon APS sensors are produced at 180nm and full frames at 300nm.  On-chip ADCs were also a very late Canon addition, a technology that Sony, and by extension, Nikon, had had for years prior.

None of these explanations fully takes into consideration Canon's sensor R&D position compared to its competitors.  They all imply that Canon's lag in sensor development is solely attributable to factors and choices that Canon fully controls.  The reality may be quite different, as a German analyst's take on major camera company sensor R&D suggests.⁴  (Translation here.)  As the referenced article points out, Sony holds approx. 40% of the total image sensor market, including smartphone sensor sales likely totaling in the billions.⁵  This volume has given Sony a distinct advantage in funds available for sensor R&D.  Canon, on the other hand, mostly manufactures its own sensors -- exceptions are largely limited to lower end point-and-shoot models.  As a result, R&D funding is limited by the lack of sensor sales comparable to that of Sony.  (That does seem to be changing -- Canon has begun offering sensors for external sale.)  From the reference above,

• Nonetheless, it becomes abundantly clear that Canon, as a total company, ceased to make a significant contribution to [sensor] research and development around the period 2005 to 2007.
• The consequences of the savings in R & D only become apparent years later and gradually. This is hardly noticeable to many observers in the short term. But after ten years, it should be clear to every analyst today. - The photographer is already clear on the basis of the now produced cameras.
• Even if it sounds hard: De facto, Canon has already ceased to be a technologically leading group in the field of photography [specifically digital sensors] about 10 years ago.
• The main problem is that Canon needs to do a lot of research and development on its large sensors (mostly APS-C and full-frame), while Sony can do this on very small sensors for the smartphones. The latter is easier, cheaper and faster to perform. Subsequently, Sony adapted this only for large sensors. Canon is missing not only the sector of small sensors for testing, but Canon are also missing the huge economies of scale and profits from that sector. Therefore, I think it is out of the question that Canon . . . can ever achieve the same sensor development efficiency as Sony. It follows, in turn, that Canon ultimately has to spend a lot more on R & D for the same image quality. That's a strategic disadvantage. [Emphasis added.]

An additional factor is that the still photography market has been in a long period of decline, partly due to the rise of smartphones and partly due to market saturation as film cameras have disappeared and the initial wave of customer buy-in to digital cameras recedes into the past.  All these developments have converged to leave Canon in a trailing position financially with respect to available sensor R&D funding.

Sony Exmor R Backside Illumination

Initial Exmor sensors, like CMOS sensors from Canon and others, were front-side illuminated, i.e. the wiring traces were on the front, light collecting side of the sensor, reducing the sensor area (fill factor) available for photon capture.  However, in 2008 sony announced the Exmor R, a backside illuminated sensor (BSI), wherein the wiring traces were behind the light sensitive photodiodes, thus making almost the entire front area of each pixel available for light collection.  The increased fill factor available for photon collection is illustrated in Sony’s 2012 announcement of the Exmore RS BSI stacked sensor (discussed next).

 Image credit:  https://www.sony.net/SonyInfo/News/Press/201208/ik0fha000000gfvq-img/figure1.png

Image credit:  https://www.sony.net/SonyInfo/News/Press/201208/ik0fha000000gfvq-img/figure2.png

By moving to a BSI design, the Exmore R thereby increased the level of circuitry that could be packed onto the sensor and also the size of the light sensitive photon wells.  Sony’s evolution was, in part, fueled by its leading role in manufacturing mobile phone camera sensors, a market not only lucrative in sales volume but also one requiring ultra high efficiency in the tiny sensor chips used in phone cameras.  Perhaps Canon’s much more limited sensor manufacturing base could simply not generate the corporate revenues needed to stay ahead of Sony -- or perhaps Canon simply chose to invest elsewhere.

 Image credit:  https://www.framos.com/media/image/68/36/3d/exmor-figure5.jpg

Sony Exmor RS Stacked BSI

In 2012, Sony announced Exmor RS technology, a stacked sensor configuration for MILCs.  The design included an additional processing layer bonded to the back of the image data collection segment of the chip.  This allowed even more of the processing pipeline to be performed on chip, with resulting reduced latency and higher data movement bandwidth.  The design was further enhanced with a third layer incorporating buffer memory in 2017.

Source credit:  Quora/Dave Haynie https://www.quora.com/BSI-sensor-is-better-But-why-expensive-DSLR-Cameras-uses-CMOS-sensor-instead-of-BSI-sensor/a>

Image credit:  https://qph.fs.quoracdn.net/main-qimg-8c87af080e060aa94ec9ee063c2bb5ba

The latest version of the Exmor RS incorporates a layer of DRAM buffer memory as a means of facilitating internal sensor data transfer.  From Sony’s 2017 3-layer stacked Exmore RS announcement:

Image credit:  https://www.sony.net/SonyInfo/News/Press/201702/17-013E/ik0fha000000f60h-img/img01.jpg

Image credit:  https://www.sony.net/SonyInfo/News/Press/201702/17-013E/ik0fha000000f60h-img/img02_02.jpg

Stacked BSI sensors confer a number of advantages.  Because more processing is done on-chip, there is less opportunity for noise to build up as is the case during external data transfers, resulting in lower noise and higher dynamic range (DR).  Furthermore, processing latency is reduced.  Sony sensors have reached the point where they are essentially ISO invariant -- that is conveying the ability to under expose a scene at a low ISO value and raise exposure in post processing without impacting image quality.⁶  In addition, the use of stacked sensor technology is understood to be one of the enablers for global electronic shutters, a feature with many benefits, as discussed in the next section.

What Next?

Tremendous progress has been made in camera sensor technology.  However, there is yet more to be done.  One of the foremost problems to be addressed next is the video phenomenon known as rolling shutter, the skewing of video images as the shutter traverses the image frame and as data is read out sequentially.  The favored technological solution to rolling shutter is global electronic shutters -- the simultaneous exposure and subsequent highly parallel readout of the entire sensor at a very high refresh rate, thus not only solving the rolling shutter phenomenon but also obviating the need for mechanical shutters.  Architecturally, global shutters require a far greater degree of parallelism, an example of which is shown in the diagram below illustrating in-pixel noise cancellation in a new Panasonic global shutter design.

Content source:  https://news.panasonic.com/global/press/data/2018/02/en180214-2/en180214-2.html

Image source:  https://news.panasonic.com/global/press/data/2018/02/en180214-2/en180214-2-3.jpg

In addition to removing rolling shutter artifacts, global shutters enable higher frame rates, higher resolution, lower noise, wider dynamic range and higher quantum efficiency.⁷  Sony is well situated with its stacked BSI Exmore RS sensor architecture to develop global shutters.  Perhaps the ultimate expression of parallelism applied to the problem of global shutter operation was introduced by Sony in early 2018.  The product, a 1.46MP stacked BSI CMOS sensor with global shutter, incorporated an ADC with each pixel.  Power consumption is a problem for such massive parallelism, perhaps explaining the low resolution of this initial offering.  However, it seems to point the way to the future.

Content source:  https://image-sensors-world.blogspot.com/2018/02/sony-presents-gs-sensor-with-adc-per.html

Image Source:  https://4.bp.blogspot.com/-PSf3n28-GFo/WoMFi18L5fI/AAAAAAAAU64/SIM-alkVNTcnjPEvqk8Vjq-DTmGMaQg2ACLcBGAs/s1600/Sony%2Bimg01.jpg

and:                   https://2.bp.blogspot.com/-Q-p11Irev-E/WoMFucvDm2I/AAAAAAAAU68/qLKr5X8zKP42Ms1Tb2l2fv7EHLbDhYEEACLcBGAs/s400/Sony%2Bimg02.jpg

Rolling shutter video artifacts continue to characterize Canon’s DSLR and MILC offerings, a problem likely attributable to insufficient parallelism and on-chip processing in the image pipeline.  Clearly it is in Canon's best interest to upgrade its sensor technology if it is to maintain market leadership.  Canon's status and future direction are covered in the section that follows.

Where is Canon? 

Canon eventually did move to 180nm technology and finally began putting ADCs on-chip with the introduction of the 1Dx Mark II in 2015.  Prior to the 1DX MkII, Canon’s sensors lagged Sony and Nikon in base ISO dynamic range by 2-3 stops.  Canon still trails by about one stop, but the gap has been substantially narrowed and as a practical matter the difference is probably not significant in most applications.  In fact, high-end Canon cameras do exhibit excellent noise characteristics.  However, although Canon has upgraded its obsolescent 500nm process technology to 300nm and 180nm, with 130nm available for some products⁸, nevertheless implementation problems remain. The EOS 6D Mark II exhibits the same relatively poor dynamic range as previous Canon sensors.  The implication of this is that either Canon is forced to reuse old sensors to control costs or that there is a manufacturing bottleneck with respect to latest generation sensor technology.

Quite possibly Sony patents are constraining Canon’s design space for BSI and stacked sensor technology.  Perhaps as a result, Canon has concentrated on dual pixel autofocus (DPAF), a highly effective approach to on-sensor phase detection autofocus (PDAF), a feature that has given Canon a strong position in video autofocus as well as a technology suitable for incorporation into mirrorless cameras.  Other mirrorless cameras devote selected pixels within the sensor array to phase detection or possbily a hybrid combination of phase detection and contrast detection.⁹

Source credit:  https://www.extremetech.com/electronics/160232-canon-70d-with-dual-pixel-cmos-af-the-first-dslr-that-can-autofocus-videos

Image credit:  https://www.extremetech.com/wp-content/uploads/2013/07/vcsPRAsset_513951_109512_6d82de1a-a274-4030-9c05-ea89cae2cb6c_0.png

Canon has also experimented with extremely high MP sensors, e.g. the 120 MP APS-H (1.3 crop factor) demonstration sensor announced a few years ago.  But Canon's sensor readout bottleneck remains despite on-chip ADCs, constraining a whole host of sensor capabilities.  As a result, Canon cameras typically have cropped 4K video output and exhibit limited MILC frame rates compared to the competition.  Canon's initial full frame offereing, the EOS R, also does not have sensor-based in-body image stabilization, a staple feature of Sony's product line for years, and now of Nikon's Z6 and Z7.  Canon cites heat dissipation concerns for the absence of IBIS, but one can only surmise that the underlying lack of this desirable feature, which turns every non-IS lens into a stabilized lens, may have its origins in Canon's trailing position in stacked BSI sensor technology.

However, Canon is not standing still.  Especially critical is the vital area of sensor readout speed, a factor which pervasively restricts Canon's CMOS sensors in high performance features relative to the competition.  A number of patents in BSI and stacked sensor technology and in global shutter technology have been filed.¹⁰  Canon also has patents for quad pixel autofocus sensors, which would improve AF performance on horizontal patterns.¹¹  These  patents suggest that Canon will catch up.  It does, however, remain to be seen if and when Canon will finally deploy these technologies in production products.  And, needless to say, others, Sony included, are not standing still.  The evolution of imaging sensors over the next few years -- and of the cameras using them -- will be interesting to watch.

The table below provides a brief and very high level summary of the state of affairs with respect to the three biggest players in the digital camera marketplace -- Canon, Nikon and Sony.  Although the focus here is largely on interchangeable lens cameras, both DSLRs and MILCs, the points listed should also apply to fixed lens digitals as well, although that market continues to contract due to the increasing dominance of smartphones for small and compact applications.

Summary of Corporate Position in Digital Cameras
	Strengths	Weaknesses	Remediation
Sony	Leading in digital sensor technical capability in almost all metrics with BSI stacked sensor technology. Sales include sensors for smartphones, etc. ~40% sensor market share. Relatively large R&D budget because of above sensor sales leadership. Recent lens products excellent.	Trails Canon and Nikon in ILC sales. Must build up full range of lenses and accessories and product support chain. Lens design constrained by relatively small mount throat diameter. Camera body ergonomics trails others.	Continue and extend leadership in sensor technology. Slowly closing range of lenses and optical quality gap with Nikon & Canon. Slowly building up support network.
Nikon	Uses Sony supplied digital sensors; thus can incorporate near market leading sensor products. Long history of product excellence. Best high end camera action autofocus.	Dependent on Sony for sensors; thus features may lag slightly behind Sony in time and functionality.12 Smaller ILC market share than Canon.	Rapidly closing lens range and optical quality gap with Canon. Focusing on high-quality full-frame product line, both bodies and lenses.
Canon	Sensors made in-house = max profit. Most complete high quality lens lineup. Dual pixel AF excellent for video. Market leader in total ILC sales ~50%. Extensive high quality support network. Industry leader in overall patent filings.	Trails Sony in digital sensor technology. Limited R&D funds available due to small number of sensors sold. Lags Nikon in AF action tracking for high end bodies.	Marketing sensor products for other uses; if successful should make more funds available for R&D. Exploiting EF lens design experience and new MILC RF-mount with optically outstanding lens products. Patents for BSI and stacked sensors and global shutters.

 

Recent Canon Digital Camera Sensor Related Patents
BSI Sensor Manufacturing	Organic Sensor	Quad Pixel Autofocus Sensor
Stacked Sensor Global Shutter	Stacked Sensor for High Frame Rates	Stacked Sensor with Memory
Global Shutter Improved Response	Stacked Sensor with Electronic Shutter	DPAF Sensor Refinement

Image credits: https://www.canonnews.com/category/canon-patents-4

¹  "CCD vs. CMOS", Teledyne DALSA Knowledge Center page.

²  "Imaging Electronics 101", Edmund Optics, Application Notes on Imaging

³  "The Sony A7R IV and what it means to Canon", Canon Rumors, 16 July 2019.

⁴  "Das Sensor-Dilemma", Foto+Design - Dr. Schuhmacher.

⁵  "Smartphones industry: Statistics & Facts", Statista.com

⁶  "ISO Invariance: What it is, and which cameras are ISO-less", Improve Photography.

⁷  "CMOS Global Shutter Cameras", Basler Knowledge Base.

⁸  "More sensor talk: Maybe a 32.5MP APS-C sensor is coming now.  Canon News.

⁹  "Sony 4D AF versus Canon Dual Pixel AF", EOSHD, Andrew Reid, February 22, 2016

¹⁰   "Canon Patents", Canon News.

¹¹ "Canon Patent Application: Quad Pixel Auto Focus sensor" Canon News.

¹²  "Sony Keeps Its Best Sensors for Its Own Cameras", PetaPixel, Michael Zhang, March 22, 2017