What do image generators know?

Intrinsic images are maps of surface properties, like depth, normal and albedo.  One usually learns methods to produce intrinsic images using various kinds of paired data; for example, image and depth.  This is tricky, and may be unnecessary.

I will show that image generators can be forced to produce many different lightings of the same scene.  But if an image generator can relight a scene, it likely has a representation of depth, normal, albedo and other useful scene properties somewhere.  I will show strong evidence that depth, normal, and albedo can be extracted from two kinds of image generators, with minimal inconvenience or training data.   Furthermore, all these intrinsics are much less sensitive to lighting changes.  This suggests that the right way to obtain intrinsic images might be to recover them from image generators.  It also suggests image generators might "know" more about scene appearance than we realize.

I will show that there are important scene properties that image generators very reliably get wrong.  These include shadow geometry and perspective geometry.  Similarly, video generators get object constancy and properties like momentum conservation wrong.  Intriguing questions follow:  can we re engineer image understanding pipelines around image generators?  why do image generators not “know” what they don’t “know”? and what can we do about their ignorance?

About Dr. David Forsyth, UIUC

I am currently Fulton-Watson-Copp chair in computer science at U. Illinois at Urbana-Champaign, where I moved from U.C Berkeley, where I was also full professor.  I have occupied the Fulton-Watson-Copp chair in Computer Science at the University of Illinois since 2014. I have published over 170 papers on computer vision, computer graphics and machine learning. I have served as program co-chair for IEEE Computer Vision and Pattern Recognition in 2000, 2011, 2018 and 2021, general co-chair for CVPR 2006 and 2015 and ICCV 2019, program co-chair for the European Conference on Computer Vision 2008, and am a regular member of the program committee of all major international conferences on computer vision.  I have served six years on the SIGGRAPH program committee, and am a regular reviewer for that conference. I have received best paper awards at the International Conference on Computer Vision and at the European Conference on Computer Vision. I received an IEEE technical achievement award for 2005 for my research.  I became an IEEE Fellow in 2009, and an ACM Fellow in 2014.  My textbook, "Computer Vision: A Modern Approach" (joint with J. Ponce and published by Prentice Hall) is now widely adopted as a course text (adoptions include MIT, U. Wisconsin-Madison, UIUC, Georgia Tech and U.C. Berkeley).  A further textbook, “Probability and Statistics for Computer Science”, is in print; yet another (“Applied Machine Learning”) has just appeared.   I have served two terms as Editor in Chief, IEEE TPAMI.  I have served on a number of scientific advisory boards.