UCB Visual Object and Activity Recognition, Class CS 294-43

 

UCB Visual Object and Activity Recognition, Class CS 294-43

Prof. Trevor Darrell, trevor@eecs.berkeley.edu, Spring 2011

 

This course will cover computer vision techniques for object and category recognition, as well as recognition of human activity from video streams.  Recognition of individual objects or activities (the coffee cup on your desk, a particular chair in your office, a video of you riding your bike) or generic categories (any cup, chair, or cycling event) is an essential capability for a variety of robotics and multimedia applications.  The advent of standardized datasets and evaluation regimes has spurred considerable innovation in this arena, with performance on benchmark evaluations increasing dramatically in recent years.  This course will review methods that have achieved success on such datasets, and will also consider the techniques needed for real-time interactive application on robots or mobile devices, e.g. domestic service robots or mobile phones that can retrieve information about objects in the environment based on visual observation.  This class will be based exclusively on readings from the recent literature, including those appearing at the CVPR, ICCV, and NIPS conferences.

 

The format of the course this year will primarily be discussion based, with each class beginning with a short overview of the topic by the instructor followed by detailed student-led presentations and structured critique of selected papers.  All students will be expected to actively discuss each paper each week.  Class size will be limited to those who have preregistered, or to 16 students, whichever is greater, to foster an environment conducive to discussion.

 

Each week will focus on a different subtopic of object and activity recognition, covering three to five different papers from the recent literature.  These papers will be presented jointly by two or three students, one acting as a primary presenter and the other student(s) as discussant.  Each student will be expected to act as presenter once and as discussant once during the term.  The presenting students will choose the papers from the list suggested for that subtopic, or they are welcome to suggest other papers. 

 

Students are expected to be involved in a related research project during the term, and be experimenting with a technique covered during the course.  (Graduate students who are not actively involved in a research project outside of the course can work on a class project specific for this course or joint with another course; undergraduates who are not actively involved in a related research project are not allowed in the course.)  Students will be expected to present their research progress during the term in a ten minute presentation in the last class.  Grades will be based entirely on in class presentations and participation.

 

This course will meet once a week, Friday 10-12noon, in the 7th floor conference room (Newton room) of Sutardja Dai Hall.

 

The first class will be jan 28th.  The introduction class which would have been scheduled jan 21st will happen virtually -- please contact the instructor if you are not already on the email list. 

 

Prerequisites: prior Computer Vision and Machine Learning courses, or permission of instructor. Advanced undergraduates allowed only with permission of instructor and if they are actively participating in a related research project.  Students should already be familiar with or be willing to learn on their own: basic image processing in MATLAB; Optic Flow; Edge Detection; Support Vector Machines;  Gaussian Mixture Models;  Hidden Markov Models, etc.; students must be able to read and understand at a basic level recent conference papers in the computer vision literature.

DRAFT Syllabus (class members please see google site for most up to date version):

January 28, 2011   Global Features    

Background readings:

1.         Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope," International Journal of Computer Vision, vol. 42, no. 3, pp. 145-175, May 2001. http://dx.doi.org/10.1023/A:1011139631724

2.         Efros, A. C. Berg, G. Mori, and J. Malik, "Recognizing action at a distance," ICCV 2003, pp. 726-733 vol.2. http://dx.doi.org/10.1109/ICCV.2003.1238420

3.         N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in CVPR '05: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), 2005, pp. 886-893.

http://dx.doi.org/10.1109/CVPR.2005.177

Contemporary readings:

4.         P. F. Felzenszwalb, R. B. Girshick, and D. McAllester, "Cascade Object Detection with Deformable Part Models", CVPR 2010.

http://dx.doi.org/10.1109/CVPR.2010.5539906

5.         T. Deselaers and V. Ferrari, "Global and efficient self-similarity for object classification and detection", CVPR 2010.

http://dx.doi.org/10.1109/CVPR.2010.5539775

February 4, 2011   Local Features

Background readings:

6.         D. G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, vol. 60, no. 2, pp. 91-110, November 2004.

http://dx.doi.org/10.1023/B:VISI.0000029664.99615.94

7.         T. Lindeberg, "Feature detection with automatic scale selection," International Journal of Computer Vision, vol. 30, no. 2, pp. 79-116, November 1998. http://dx.doi.org/10.1023/A:1008045108935

8.         J. Matas, O. Chum, U. Martin, and T. Pajdla, "Robust wide baseline stereo from maximally stable extremal regions," in Proceedings of British Machine Vision Conference, vol. 1, London, 2002, pp. 384-393. http://citeseer.ist.psu.edu/608213.html

9.         K. Mikolajczyk and C. Schmid, "Scale & affine invariant interest point detectors," Int. J. Comput. Vision, vol. 60, no. 1, pp. 63-86, October 2004.

http://dx.doi.org/10.1023/B:VISI.0000027790.02288.f2

10.     I. Laptev, "On space-time interest points," International Journal of Computer Vision, vol. 64, no. 2-3, pp. 107-123, September 2005. http://dx.doi.org/10.1007/s11263-005-1838-7

Contemporary readings:

11.     L. Bo, X. Ren, and D. Fox, "Kernel Descriptors for Visual Recognition", NIPS 2010, http://books.nips.cc/papers/files/nips23/NIPS2010_0821.pdf

12.     L. Bourdev, S. Maji, T. Brox, and J. Malik, "Detecting People Using Mutually Consistent Poselet Activations", ECCV 2010,

http://dx.doi.org/10.1007/978-3-642-15567-3_13

February 11, 2011          Bag-of-word and Correspondence Kernels         

Background readings:

13.     C. Dance, J. Willamowski, L. Fan, C. Bray, and G. Csurka, "Visual categorization with bags of keypoints," in ECCV International Workshop on Statistical Learning in Computer Vision, 2004. http://www.xrce.xerox.com/Publications/Attachments/2004%2D010/2004_010.pdf

14.     K. Grauman and T. Darrell, "The pyramid match kernel: discriminative classification with sets of image features," ICCV, vol. 2, 2005, pp. 1458-1465 Vol. 2. http://dx.doi.org/10.1109/ICCV.2005.239

15.     S. Lazebnik, C. Schmid, and J. Ponce, "Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories," CVPR, vol. 2, 2006, pp. 2169-2178. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1641019

Contemporary readings:

16.     S. Maji and A. C. Berg, "Max-margin additive classifiers for detection", ICCV 2009, http://dx.doi.org/10.1109/ICCV.2009.5459203

17.     A. Vedaldi and A. Zisserman, "Efficient Additive Kernels via Explicit Feature Maps", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5539949

18.     A. Kovashka and K. Grauman, "Learning a hierarchy of discriminative space-time neighborhood features for human action recognition", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5539881

February 18, 2011          Segmentation and Region Proposals

Background readings:

19.     J. Shotton, M. Johnson, and R. Cipolla, "Semantic texton forests for image categorization and segmentation," in Computer Vision and Pattern Recognition, 2008. CVPR 2008. http://dx.doi.org/10.1109/CVPR.2008.4587503

Contemporary readings:

20.     Y. Yang, S. Hallman, D. Ramanan, and C. Fowlkes, "Layered Object Detection for Multi-Class Segmentation", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5540070

21.     F. Li, J. Carreira and C. Sminchisescu, "Object Recognition as Ranking Holistic Figure-Ground Hypotheses", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5539839

22.     B. Alexe, T. Deselaers, V. Ferrari, "What is an object?", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5540226

23.     B. Packer, S. Gould, and D. Koller, "A Unified Contour-Pixel Model for Figure-Ground Segmentation", ECCV 2010, http://dx.doi.org/10.1007/978-3-642-15555-0_25

24.     I. Endres and D. Hoiem, "Category Independent Object Proposals", ECCV 2010, http://dx.doi.org/10.1007/978-3-642-15555-0_42

March 4, 2011        Descriptor Sparse Coding and Topic Models                 

Background reading:

25.     Olshausen B. and Field D. Sparse coding with an overcomplete basis set: A strategy employed by V1?. Vision Research (1997) vol. 37 (23) pp. 3311-3325 http://www.chaos.gwdg.de/~michael/CNS_course_2004/papers_max/OlshausenField1997.pdf

Contemporary readings:

26.     Raina et al. Self-taught learning: Transfer learning from unlabeled data. ICML (2007). http://dx.doi.org/10.1145/1273496.1273592

27.     Fritz M., Black M., Bradski G., Karayev S., Darrell T. An Additive Latent Feature Model for Transparent Object Recognition. NIPS (2009) http://books.nips.cc/papers/files/nips22/NIPS2009_0397.pdf

28.     Wang et al. Locality-constrained Linear Coding for Image Classification. CVPR (2010) http://dx.doi.org/10.1109/CVPR.2010.5540018

March 11, 2011      Hashing and Metric Learning       

Background readings:

29.     G. Shakhnarovich, P. Viola, and T. Darrell, "Fast pose estimation with parameter-sensitive hashing," ICCV 2003, http://dx.doi.org/10.1109/ICCV.2003.1238424

30.     A. Frome, Y. Singer, F. Sha, and J. Malik, "Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification", ICCV 2007, http://dx.doi.org/10.1109/ICCV.2007.4408839

Contemporary readings:

31.     P. Jain, B. Kulis, and K. Grauman, Fast Similarity Search for Learned Metrics, CVPR 2008/PAMI 2009, http://doi.ieeecomputersociety.org/10.1109/TPAMI.2009.151

32.     B. Kulis and T. Darrell, "Learning to Hash with Binary Reconstructive Embeddings", NIPS 2009, http://books.nips.cc/papers/files/nips22/NIPS2009_0971.pdf

March 18, 2011       Temporal Models          

Background readings:

33.     J. Niebles, H. Wang, and L. Fei-Fei, "Unsupervised learning of human action categories using spatial-temporal words," International Journal of Computer Vision. 79(3): 299-318. 2008 Available: http://dx.doi.org/10.1007/s11263-007-0122-4

Contemporary readings:

34.     K. Prabhakar, S. Oh, P. Wang, G. D. Abowd, J Rehg, "Temporal Causality for the Analysis of Visual Events", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5539871

35.     A. Yao, J. Gall, L. Van Gool, "A Hough Transform-Based Voting Framework for Action Recognition", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5539883

36.     J.C. Niebles, C. Chen, and L. Fei-Fei, "Modeling Temporal Structure of Decomposable Motion Segments for Activity Classification", ECCV 2010, http://dx.doi.org/10.1007/978-3-642-15552-9_29

37.     D. Weinland1, M. Ozuysal and P. Fua, "Making Action Recognition Robust to Occlusions and Viewpoint Changes", ECCV 2010, http://dx.doi.org/10.1007/978-3-642-15558-1_46

38.     P. Matikainen, M. Hebert and R. Sukthankar, "Representing Pairwise Spatial and Temporal Relations for Action Recognition", ECCV 2010, http://dx.doi.org/10.1007/978-3-642-15549-9_37

39.     T. Lan, Y. Wang, W. Yang and G. Mori, "Beyond Actions: Discriminative Models for Contextual Group Activities", NIPS 2010, http://books.nips.cc/papers/files/nips23/NIPS2010_0115.pdf

April 1, 2011 Image and text models           

Background readings:

40.     K. Barnard and D. Forsyth, "Learning the Semantics of Words and Pictures," International Conference on Computer Vision, vol 2, pp. 408-415, 2001, http://doi.ieeecomputersociety.org/10.1109/ICCV.2001.937654

41.     D. Blei and M. Jordan, "Modeling Annotated Data", SIGIR '03 Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, http://dx.doi.org/10.1145/860435.860460

42.     T. Berg and D. Forsyth, "Animals on the Web", CVPR 2006, http://dx.doi.org/10.1109/CVPR.2006.57

Contemporary readings:

43.     Chong Wang, D. Blei, Fei-Fei Li, "Simultaneous image classification and annotation," CVPR 2009, http://doi.ieeecomputersociety.org/10.1109/CVPRW.2009.5206800

44.     K. Saenko and T. Darrell, “Filtering Abstract Senses From Image Search Results”, NIPS 2009, http://books.nips.cc/papers/files/nips22/NIPS2009_1143.pdf

45.     A. Farhadi, M. Hejrati , M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier and D. Forsyth, "Every Picture Tells a Story: Generating Sentences from Images", NIPS 2010, http://dx.doi.org/10.1007/978-3-642-15561-1_2

46.     B. Siddiquie and A. Gupta, "Beyond Active Noun Tagging: Modeling Contextual Interactions for Multi-Class Active Learning", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5540044

April 8, 2011 Crowd sourcing and Active Learning    

Background readings:

47.     L. von Ahn and L. Dabbish, "Labeling images with a computer game", SIGCHI 2004, http://dx.doi.org/10.1145/985692.985733

48.     A. Kapoor, K. Grauman, R. Urtasun, and T. Darrell, "Active Learning with Gaussian Processes for Object Categorization" ICCV 2007. http://doi.ieeecomputersociety.org/10.1109/ICCV.2007.4408844

Contemporary readings:

49.     J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. "ImageNet: A Large-Scale Hierarchical Image Database". In CVPR, 2009. http://doi.ieeecomputersociety.org/10.1109/CVPRW.2009.5206848

50.     S. Vijayanarasimhan, P. Jain, K. Grauman, "Far-sighted active learning on a budget for image and video recognition", CVPR 2010. http://dx.doi.org/10.1109/CVPR.2010.5540055

51.     P. Welinder, S. Branson, S. Belongie, P. Perona, "The Multidimensional Wisdom of Crowds", NIPS 2010. http://books.nips.cc/papers/files/nips23/NIPS2010_0577.pdf

52.     S. Branson, C. Wah, B. Babenko, F. Schroff, P. Welinder, P. Perona, S. Belongie, "Visual Recognition with Humans in the Loop", ECCV 2010. http://dx.doi.org/10.1007/978-3-642-15561-1_32   

April 15, 2011         Scene and Image Context      

Background readings:

53.     A. Torralba, K. P. Murphy, and W. T. Freeman, "Contextual models for object detection using boosted random fields," in Advances in Neural Information Processing Systems 17 (NIPS), 2005, pp. 1401-1408. http://dspace.mit.edu/handle/1721.1/6740

54.     D. Hoiem, A. A. Efros, and M. Hebert, "Putting objects in perspective," in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2, 2006, pp. 2137-2144. http://dx.doi.org/10.1109/CVPR.2006.232

55.     L.-J. Li and L. Fei-Fei, "What, where and who? classifying events by scene and object recognition," in Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, 2007, pp. 1-8. http://dx.doi.org/10.1109/ICCV.2007.4408872

Contemporary readings:

56.     S. Bao, M. Sun, S. Savarese, "Toward coherent object detection and scene layout understanding", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5540229

57.     B. Yao and L. Fei-Fei. "Modeling Mutual Context of Object and Human Pose in Human-Object Interaction Activities.", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5540235

58.     A. Gupta, A. Efros and M. Hebert, "Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics". ECCV 2010, http://dx.doi.org/10.1007/978-3-642-15561-1_35 

April 22, 2011         Taxonomies and Sub-category Recognition                 

Background readings:

59.     A. Zweig and D. Weinshall, "Exploiting object hierarchy: Combining models from different category levels," in Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, 2007, pp. 1-8. Available:http://dx.doi.org/10.1109/ICCV.2007.4409064

60.     G. Griffin and P. Perona, "Learning and using taxonomies for fast visual categorization," in Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, 2008, pp. 1-8. Available:http://dx.doi.org/10.1109/CVPR.2008.4587410

61.     J. Sivic, B. C. Russell, A. Zisserman, W. T. Freeman, and A. A. Efros, "Unsupervised discovery of visual object class hierarchies," in Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, 2008, pp. 1-8. Available: http://dx.doi.org/10.1109/CVPR.2008.4587622

Contemporary readings:

62.     L.-J. Li, C. Wang, Y. Lim, D. Blei and L. Fei-Fei. "Building and Using a Semantivisual Image Hierarchy", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5540027

63.     M. Rohrbach, M. Stark, G. Szarvas, I. Gurevych, and B. Schiele, "What helps where – and why? Semantic relatedness for knowledge transfer", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5540121

April 29, 2011         Domain Adaptation

64.     K. Saenko, B. Kulis, M. Fritz, and T. Darrell, "Adapting Visual Category Models to New Domains", ECCV 2010, http://dx.doi.org/10.1109/10.1007/978-3-642-15561-1_16

65.     A. Bergamo and L. Torresani, "Exploiting weakly-labeled Web images to improve object classification: a domain adaptation approach", NIPS 2010, http://books.nips.cc/papers/files/nips23/NIPS2010_0093.pdf

66.     L. Cao, Z. Liu, T. Huang, "Cross-dataset action detection", CVPR 2010, http://dx.doi.org/10.1109/CVPR.2010.5539875