COMP90086 Computer Vision

COMP90086 Computer Vision, 2024 Semester 2

Assignment 2: Patch matching with neural networks

Introduction

Finding correspondences between keypoints is a critical step in many computer vision applications. It can be used to align images when constructing a panorama from lots of separate photogtraps, and it is used to find point correspondences between keypoints detetected in multiple views of a scene.

This assignment uses a dataset generated from many views of the Trevi fountain in Rome. Finding correspondences between detected keypoints is a critical step in the pipeline for reconstructing a 3D representation of the fountain from individual photographs.

The dataset in this assignment is generated as a set of pairs of image patches taken centred at detected  keypoints. The image patches are 64x64 pixels each and each training sample is made of two patches  placed side by side to make a 128x64 image. For half the training set (10,000 examples in the ’1good’ subdirectory) the two patches are from two separate views of the same keypoint. For the other half (10,000 examples in the ’0bad’ subdirectory) the two patches are from two different keypoints. Figure 1 shows an example of each of these. The validation directory is similarly structured but contains four times as many non-matching pairs (2000 examples in ’0bad’) as matching pairs (500 examples in  ’1good’).

Figure 1: Corresponding (left) and non-corresponding (right) pairs of image patches

Your task is to create and train some neural networks that can tackle the problem of determining whether the two patches correspond or not.

1. Baseline Neural Network [2 pt]

Run the baseline neural network implementation in the provided python notebook and in your report, you should include the loss and accuracy curves for the training and validation sets in your report and discuss what these imply about the baseline model.

The validation set contains more bad examples than good.  Why might this be a sensible way of testing for the task of finding feature correspondences?  Should the training environment also reflect this imbalance?

2. Regularizing your Neural Network [2pt]

To regularize the network, your should try adding a regularization layer (see the Keras documenation for these layers). Try adding a Dropout() layer after Flatten() and try different rate values to see what the effect of this parameter is.  Include the loss and accuracy plots in your report for three different choices of the rate parameter.  Describe the changes you see in these loss and accuracy plots in your report and suggest what the best choice of rate value is from the three you have reported.

3. Convolutional Neural Network [3pt]

Design a Convolutional Neural Network to solve this challenge.  If you use Conv2D() layers imme- diately after the LayerNormalization layer these convolutions will apply identically to both image patches in each input sample. Try using one or two Conv2D() layers with relu activations. You should explore the value of having different numbers of filters, kernel sizes, and strides before the Flatten() layer.

Briefly describe the set of settings you tried in your report in a table (this should be around 10 settings). For each setting, report the final training loss and accuracy as well as the validation loss and accuracy.

Include a discussion of the results of these experiments in your report. Identify your best performing design and discuss why you think this may have been best.

Submission

You should make two submissions on the LMS: your code and a short written report explaining your method and results. The response to each question should be no more than 500 words.

Submission will be made via the Canvas LMS. Please submit your code and written report separately under the Assignment 2 link on Canvas.

•  Your code submission should include the Jupyter Notebook (please use the provided template) with your code and any imagefiles we will need to run your code. Please include the cell output in your notebook submission if possible.

•  Your written report should be a .pdf with your answers to each of the questions.  The report should address the questions posed in this assignment and include any images, diagrams, or tables required by the question.

Evaluation

Your submission will be marked on the correctness of your code/method, including the quality and ef- ficiency of your code. You should use built-in Python functions where appropriate and use descriptive variable names. Your written report should clearly address the questions, and include all of the specific outputs required by the question (e.g., images, diagrams, tables, or responses to sub-questions).

Late submission

The submission mechanism will stay open for one week after the submission deadline. Late submis- sions will be penalised at 10% of the total possible mark per 24-hour period after the original deadline. Submissions will be closed 7 days (168 hours) after the published assignment deadline, and no further submissions will be accepted after this point.

To request an extension on this assignment, please see the FEIT extension policy and follow the steps below:

• To request an extension of 1-3 days (without AAP), complete the declaration form at the website above and upload it to Canvas under Assignment 2 extension request.

• To request a longer extension (without AAP), please apply for Special Consideration.

• If you have an AAP, please request an extension by completing the Assignment 2 extension request form. on Canvas and uploading your AAP.

Please note that we can only accept extension requests via Canvas up until the assignment deadline. Late extension requests can only be granted through Special Consideration.  The longest extension granted on this assignment is 7 days (5 working days + weekend).