COMM1190: DATA INSIGHTS AND DECISIONS FINAL EXAMINATION TERM 1 2024R

Java Python TERM 1 2024

COMM1190: DATA, INSIGHTS, AND DECISIONS

FINAL EXAMINATION

QUESTION 1 10 MARKS

Suppose you work as an actuary and want to model the 5-year mortality rates (i.e., the probability of death within five years) for a group of 1,000 life insurance policyholders aged 50. You have the following information for each:

•   Gender

• Type of work (manual or non-manual);

•    Living area (High affluence, Middle class, Deprived);

•    Health condition (Terminally ill, Sick, healthy)

The following classification tree is obtained for the binary variable denoting the Status of the policyholder as observed in the data. The Status variable takes a value equal to 0 if the policyholder is alive at the end of the study (5 years, where the policyholders are observed starting at the age of 50 until age 55) and 1 if dead.

Required:

With reference to the above case, please answer all of the following questions:

Part A [max 100 words] (5 marks)

What does each number in a node represent? Please describe the results in the node circled in red.

Part B [max 100 words] (5 marks)

What are the most influential variables characterising policyholders’ mortality in order of importance?

QUESTION 2                                                                                                 30 MARKS

A financial institution has decided to implement a Machine Learning (ML) algorithm to automate its loan approval process. The algorithm, developed by a third-party AI solutions provider, is designed to analyse a range of consumer data to determine loan eligibility and terms. To maintain a transparent process, the firm offers applicants the option to discuss the loan outcome decision with a representative. The representatives are well-trained in explaining how the applicant's data influenced the  loan decision.

The third-party developer has provided the firm with two versions of the algorithm. The first version is "gender-blind," designed to exclude any variables that could directly or indirectly reveal an applicant's gender. The second version incorporates   gender-specific features, which can enhance the accuracy of loan predictions based on statistical differences in financial behaviour across genders. The third-party developer is also open to feedback on variables to include in the model and how to determine who is loan-worthy.

Required:

With reference to the above case, please answer all of the following questions:

Part A [max 100 words] (4 marks)

Given a choice between the gender-blind version and the gender-specific version of the loan decision algorithm, identify which option the firm should choose to use and any general adjustments that could be made to make the algorithm more responsible.

Part B [max 200 words] (6 marks)

Considering the scenario, identify and discuss the issues that raise concerns under the Australian Responsible AI Principles, focusing specifically on fairness, accountability, transparency, and contestability.

Part C [max 200 words] (12 marks)

The firm has a dataset of 15,000 people. The results of the model are graphically presented as follows:

Create confusion matrices to determine the following:

I.     The overall accuracy rate

II.     The overall false positive rate

III.     The accuracy rate for men

IV.     The false positive rate for men

V.     The accuracy rate for women

VI.     The false positive rate for women

Part D [max 200 words] (8 marks)

I.     Based on the confusion matrix, explain why you believe or do not believe the predictive model is fair.

II.     1500 new clients (1000 men and 500 females) have applied for loans and need to be assessed. The third-party developer can adjust the model so that it selects 500 men and 250 females to be selected for loans. What are the advantages and disadvantages of this approach? Should the firm update the model to this approach?

QUESTION 3                                                                                                 30 MARKS

In addition to stores in urban areas, a grocery store chain operating throughout Australia has 530 stores in regional and rural areas. Management has long recognised the need to consider these stores and the markets they serve differently from stores located in urban areas. These stores are located in well-defined geographical markets, and while the chain may have several stores in a market, typically, there will only be a small number of competing stores in any market. Because of this market structure, there has been a view that there are benefits in local advertising campaigns in addition to the chain’s nationwide brand image marketing, but until recently, there has been little formal investigation of this view.

Management decided to run a local advertising campaign via community newsletters and local radio and TV stations in all the markets, including their 530 stores in regional and rural areas, during the fourth quarter (q4) of 2023.  Quarterly sales per store in these markets are routinely tracked over time. A junior administrative assistant, Evie, is tasked with tracking this data and reporting it back to Marley, the Head of Marketing. After inspecting the data, Evie reports that the local ad campaign had been successful, as there was a discernible jump in sales associated with the campaign's introduction in 2023q4.

Marley is not convinced and asks for evidence to support Evie’s conclusion. In response, Evie produces Figure 1, which provides a time series of quarterly sales averaged over 530 stores. After seeing this figure, Marley agrees there was a jump in average sales in 2023q4 but remains unconvinced that this alone is sufficient evidence to support the conclusion that the ad campaign caused the increase. Further, she notes that due to unforeseen logistical problems, some markets had to delay the introduction of the ad campaign. She asks Evie to repeat the Figure, this time separating out the 318 stores in markets where the ad campaign had been conducted in 2023q4 and compare this to the sales of the 212 stores where the campaign was delayed and was not conducted in 2023q4. Figure 2 provides this information, noting that the division of stores is based on the ad campaign that was only run in 2023q4.

Required:

With reference to the above case, please answer all of the following questions:

Part A [max 160 words] (8 marks)

In arguing that the ad campaign had a positive impact on sales, Evie noted  the $341,000 (or 22%) increase in average store sales from 2023q3 to 2023q4. Using the information provided in Figure 1, argue why Marley is warranted in having concerns about using this increase as an estimate of the impact of the ad campaign.

Part B [max 80 words] (4 marks)

Do the new insights from Figure 2 add any further concerns about the initial results provided by Evie? Justify your answer.

Part C [max 120 words] (6 marks)

Evie also provided the average sales for the fourth quarter (q4) of 2022 and 2023, in total and separately for those markets that did conduct the ad campaign and those that did not. These results are produced in Table 1. What if we instead compare total sales in 2022q4 to 2023q4. Is this a better approach than that used by Evie and reported in part (a). If so, why?  Based on the data in Table 1, what is the best estimate of the ad campaign's impact? Justify your choice of estimate.

Table 1: Average sales for quarter 4 in 2022 and 2023 of stores

Variable definitions	Total	COMM1190: DATA, INSIGHTS, AND DECISIONS FINAL EXAMINATION TERM 1 2024R Ad campaign	No ad campaign
Average store sales in quarter 4 of 2022 ($millions)	1.801	1.820	1.772
Average store sales in quarter 4 of 2023 ($millions)	1.909	1.935	1.869
Number of stores	530	318	212

Part D [max 100 words] (5 marks)

Marley had previously decided that it would be useful to explore further the possibility of using local ad campaigns to boost sales in regional and rural markets. To do so, she took advantage of the failure to introduce the ad campaign in some markets at the end of 2023, and rather than introducing the ad campaign in 2024 in all of these markets, she instead designed a field experiment and randomly allocated these markets to treatment and control groups. Thus, in the first quarter of 2024, there were a sample of 109 stores exposed to the ad campaign, those in a treated market, which could be compared to the 103 stores in markets in the control group where no ad campaign was conducted.

The results from that experiment are now in and are summarised in Table 2. Based on Table 2, is there any evidence to suggest that random assignment in the experiment was successful? Explain your answer. Explain why it was a good design choice to randomise over markets rather than stores.

Table 2: Sample means for key variables divided into 3 groups of stores*

Variable definitions	Original	Treatment	Control
Sales Average store sales in quarter 1 of 2024 ($millions)	1.876	1.897	1.850
Competitors Number of competing stores in the market	2.38	2.39	2.36
Regional = 1 if store is in a regional area; = 0 if store is in a rural area	0.65	0.63	0.61
Advantaged** = 1 if store is in an advantaged area according to the SEIFA  index; = 0 otherwise	0.44	0.37	0.37
Number of stores	318	109	103

Notes: *The 3 groups of stores are: Original=those subject to the ad campaign in 2023q3; Treatment=those not subject to     the ad campaign in 2023q3 but have an ad campaign in 2024q1 as part of the experiment; Control=those not subject to the ad campaign in 2023q3 or in 2024q1. **SEIFA index combines Census data such as income, education, employment, occupation, housing and family structure to summarise the socio-economic characteristics of an area. A low score indicates relatively greater disadvantage and a lack of advantage in general. Conversely a high score indicates a relative lack of disadvantage and greater advantage in general.

Part E [max 80 words] (4 marks)

Based on the Table 2 results, what was the estimated effect of the ad campaign? To further analyse the experimental results, define treatment as being exposed to the ad campaign or not:treat   =  1 if store in a market with a local advertising campaign in 2024q1 and =  0 otherwise and specify two regression models with store sales (salesi ) as the dependent variable:

salesi  = β0  + β1 treati  + ui

salesi  = β0  + β1 treati  + β2x1i  + … + β2xpi  + ui

The results from these regression models are presented in Table 3. Are the results for the treatment effect in column (1) consistent with that in column (2) and explain whether this is what you would expect?

Part F [max 60 words] (3 marks)

Based on all the evidence provided by the observational and experimental data, summarise what has been learnt about the impact of local ad campaigns.

QUESTION 4                                                                                                 30 MARKS

Required:

Part B and Part C are only assessed if Part A is complete.

Part A [0 words] (0 marks)

Provide screenshots of each graph from your first (the purely individual) assessment.

Part B [max 200 words] (10 marks)

Critically evaluate the quality of your graphs from a “Good Charts” perspective. This can include aspects of the chart that represent good design execution, high contextual awareness and areas that could be improved.

Part C [max 200 words] (20 marks)

Using “Good Charts” methods, recreate two graphs from your assignment to improve the design execution and contextual awareness. Explain the changes made in each graph and why they achieve higher design execution. You can use any software of your choosing or sketch the graph by hand and scan the image to your exam response document