Going blind to see more clearly: unconscious bias in Australian Public Service (APS) shortlisting processes

Authored on
6 years 9 months ago
Complete
Project Type
Evaluation report
Policy Area
Social and Health
Partner agencies
Attorney General’s Department
Australian Public Service Commission
Australian Taxation Office
Department of Agriculture and Water Resources
Department of Defence
Department of Employment
Department of Environment and Energy
Department of Foreign Affairs and Trade
Department of Health
Department of Industry, Innovation and Science
Department of Social Services
Department of the Prime Minister and Cabinet
Department of the Treasury
Fair Work Ombudsman
Offices of National Assessments
Registration date
Monday, 14 November 2016

This study assessed whether women and minorities are discriminated against in the early stages of the recruitment process for senior positions in the Australian Public Service (APS). It also tested the impact of implementing a ‘blind’ or de-identified approach to reviewing candidates.

Over 2,100 public servants from 15 agencies participated in the trial. They completed an exercise in which they shortlisted applicants for a hypothetical senior role in their agency. Participants were randomly assigned to receive application materials for candidates in standard form or in de-identified form (with information about candidate gender, race and ethnicity removed).

Overall, the results indicate the need for caution when moving towards ’blind’ recruitment processes in the APS, as de-identification may frustrate efforts aimed at promoting diversity.

Authors: Professor Michael J. Hiscox, Tara Oliver, Michael Ridgway, Lilia Arcos-Holzinger, Alastair Warren and Andrea Willis.

Pre-registration information

Women are under-represented in management and executive level positions across the private and public sectors. In 2016, women comprised 59.0% of the APS, but accounted for 48.9% of its executive level officers and only 42.9% of its Senior Executive Service (SES) officers. These statistics may reflect gender discrimination in hiring and promotion processes as a result of unconscious cognitive biases that affect decision-making.

Addressing the gender imbalance across the APS is the key priority of the Australian Public Service Gender Equality Strategy 2016-19. Aimed at driving high performance and boosting productivity the strategy calls for the APS to reflect contemporary reality and states that the APS must embrace diversity and that it should benefit from people of all backgrounds.

Study

This study aimed to test the magnitude of gender and ethnic minority bias in APS shortlisting processes, and the impact of introducing a gender/ethnicity-blind approach to reviewing job applications for shortlisting purposes.

The study was a randomised controlled trial conducted in partnership with 15 APS agencies. Participants were drawn from senior and executive level officers in these agencies. To identify any effects participants were asked to complete a fictitious shortlisting exercise with 16 fictitious CVs. The aim was to see whether de-identifying a CV (by removing a candidate’s name and personal information) changed the way it was assessed.

Results

The results showed that overall, de-identifying applications at the shortlisting stage does not appear to assist in promoting diversity within the APS in hiring. Overall, APS officers discriminated in favour of female and minority candidates.

Intervention start and end date: Monday, 14 November 2016 to Tuesday, 22 November 2016

BETA ethics pre-registration number: BETA ETH 2016 – 001, 21 October 2016

Experimental design: The design of this trial is an individually randomised controlled trial. Specifically, the trial is a framed field experiment where participants will complete a task that is normal for employees at their levels – assessing candidates for an APS job vacancy. Participants have been informed that in this instance that the exercise is hypothetical.

Individuals who agree to participate in the study are randomly assigned to either the treatment or one of two control conditions. Pre-randomisation stratification is based upon agency, age, gender, and APS level and role type, conditional on the number of participants being large enough to stratify based on these characteristics.

Data is collected from participants at two points. Firstly, when candidates register to participate for the trial (24-31 October), and secondly when they complete the exercise (14-22 November).

Intervention(s): The intervention (treatment) in this trial involves removing the names from CVs. By de-identifying CVs in this way, any indication of gender and ethnicity is removed.

Women are under-represented in management and executive-level positions in the private sector and in many areas of the Australian Public Service (APS). This may be a result of gender discrimination in hiring and promotion decisions. De-identifying CVs may offer an effective way to mitigate biases in hiring and promotion decisions.

In the 1970s and 1980s American symphony orchestras attempted to overcome biases in hiring by introducing a screen during auditions to conceal the identity of the musician from the jury evaluating the performance. In a well-known study analysing data on auditions and hiring by orchestras over this period, Goldin & Rouse (2000) found that the use of blind auditions had a major impact on gender bias in orchestras, increasing the likelihood of female musicians being selected by 25-40%.

A small number of public and private sector organisations in Europe have adopted anonymous job application procedures and several studies suggested that de-identification could significantly reduce bias. There have been no similar studies conducted in Australia and the APS to determine the effect of de-identifying CVs on shortlisting outcomes.

Control condition: In the control condition, participants will review the CVs in the usual identified form (i.e.: with names listed on each CV). There will be two control groups, the only difference being that each CV will have the name swapped to a similar name of the opposite gender (e.g.: Jane Smith’s CV in control group 1 will be John Smith’s CV in control group 2).

Having these two control groups will allow us to fully control for all other characteristics of the CVs while altering only gender (i.e.: the name in the CVs), and provides another level of assurance that applicant characteristics are balanced across gender.

Outcome(s): Primary outcomes of interest are:

  • gender discrimination effects
  • the effect of CV de-identification on gender discrimination
  • ethnic discrimination effects.

Secondary outcomes of interest include:

  1. A range of demographic characteristics and other variables that previous research has identified as being related to bias. These include (but are not limited to):
    • Demographic characteristics: age, gender, APS classification level, role type/function.
    • Other variables: how efficient the process was, confidence in shortlisted candidates, intensity of current workload.

Expected sample size: 2,500 adults

Other:

Hiscox, Michael. 2016. Unconscious Bias in Hiring in the Australian Public Service: Evidence from a Framed Field Experiment in Shortlisting of Job Candidates. AEA RCT Registry. November 22.