Employers are turning to job auditions over interviews to identify candidates they want to make offers to. How can you ethically make this work for accessibility?

“Job Auditions” — an interview process that includes doing actual work to prove that you CAN do a job, before you are offered the job — is becoming a significant employment trend. Why are employers turning to this more intensive process? Because turnover is expensive, and bad hires are even more expensive. LinkedIn says 32% of hiring managers report using job auditions to screen candidates and 84% consider it an effective hiring tool.

Job auditions can range from:

Brainstorming sessions
Whiteboarding sessions
Completing a small assignment

all the way to “try it on for size” roles like working for a company for several weeks in a temporary role until both you and the employer decide you want to move forward.

Job auditions have effectively replaced probationary periods. During a “probationary period” cause is not required to terminate an employee. However, this has become illegal in some states. Even if you limit the damage by getting rid of a bad employee fast, it is still disruptive and demoralizing to existing employees to have people constantly dropping in and out.

If the “best bug” is the one that was prevented from happening in the first place, then the “best bad employee” is the one that was never hired.

Are job auditions fair?

Using a job audition for all candidates for a particular job shows that an employer is more interested in what skills the candidate brings to the job and how they solve problems than what is written on their resume. So from that aspect, they are extremely fair.

On the flip side, employers are asking candidates to commit personal time to the audition. If the audition is truly time-boxed to a very short amount of time (1–2 hours), at first glance it would appear to be a small amount of unpaid time commitment for the potential of a significant pay off. However, this is where employers start to stray into disproportional impact. A number of groups of people may have constraints to taking time off include:

People with caretaking responsibilities (i.e. young children or elderly relatives)
People who hold down multiple jobs
People who have trouble getting the needed time off (inflexible employer, no PTO, audition is too long)

Where job auditions goes over the line to full blown unfair is where the work is expected to be provided for free and used by the employer. It’s even worse when the candidate doesn’t get the job, but the work is good enough to get used. That is just flat out unethical. Also borderline unethical is employers who say “take this project, spend a couple of hours on it” knowing full well that most successful candidates will have spent 30 or 40 hours on the project because of the competitive nature and reputation of the employer.

Accessibility Tester Job Auditions

Despite the caveats above, job auditions can be a great fit for accessibility hands-on testing roles, if you think through the following questions first.

1. What do you want to get out of the audition?

The first thing you need to do is define what you want to get out of the audition. My recommendation is to focus on soft skills, to understand the tester’s process/methodology, and what they consider important. Some of the things that will help you understand the candidate’s methodology and accessibility philosophy at a deeper level are:

Do they test keyboard access first or screen readers?
Do they deeply test one AT before moving on to the next, or do they mix it up and focus when they find inaccessible features
Do they test templates first or content?
Do they test color contrast in color blindness mode?
What does the candidate consider a critical defect?
How many well-crafted defects can the candidate identify in an hour?

All of this effectively requires you to to give the candidate a site to test where there are way more defects than they can find in an hour, to ramp up the time pressure and understand what the candidate thinks is important. I really don’t recommend it being the employer’s site, as the candidate should have made themselves familiar with that one before the audition.

It is not hard to find sites that have terrible accessibility, just look at the WebAIM report if you need help finding one. But pick one site and stick with it for all interviews to get an “apples to apples” comparison between your candidates abilities.

2. Ask EVERY candidate if they need reasonable accommodations

I have seen nothing written about job auditions and reasonable accommodations. However, the right to receive reasonable accommodations for pre-employment examinations has been exhaustively litigated, and the answer is “yes, candidates are absolutely entitled to reasonable accommodations for a pre-employment examination”. And what is a “job audition” but yet another type of pre-employment examination? Don’t mess about with this suggestion, I think it is really non-negotiable.

The chances of an accessibility testing role attracting candidates with disabilities is vastly higher than non-accessibility positions
No one is capable of determining who needs reasonable accommodations and who doesn’t just by looking at them and talking to them

You don’t want to end up on the front page of LinkedIn or on the wrong side of an EEOC investigation by not providing needed accommodations, or failing to ask if they are even needed.

3. Write a formal audition script and rubric

Again to be the most fair, you need to make sure all the candidates get the same information and are scored the same way. This is most easily accomplished by writing an audition script with instructions, and a formal scoring rubric. Make sure the script is accessible 🙂

If you are going to ask the candidate to log defects, give them an accessible excel template to log them in so they know what information you want for each identified problem

4. Keep the audition short

Go back to your goals. Is there anything that you need two hours to understand that you won’t know after one? Or eight hours that you won’t understand in two? Go with the smallest number of hours as possible, with the understanding that the longer you make the audition, the more likely it is that it will be unfair and discriminate against people with disabilities.

5. Pay for the work if you are going to use it

That may require a little bit of additional paperwork up front. Possibly an NDA (if that isn’t part of your job process already), and the completion of W9 and in extreme cases, IP transfers (in case you discuss ideas that can be used by your organization).

Some auditions are done as contests as a “many to many” type of event where you have many candidates applying for many jobs with the winner getting a prize. Bug bounties such as those paid by Apple and Tesla can also be considered to come under this category. Bug bounty programs in California may be affected by AB5, so if you are thinking about doing accessibility contests or bug bounties, pay attention to changes and interpretations with that law.

Nothing I write should be construed as legal advice, it is best to sort all of these issues out with your HR/legal group long in advance of the audition.

— — — — — — — — — — — — — — — — — — —

Here are other articles I’ve written about disability, accessibility, and employment:

Accessibility Job Auditions