TL;DR — you will get an inaccurate, incomplete, and biased report of your actual accessibility status, plus potentially create significant legal liability
This article is not legal advice. This is a general opinion article and should not be relied upon for any legal situation. Always consult an attorney who specializes in accessibility for your legal issues.
I had an algebra teacher in middle school who was famous for yelling “TANSTAAFL” at his students — which stands for Their Ain’t No Such Thing As A Free Lunch.
That principle is also true for vendors selling services or accessibility “solutions.” I put the word solutions in quotes because many accessibility vendors are selling overlay-based solutions that aren’t solutions. Read more about that here. You may be thinking to yourself:
- I heard about an ADA digital accessibility lawsuit against a competitor. I wonder what shape my website is in, or;
- I wonder how I can reduce my risk of being targeted for one of these ubiquitous accessibility lawsuits?
Combine those feelings with the standard human response, “what is better than free?” and you might be chomping at the bit to try out one of these services that reportedly tells you where you are accessibility-wise.
Keep the following facts in mind if you decide to go this route after finishing this article.
Problem #1: These vendors are primarily trying to sell tools and services.
Frequently the reports emphasize the doom and gloom (i.e. errors). The unstated implication is that the only people who can help you out of the accessibility quicksand your site is in is, of course, them. For a price, that is not free. These vendors have ZERO interest in telling you that your organization’s website is good enough or that the site requires only a few updates.
Problem #2: These vendors are offering automated reviews only.
Why is an automated review only an issue?
- Automated accessibility testing can only pick up 30 % of accessibility violations on average.
- Automated accessibility testing only looks at static code and not standard single-page app behavior or multi-step functions such as checking out, registering, or performing a search.
- Automated accessibility testing is notorious for generating false positives, telling you there is a problem when there isn’t. False positives take a lot of time for you to investigate, because trust me, the vendor isn’t going to do that work for free.
- False negatives are an even bigger problem with automated accessibility testing than false positives because false negatives make you think things are okay when they are not.
- Automated accessibility testing won’t catch things like ARIA misuse — they will only flag if the ARIA is bad, not if HTML should have been used instead of ARIA, for example.
Bottom line: Only a full accessibility audit, which must include manual review (preferably executed by individuals with disabilities), will inform you of your true accessibility status.
Problem #3: Free testing services create an evidence trail that can be used against your organization if it gets sued.
During the “document production” discovery phase of litigation, the plaintiffs’ attorney will ask you to produce every single document ever created at your organization that includes the word “accessibility.” That will include this report, which you will likely have to turn over to them.
Litigation is all about one party trying to prove that the other party was a bad actor and cannot be believed. What looks worse than a defendant that has:
- Gone to an automated accessibility testing site.
- Received results that indicate the organization’s site is not accessible. The average webpage has more than 40 accessibility errors. The chances of the organization’s website getting a pass when the vendor offering the free test profits off of telling you that you’ve failed are vanishingly close to zero, don’t delude yourself.
- Failed to fix every single issue raised in the report.
Plaintiffs’ attorneys will characterize this common set of actions as “willful disregard” — your organization had the report, and anything short of complete remediation means you disregarded the report and hence intentionally continued your discriminatory actions.
Problem #4: The vendors generate inconsistent accessibility test results.
As an experiment, I ran the home page of the most accessible website I know, gov.uk, through a couple of different free test audits. Disclaimer — most of these vendors don’t give you results without disclosing your name and email. I didn’t feel like doing that and was too lazy to lie. These are the results of the two vendors that provided at least some information without providing contact details.
Level Access reported that the gov.uk had a health score of 100 %, which I expected.
In contrast, AccessiBe scored gov.uk as “Semi compliant,” which I did NOT expect. AccessiBe did not provide further details at this level.
Problem #5: You don’t get to peek behind the curtain to see what tests the vendors are executing.
Note the Level Access test report on gov.uk stated that 276 tests were run. But, what 276 tests? You have no idea, but at least you know those tests passed because the health score was 100 %. What goes into the “health score” is a Level Access trade secret, and not something consistently executed from company to company. Most websites don’t get a 100 % health score.
AccessiBe didn’t even comment on the number of tests. With only “semi compliant” as the stated result, I am inferring that one or more AccessiBe tests failed that Level Access wasn’t running. The failed test from the AccessiBe report could be a false positive? But you don’t know, because you don’t get any test details. It is possible more details are available if you provide contact information but I didn’t take this experiment that far.
What is an inclusively-minded product contributor supposed to do instead?
If your corporate attorney orders an accessibility test/audit/review in anticipation of potential litigation, the test results might be protectable as attorney-client work product when that pesky document production request comes in during the discovery phase once a lawsuit is actually filed.
If an individual not acting as corporate counsel orders the accessibility testing directly and then the organization gets sued, you will most certainly have to produce it as well as answer very detailed questions about what you did with the results.