Abstract:

Automated browser-based tools are widely used to evaluate web accessibility compliance with the Web Content Accessibility Guidelines (WCAG). However, these tools use different underlying engines and rule sets, and it remains unclear whether they produce consistent results when applied to the same websites. This study compares three browser-based automated accessibility evaluation tools, WAVE, Accessibility Insights for Web, and QualWeb, by applying them to the homepages of 20 healthcare-related websites from five European Union countries: Poland (PL), Germany (DE), Sweden (SE), Finland (FI), and Romania (RO). The countries were selected to represent a range of accessibility maturity, from leading performers in digital maturity (Sweden, Finland) to lower-performing ones (Romania), based on the Digital Economy and Society Index (DESI, European Commission, 2024). To enable systematic cross-tool comparison, this study proposes the Weighted Accessibility Concern Index (WACI), a weighted composite metric that aggregates detected errors, contrast failures, and alerts by severity. The absence of a standardised output format across tools was a key finding of this study. It also motivated the development of replicable extraction procedures that enabled the application of a single WACI formula across all three tools. Results indicate moderate but statistically significant rank agreement between tools (Kendall’s τ = 0.343–0.516, all p < 0.05), while substantial differences in absolute WACI scores were observed. Sweden and Finland achieved the lowest mean WACI scores, Romania recorded the highest burden across all tools. Germany showed unexpectedly high scores despite its digital maturity ranking. The study contributes a replicable methodology for browser-based tool comparison in a European regulatory context, following the entry into force of the European Accessibility Act in June 2025.