Methodological details of monitoring
Which accessibility criteria were checked and how the selection of subpages or screens for the checks was made
Most of the checks were carried out in cooperation with agencies commissioned for this purpose. In Vienna, the simplified checks were carried out by internal experts, in Tirol, all checks were carried out by internal experts.
Overall, a uniform audit and reporting procedure was sought for all three types of checks for the whole of Austria, which could be achieved as far as possible.
“The transformation to the digital society needs accessibility as a basis for democracy and equality. “Leave no one behind” must not remain a catchphrase, but needs not only laws and standards but also monitoring, quality assurance and documentation of progress. This can only be guaranteed by a professional framework such as that provided by the FFG.”
In-depth checks websites
For the in-depth checks of the websites, all requirements listed in Table A.1 of Annex A of EN 301 549 V3.2.1 (2021-03) (PDF) (external link) were checked. For documents (PDF documents or similar), the criteria set out in Chapter 10 of EN 301 549 V3.2.1 (2021-03) were checked. These largely correspond to the WCAG success criteria of version 2.1, conformity level A and AA. A detailed table of equivalents for unfolding is available below.
According to EN 301 549, the following WCAG criteria are specifically excluded for the check of documents:
- 2.4.1 Bypass blocks
- 2.4.5 Multiple ways
- 3.2.3 Consistent navigation
- 3.2.4 Consistent identification
All websites checked in detail were checked for compliance with the WCAG criteria by means of detailed manual checks. Blind and visually impaired testers were involved in the checks.
"Comprehensive accessibility is the basis for a self-determined and equal-opportunity life. Especially in the digital sector, there is great potential to improve processes for blind and visually impaired people – but unfortunately, the opposite is often the case. That is why it is important to involve experts in the planning phases in order to exclude barriers and errors from the outset. For existing websites and applications, errors can be detected and subsequently improved with the help and expertise of blind screen reader users. Integrating them into accessibility checks is therefore a safe path towards digital accessibility!”
The checks were based on the WCAG-EM (Website Accessibility Conformance Evaluation Methodology) (external link). For this purpose, the following test steps were carried out:
- Use of analysis tools for a first impression of the structure and structure of the pages
- Visual and auditory examination (contrasts, examination of multimedia content, etc.)
- Analysis of the source code
- Verification of keyboard operability
- Verification of gesture and motion control (multipoint or path-based gestures, alternatives to control by moving the device)
- Testing with screen readers
The following tools were used to support this:
- QualWeb, W3C Markup Validation Service, Colour Contrast Analyser (CCA), Text Spacing Bookmarklet, Taba11y
- Screenreader: NVDA, TalkBack and VoiceOver, JAWS (in Tyrol)
For the additional criteria from EN 301 549, it was checked whether an error or no errors could be found for the respective criterion. Not every, possibly existing error was documented, but only the first error found, or, if no error was found in total, the criterion was rated as "fulfilled".
Sampling of the subpages to be checked
For the incoming checks, the following subpages of each website were selected for review (if available):
- Homepage, registration, sitemap, contact, help and pages with legal information
- At least one relevant page for each type of service provided by the website and for each other main purpose including search function
- The pages with the accessibility statement or information and the pages with the feedback mechanism
- Five sample pages with a distinctly different look and feel or different types of content
- At least one relevant retrievable document, if any, for each type of service provided by the website and for each other main purpose
- Randomly selected pages of at least 10% of the sample defined by the previous steps
If any of the subpages selected above contains a step in a procedure, all steps of the procedure are checked. This means that all the subpages necessary to go through the entire process are checked.
Correlation table in-depth checks websites
Clauses from EN 301 549 with the number 9 refer to classic websites, clauses with the number 10 to documents. In addition, further criteria from EN 301 549 were reviewed. Note: Appropriate tools were used to support the manual checks carried out.
Simplified checks websites
For the simplified checks, a selection of the WCAG success criteria was checked. The checks were limited to 13 WCAG success criteria, which can be tested at least partially automatically. They include success criteria for all 4 principles: perceptible, operable, understandable and robust. The success criteria depend on the needs of the users with regard to barrier-free access. The "Simplified Checks Websites Correspondence Table", which is available below for unfolding, describes the relationship with the selected success criteria in accordance with EN 301 549. For the federal government, only classic web content was checked; Tyrol also checked PDF documents during the simplified checks, if available.
The simplified checks are non-conformity checks. Due to the automated checks of a selection of WCAG success criteria, compliance cannot be determined. In addition, the criteria are usually not fully checked, but a number of individual automated checks are carried out. These are listed in the column "Qual Web Rule ID" in the "Simplified Checks Websites correlation table below". In the case of the simplified checks, it is therefore checked whether errors can be found with the individual automated checks for the selected criteria, and not whether criteria are met.
Explanations of the checks mentioned in the correlation table can be found on the website of the ACT-Rules Community (external link). The respective declaration is linked to each rule.
Tyrol also carried outpartial manual checks for the simplified checks on the accessibility declaration, imprint and for existing contact functionalities.
The following test tools were used during the three monitoring periods:
- Monitoring period 2022:
- QualWeb Core (version 0.7.10, 23.11.2021)
- Tyrol: QualWeb and additional HTML Validator from W3C, Koa11y, WAVE-Chrome Extension, JAWS-Fusion 2022
- Vienna: Google lighthouse
- Monitoring period 2023:
- QualWeb Core (version 0.7.28, 21.11.2022)
- Tyrol: QualWeb and additional HTML Validator from W3C, WAVE- Chrome Extension, JAWS-Fusion 2023
- Vienna: QualWeb and Google Lighthouse
- Monitoring period 2024:
- QualWeb Core (Version 0.7.46, 01.12.2023)
- Tyrol: QualWeb and additionally Koa11y, WAVE-Chrome Extension, JAWS-Fusion 2024
- Vienna: QualWeb and Google Lighthouse
Sampling of the subpages to be checked
For the simplified checks, the following subpages of each website were selected for review:
- Homepage
- A number of subpages proportionate to the estimated size and complexity of the website: In addition to the start page, 15 to 20 subpages were also selected for the examination. Based on the main navigation, subpages were selected that have a different appearance or functionalities than the subpages previously included in the sample (e.g. forms, actionable elements, images, tables, dynamic content, structure, other layout).
In Tyrol, subpages with different functionalities or different technical implementations as well as a PDF file were also selected. 5 to 8 subpages were checked.
Correlation table simplified checks websites
In-depth checks apps
For the in-depth checks of the apps, all requirements listed in Table A.2 of Annex A to EN 301 549 V3.2.1 (2021-03) were checked. For documents (PDF documents or similar), the criteria set out in Chapter 10 of EN 301 549 V3.2.1 (2021-03) were checked. These largely correspond to the WCAG success criteria of version 2.1, conformity level A and AA. A detailed correlation table is available below for unfolding.
As with the incoming checks of the websites, the following WCAG criteria are excluded from the check of documents according to EN 301 549:
- 2.4.1 Bypass blocks
- 2.4.5 Multiple ways
- 3.2.3 Consistent navigation
- 3.2.4 Consistent identification
In addition, according to EN 301 549, the following WCAG criteria are excluded when checking apps:
- 2.4.1 Bypass blocks
- 2.4.2 Page titled
- 2.4.5 Multiple ways
- 3.1.2 Language of Parts
- 3.2.3 Consistent navigation
- 3.2.4 Consistent identification
All in-depth checked apps were checked for compliance with the criteria by means of detailed manual checks. Blind and visually impaired testers were involved in the checks. The checks were based on the WCAG-EM (Website Accessibility Coonformance Evaluation Methodology) (external link). For this purpose, the following test steps were carried out:
- Use of analysis tools (where possible) for a first impression of the apps
- Visual and auditory examination (contrasts, examination of multimedia content, etc.)
- Usually no access to the source code possible, only with Progressive Web Apps a code analysis can be made
- Verification of gesture and motion control (multipoint or path-based gestures, alternatives to control by moving the device)
- Testing with screen readers
The following tools were used to support this:
- Android
- Accessibility scanner
- Color Picker and WebAIM Contrast Checker
- Screenreader TalkBack
- iOS
- Color contrast
- Screenreader VoiceOver
For the additional criteria from EN 301 549, it was checked whether an error or no errors could be found for the respective criterion. Not every, possibly existing error was documented, but only the first error found, or, if no error was found in total, the criterion was rated as "fulfilled".
Sampling of screens to be checked
For the incoming checks of apps, the following screens of the individual apps (if available) were selected for review – the selection was analogous to the incoming checks of the websites:
- Start screen, screens with login area, screen overview (sitemap), contact, help and screens with legal information
- At least one relevant screen for each type of service provided by the app and for each other main purpose including search function
- The screens with the accessibility statement or information and the screens with the feedback mechanism
- Five sample screens with a distinctly different look and feel or different types of content
- At least one relevant retrievable document, if any, for each type of service provided by the App and for each other main purpose
- Randomly selected screens of at least 10% of the sample defined by the previous steps
If any of the subpages selected above contains a step in a procedure, all steps of the procedure are checked. This means that all screens necessary to go through the entire process are checked.
Correlation table in-depth checks apps
The majority of clauses in EN 301 549 with clause 11 refer to classic apps, clauses with clause 10 to documents. In addition, further criteria from EN 301 549 were reviewed. Note: Appropriate tools were used to support the manual checks carried out.
Communication with sampled entities
For all three types of checks (simplified checks websites, in-depth checks websites and in-depth checks apps),the results were documented in the form of a report using the WAD Report Tool (external link). The result of the audit was sent to the audited entity after a quality check.
Tirol submitted a more detailed report to the simplified audited institutions. On the one hand, this is an error report with background information, but also contains recommendations for action and drafting suggestions for the accessibility declaration.
In order to ensure that there were no delays during the audit process, the incoming checks contacted the entities immediately after the sampling. On the one hand, the responsible contact person was determined and on the other hand, access data were queried if necessary.
Particularly the in-depth audited institutions were offered consultations, which were gladly accepted by the public institutions. These appointments lasted about 90 minutes. Parts of the report were discussed and concrete questions of the institutions or the supervising web agencies could be answered.