Abstract
There are a number of elements within Web pages that currently make the World Wide Web difficult for people with disabilities to use, but there are also simple techniques that can be used to improve accessibility. This paper will describe a study which evaluates the Web Page Author Guidelines and Checklist of the Web Accessibility Initiative. These are currently being developed to assist Web page authors in making their pages accessible to people with disabilities. It is often the visual nature of much of the content of the Web that makes access difficult for blind people, and so it is this group that this study focuses on. The study involves both Web page authors and blind people. The three aims of the study are to investigate: (a) whether Web page authors are able to use the Guidelines effectively; (b) whether the Guidelines and Checklist can be used to evaluate the accessibility of existing Web pages and suggest possible improvements; and (c) whether Web pages designed to the Guidelines are accessible to blind people. In order to investigate (a) and (b) two groups of students will use the Guidelines and Checklist to design or evaluate web pages. Blind people will then evaluate the students’ Web pages in order to examine (c).
Introduction
There are a number of elements within Web pages that are currently making the World Wide Web difficult for people with disabilities to use, but there are also simple techniques that can be used to make pages more accessible. This paper will describe a study which evaluates the Web Page Author Guidelines [10] and associated Checklist [9]. These Guidelines were originally developed by the Trace Centre [7] and are now the responsibility of the Web Accessibility Initiative (WAI) [8], which is a project of the World Wide Web Consortium (W3C) [11]. The Guidelines are being developed to assist Web page authors in making their pages accessible to people with disabilities.
It is important to evaluate guidelines for two reasons. Firstly to test whether they provide sufficient and relevant advice to the designer, and secondly to test whether the result is usable by the intended users. Therefore this study will evaluate both of these aspects of the WAI Guidelines: whether Web page authors are able to implement the advice contained in the guidelines effectively; and whether pages that are designed to the Guidelines are accessible to blind people.
There are two main uses of Web page guidelines: the design of new Web pages; and the evaluation and adaptation of existing web pages. Therefore this study will examine the use of the Guidelines by two groups of students, of Computer Science and Human Computer Interaction (HCI), who are learning about the Web. These students will be designing new pages or adapting existing pages, and then evaluating the Web pages for accessibility for blind people and usability for everyone.
The Computer Science students have experience of creating Web pages and have had a lecture on general issues of accessibility and specific techniques. The group are now either designing new pages or adapting existing pages using the Guidelines and Checklist. The HCI students have also been taught about accessibility issues, in terms of the usability of the Web. This group have been given the Guidelines and Checklist with the task of evaluating an existing Web site for both usability and accessibility. Both groups of students will test their pages using a text-based browser (Lynx [5]) and an automatic accessibility checker (Bobby [1]).
A Web site containing the pages produced by the students will then be evaluated by a range of blind people with different levels of experience of the Web. The task will involve visiting the Web site and then completing a questionnaire giving their opinions and experiences of the accessibility of the site. Whether pages are accessible to a blind person may depend on the browser and screenreader they are using. Therefore people will be asked about the software they are using which will allow examination of whether particular browsers, screenreaders (and combinations of the two) affect Web accessibility. This involvement of blind people will investigate whether the students’ interpretation of the Guidelines and Checklist produce accessible Web pages.
This study is set within the context of previous investigations of the use of guidelines. Mosier & Smith [6] conducted a survey of recipients of a report on guidelines for designing user interface software. While respondents reported that they had found the guidelines useful, they also reported that they had difficulties with the practical application of the guidelines. Specifically these problems were: locating relevant guidelines in the report, choosing which guidelines to use, establishing priorities amongst guidelines, and translating ‘general’ guidelines into specific design rules. The current evaluation will investigate whether the students using the Guidelines encounter the same difficulties. Hammond, Gardiner, Christie & Marshall [3] discuss the problem of guidelines having to be general enough to be applicable in most situations which makes them too general to be applied in any specific situation. Löwgren & Nordquist also found that it is difficult to capture context dependencies in general guidelines [4]. De Souza & Bevan [2] found that designers had difficulty in interpreting over 90 per cent of the guidelines provided for an experimental task.
While this previous work has investigated the use of guidelines for the design of interfaces and also the attitudes of designers towards the guidelines, there has been very little work on the evaluation of interfaces designed using guidelines. The results of this study will not only be relevant to the use of guidelines for the design of accessible web pages, but also the design of other user interfaces. The WAI encourages input from all interested parties and so the results of this study will be fed into the ongoing process of the development of the Guidelines and Checklist.
References
[ Evaluation ] [Proceedings]
There are a number of elements within Web pages that currently make the World Wide Web difficult for people with disabilities to use, but there are also techniques that can be used to improve accessibility. This position paper will describe two studies currently in progress which evaluate the Web Page Author Guidelines and Checklist which are under development by the Web Accessibility Initiative. They are being developed to assist Web page authors in making their pages accessible to people with disabilities. It is often the visual nature of much of the content of the Web that makes access difficult for blind people, and so it is this user-group on which these studies focus. The current studies involve 3 groups: Web page authors; Web page evaluators; and blind people. The three aims of the studies are to investigate: (a) whether Web page authors are able to use the Guidelines effectively; (b) whether the Guidelines and Checklist can be used to evaluate the accessibility of existing Web pages and suggest possible improvements; and (c) whether Web pages designed to the Guidelines are accessible to blind people. In order to investigate (a) and (b) two groups of students are using the Guidelines and Checklist to design or evaluate web pages. Blind people will evaluate the students' Web pages in order to examine (c).
There are a number of elements within Web pages that are currently making the World Wide Web (WWW) difficult for people with disabilities to use, but there are also techniques, some simple others more complex, that can be used to make pages more accessible. This position paper will describe two studies which evaluate the Web Page Author Guidelines [17] and associated Checklist [16] currently under development by the Web Accessibility Initiative (WAI) [15]. These Guidelines were originally developed by the Trace Centre [14] but are now the responsibility of the WAI, which is a project of the World Wide Web Consortium (W3C) [18]. The Guidelines are being developed to assist Web page authors in making their pages accessible to people with disabilities.
What are guidelines?
Guidelines can be conceptualised as a tool that enables the knowledge held by experts in a domain to be available to others working in the domain. This knowledge may be drawn from experience, research, and theory. For example, guidelines for the design of user interfaces are composed from the knowledge of expert user interface designers, who have experience of designing interfaces and have knowledge of psychological or ergonomic theory [3]. Guidelines for designing user interfaces can therefore be used to solve design problems when designing user interfaces [11].
There are three main uses of Web page guidelines: the design of new Web pages; the evaluation of existing web pages; and adaptation of existing web pages. These uses will vary depending on the stage of the design process, and the level of experience of the designers.
Why do we need to evaluate guidelines?
There are four aspects of guidelines that can be identified: the content; the organisation of the document and its presentation; the use of the guidelines; and the result of the use of the guidelines, i.e. the interface. The content needs to be evaluated to test whether it provides sufficient and relevant advice to the guidelines-user (at both design and evaluation stages of development). The organisation and presentation of guidelines needs to be evaluated to test whether the guidelines-user can search for, and find, the advice they require. Also, whether the guidelines-user can correctly identify the advice that is relevant to them, and apply it appropriately. The actual use of guidelines needs to be investigated to find whether the designer is able to implement the advice contained in the guidelines effectively. Finally, the usability interface needs to be evaluated to test whether it is usable by the intended users. The current studies will evaluate all of these aspects of the WAI Guidelines.
The current studies are set within the context of previous investigations of both the use of guidelines and the usability of the Web. It is acknowledged in the Human-Computer Interaction (HCI) literature on the use of guidelines that there are limitations in their use. Newman & Manning [11] believe that most limitations are in the selection and application of guidelines, not the actual content of the guidelines. The current studies are investigating these issues further. Mosier & Smith [10] conducted a survey of recipients of a report on guidelines for designing user interface software. Respondents reported that they had found the guidelines useful, but they also had difficulties with the practical application of the guidelines. Specifically these problems were: locating relevant guidelines in the report, choosing which guidelines to use, establishing priorities amongst guidelines, and translating 'general' guidelines into specific design rules. The current evaluation is investigating whether the students using the WAI Guidelines encounter the same difficulties.
Hammond, Gardiner, Christie & Marshall [5] discuss the problem of guidelines having to be general enough to be applicable in most situations which makes them too general to be applied in any specific situation. Löwgren & Nordquist [8] also found that it is difficult to capture context dependencies in general guidelines. De Souza & Bevan [2] found that designers had difficulty in interpreting over 90 per cent of the guidelines provided for an experimental task.
Jeffries, Miller, Wharton & Uyeda [6] compared the use of guidelines as a usability evaluation technique with three other evaluation techniques; heuristic, usability testing, and cognitive walkthrough. They highlight many of the issues associated with various evaluation techniques (severity of problems found, cost/benefit analyses, and evaluator experience). They found that heuristic evaluation discovered the most usability problems, and more of the most serious problems, but requires several evaluators. In terms of the ratio between ‘problems found’ and ‘time spent’ this technique was found to have a four-to-one advantage over the other methods. Usability testing found serious, recurring and general problems, but was the most expensive of the four techniques in terms of problems found per hour. The cognitive walkthrough found less general and less recurring problems than the other techniques and was found to be the most tedious. The guidelines evaluation found the most general and recurring problems compared to the other techniques, but missed some severe problems. The issues associated with guidelines (presentation, generality/specificity, selection, interpretation, application, cost/benefits) all require investigation, particularly in relation to Web page guidelines. An important question is whether the use of Web page guidelines will have the same associated problems and advantages as user interface guidelines. This question is also being asked about the Web: whether general design and usability issues also apply to the Web.
Usability of the Web is acknowledged to be an issue for all Web users and is the subject of much debate, for example [13, 12, 4]. Spool, Scanlon, Schroeder, Snyder & DeAngelo [13] have performed one of the first serious usability evaluations of the Web. They believe that it is not yet known how to design for effective information-finding, or how to use graphics, text, animation or interactivity in the most effective way. Spool et al. therefore conclude that the Web is different from ordinary software and so it cannot be designed or evaluated in the same way as software. Butler [in 12] on the other hand, disagrees and states that HCI professionals have dealt with this type of uncertainty before. User definition, task analysis, and learning are as important for the Web as for ordinary software, and that HCI has much knowledge and many techniques that can be applied to Web sites in order to improve usability. The debate on Web usability highlights that there is much work to be done on evaluating both the Web itself and the techniques used in its design and evaluation.
The three of the uses of guidelines identified in the introduction (design, evaluation, and adaptation) are being investigated during the current studies. These studies are examining the use of the WAI Guidelines by two groups of students: one group of Human Computer Interaction (HCI) students, and one group of Computer Science students, both of whom are learning about the Web.
Use of WAI Guidelines in evaluation
The HCI students have been taught about the accessibility and usability issues of the Web. This group were given the task of evaluating three existing Web sites for both usability and accessibility. The group were given a selection of relatively large, popular Web sites from which they each chose three to evaluate. In order to evaluate the accessibility of the sites, the students were given the WAI Guidelines and Checklist and a form for reporting the accessibility problems found. To evaluate the usability of the sites, they were given a set of usability heuristics that were developed specifically for evaluating the Web by Jones & Hewitt [7] and a form for recording the problems found. For each site evaluated the group completed a questionnaire assessing how much they liked each site and how easy they found the site to use. The group were asked to complete another questionnaire assessing how easy they found the Guidelines and Checklist to use and any problems they encountered during the task. The data from the questionnaires will be correlated with the findings from the usability and accessibility evaluations.
Use of WAI Guidelines in design and adaptation
The Computer Science students have experience of creating Web pages and have had a lecture on general issues of accessibility, including specific techniques for improving accessibility. The group is using the WAI Guidelines and Checklist in two different tasks: some students are adapting existing pages, others are designing new pages. The students who have developed their own web sites are evaluating their site using the WAI Checklist and, if necessary, adapting the site to make it more accessible. The students who do not have a Web site are designing new Web pages. These pages contain text, links, tables, frames, images, image-maps, etc. To save time, the students have been provided with content to include within the tables, frames etc. They are using the browsers, authoring tools and HTML guides that they normally use when creating Web pages. The students have been encouraged to test their pages using a text-based browser (Lynx [9]) and an automatic accessibility checker (Bobby [1]).
The experiment is taking place under laboratory conditions, using a technique known as 'think aloud protocol' [3]. This involves the participant telling the experimenter what they are doing, why, and describing any problems as they encounter them. The technique allows the experimenter to observe participants working which will provide useful insights into the ways in which the students use the Guidelines. The experiments are being video-recorded to allow further analysis.
The Web pages produced by the students will be collated on a Web site which will be made available for evaluation by a range of blind people with different levels of experience of the Web. The task will involve visiting the Web site and then completing a questionnaire giving their opinions and experiences of the accessibility of the site. Whether pages are accessible to a blind person may depend on the browser and screenreader they are using, therefore people will be asked about the software they are using. This will allow investigation of whether particular browsers, screenreaders (and combinations of the two) affect Web accessibility. This involvement of blind people will show whether the students' interpretation of the Guidelines and Checklist produce accessible Web pages.
There are both advantages and disadvantages of the use of guidelines in both design and evaluation. While previous work has investigated the use of guidelines for the design of interfaces, and the attitudes of designers towards guidelines, there has been very little work on the evaluation of interfaces designed using guidelines. The WAI encourages input from all interested parties and so the results of this study will be fed into the ongoing process of the development of the Page Author Guidelines and Checklist. The results will also be relevant to research on the use of guidelines in the design of user interfaces.
The initial results of these studies will be presented at the conference.
Chetz Colwell is a PhD student supported by the Economic and Social Research Council, UK, Grant No: S00429637041.
Introduction
Il existe dans les pages Web un certain nombre d'éléments qui rendent le World Wide Web difficile à utiliser pour les personnes handicapées, mais il existe également des techniques simples pouvant être utilisées pour améliorer l'accessibilité. Cet article décrira une étude qui évalue les Guides de conception pour les Auteurs de Pages Web [10] et la Checklist qui va avec. Ces Guides de conception ont été mises au point à l'origine par le Trace Centre [7] et relèvent désormais des responsabilités de l'Initiative pour l'Accessibilité du Web [8], qui est un projet du World Wide Web Consortium (W3C) [11]. Ces guides de conception sont en cours de mise au point afin d'aider les auteurs de pages Web à rendre leurs pages accessibles aux personnes handicapées.
L'évaluation des guides de conception est importante pour deux raisons. D'une part, pour vérifier si elles fournissent au concepteur des conseils suffisants et pertinents, d'autre part pour vérifier que le résultat obtenu peut être utilisé par les utilisateurs auxquels il est destiné. Cette étude procédera donc à l'évaluation de ces deux aspects des Guides de conception WAI : si les auteurs de pages Web peuvent appliquer efficacement les conseils contenus dans les guides de conception, et si les pages élaborées conformément à ces guides de conception sont accessibles aux non-voyants.
Les guides de conception pour les pages Web ont deux usages principaux : l'élaboration de nouvelles pages Web ; et l'évaluation et l'adaptation des pages Web existantes. Par conséquent, cette étude examinera l'utilisation des Guides de conception par deux groupes d'étudiants en Informatique et Interaction Homme-Ordinateur (HCI), qui étudient le Web. Ces étudiants élaboreront de nouvelles pages ou adapteront des pages existantes, puis évalueront l'accessibilité aux non-voyants de ces pages Web et leur facilité d'utilisation pour tout le monde.
Les étudiants en informatique ont de l'expérience dans la création des pages Web et ont reçu un cours sur les problèmes généraux de l'accessibilité et les techniques spécifiques. Ce groupe est actuellement en train soit d'élaborer de nouvelles pages, soit d'adapter des pages existantes en utilisant les Guides de conception et la Checklist. Les étudiants en HCI ont également été formés aux problèmes d'accessibilité, en termes de facilité d'utilisation du Web. Ce groupe a reçu les Guides de conception et la Checklist avec pour tâche d'évaluer la facilité d'utilisation et l'accessibilité d'un site Web existant. Les deux groupes d'étudiants testeront leurs pages en utilisant un navigateur à base textuelle (Lynx [5]) et un contrôleur d'accessibilité automatique (Bobby [1]).
Un site Web contenant les pages réalisées par les étudiants sera ensuite évalué par divers non-voyants avec différents niveaux d'expérience du Web. Leur tâche consistera à visiter le site Web puis à remplir un questionnaire où ils feront part de leur avis et de leurs expériences de l'accessibilité du site. L'accessibilité des pages à un non-voyant peut dépendre du navigateur et de l'éditeur d'écran qu'il utilise. Les participants seront donc interrogés sur le logiciel qu'ils utilisent, ce qui permettra d'examiner si certains navigateurs et certains éditeurs d'écran (et les combinaisons des deux) ont une influence sur l'accessibilité du Web. Cette participation de non-voyants démontrera si l'interprétation par les étudiants des Guides de conception et de la Checklist a débouché sur des pages Web accessibles.
Cette étude se place dans le contexte de recherches antérieures sur l'utilisation de guides de conception. Mosier et Smith [6] ont mené une enquête auprès des destinataires d'un rapport sur les guides de conception quant à la conception des logiciels d'interface utilisateur. Quoique les personnes interrogées aient déclaré avoir trouvé les guides de conception utiles, elles ont aussi signalé avoir rencontré des difficultés dans l'application de ces guides de conception en pratique. Ces problèmes étaient en particulier : identifier les guides de conception pertinentes dans le rapport, choisir quelles guides de conception utiliser, établir des priorités entre ces guides de conception, et traduire les guides de conception "générales" en règles de conception précises. L'évaluation actuelle cherchera à savoir si les étudiants rencontrent des difficultés semblables dans l'utilisation des guides de conception. Hammond, Gardiner, Christie et Marshall [3] discutent le problème de la nécessité d'avoir des guides de conception assez générales pour être applicables dans la plupart des situations, ce qui rend ces guides de conception trop générales pour être appliquées à telle ou telle situation précise. Löwgren et Nordquist ont aussi constaté qu'il est difficile d'isoler des dépendances contextuelles dans des guides de conception générales [4]. De Souza et Bevan [2] ont constaté que les concepteurs rencontraient des difficultés dans l'interprétation de plus de 90% des guides de conception fournies pour une tâche expérimentale.
Quoique les travaux antérieurs aient examiné l'utilisation de guides de conception pour la conception d'interfaces ainsi que l'attitude des concepteurs vis-à-vis de ces guides de conception, très peu de travail a été fait sur l'évaluation des interfaces élaborées selon ces guides de conception. Les résultats de cette étude seront pertinents non seulement en ce qui concerne l'utilisation des guides de conception pour l'élaboration de pages Web accessibles, mais aussi pour l'élaboration d'autres interfaces utilisateur. Le WAI encourage toutes les parties intéressées à fournir des données, et les résultats de cette étude viendront donc s'ajouter au processus de mise au point des Guides de conception et de la Checklist qui se poursuit actuellement.
Références
[1] Contrôleur d'accessibilité Bobby disponible à : http://www.cast.org.bobby/ (12 fév 98)
[2] de Souza, F., et Bevan, N. (1990) The use of guidelines in menu interface design: evaluation of a draft standard. Dans D. Diaper, D. Gilmore, G. Cockton, et B. Shackel (Eds) Human Computer Interaction - INTERACT ‘90, pp 435-440.
[3] Hammond, N., Gardiner, M., Christie, B., & Marshall, C. (1987) The role of cognitive psychology in user-interface design. Dans M. Gardiner & B. Christie (Eds) Applying cognitive psychology to user-interface design, John Wiley & Sons, Chichester, pp 13-53.
[4] Löwgren, Jonas et Nordquist, Tommy (1992) Knowledge-based evaluation as design support for graphical user interfaces in Proceedings of CHI ‘92 pp 181-188.
[5] Explorateur Lynx disponible à : http://www.delorie.com/Web/lynxview.html (12 fév 98)
[6] Mosier, Jane, N., et Smith, Sidney, L. (1986) Application of guidelines for designing user interface software. Behaviour and Information Technology, Vol 5, No 1. pp 39-46.
[7] Page d'accueil de l'Initiative pour l'Accessibilité du Web, disponible à : http://www.w3.org/WAI/ (12 fév. 98)
[8] Checklist pour les auteurs de pages, de l'Initiative pour l'Accessibilité du Web, disponible à : http://www.w3.org/WAI/GL/authorcl.htm (12 fév. 98)
[9] Guides de conception pour les Auteurs, disponibles à : http://www.w3.org/WAI/GL/author.htm (12 fév. 98)
[10] Page d'accueil du World Wide Web Consortium, disponible à : http://www.w3.org/ (12 fév.98)
at: http://www.w3.org/ (12 Feb 98)
[Actes]