SEARCH WITHIN CONTENT
Citation Information : International Journal of Orientation & Mobility. Volume 8, Issue 1, Pages 37-61, DOI: https://doi.org/10.21307/ijom-2017-066
License : (CC-BY-NC-ND-4.0)
Published Online: 16-April-2018
A project team at Guide Dogs NSW/ACT, Australia designed, tested, and statistically validated over a five-year period, a client evaluation tool to assess the progress or achievement of adult clients (18+ years) who participate in orientation and mobility (O&M) programs (including Guide Dog programs). The tool is administered at the commencement of the client’s mobility program and again at its conclusion to provide a pre/post measure. The tool comprises two parts. Part 1, completed by the O&M specialist, measures the degree of vision impairment; client complexity, and the environmental complexity in which the training is being conducted. Part 2 includes the mobility specialist’s rating of the client’s skill and confidence level; and the client’s rating of their own skill and confidence as well as a rationale for their ratings. The tool appears effective in measuring client outcomes for all types of O&M program in any environment; is quick and easy to use, and non-invasive. The paper includes the results of the tool applied by 57 O&M specialists to 361 clients from July 2014 to July 2015 revealing significant improvements in skill and confidence post-program.
There has been considerable anecdotal evidence reporting that people with vision impairment often appear more skillful, mobile, and confident as a result of orientation and mobility (O&M) training (Edwards, 2002; Howe, 1872; Lloyd, La Grow, Stafford, & Budge, 2008; Malamazian, 1970; The Seeing Eye, 2007, as cited in Franck, Haneline, Brooks, & Whitstock, 2010). However, proving this has been somewhat difficult. Deducing the way to measure O&M outcomes that can be applied to many types of mobility programs (e.g., Guide Dog, GPS, neurological) and administered by O&M specialists in a succinct, easy to use, and meaningful way has been challenging. Only in recent years have O&M outcome tools begun to be adequately researched and developed (La Grow, Ebrahim, & Towers, 2013-14; La Grow, Towers, Kim, & Haneline, 2015).
The importance of measuring client outcome cannot be overemphasised. It is central to demonstrating whether or not an O&M program benefits a person with vision impairment (National Resource Center, 2010, p. 10). Outcome measures can be used to report on the quality of the training provided and as such, could be an essential reporting item to government, funding agencies, corporate supporters, major donors, and stakeholders to secure the future of O&M services and the organisations that provide them. In an increasingly competitive O&M service environment (like that which currently exists in Australia) applying O&M outcome tools to programs will assist to identify quality service providers.
To date, only two O&M outcome measures have been developed, piloted, and validated albeit using small sample sizes. First, La Grow, Ebrahim, and Towers (2013-14) developed a 23-item difficulty with mobility questionnaire (DMQ) involving 32 participants. The DMQ is based on the premise that the purpose of O&M is to restore a person’s ability to travel around environments using adaptive aids, and a range of techniques and strategies, thereby enabling reduced difficulty with the 23 items (e.g., crossing busy streets with pedestrian control devices; getting around in the immediate neighbourhood; getting around in supermarkets; negotiating parking lots; travelling in unfamiliar outdoor environments; using public transportation). The DMQ assesses the person’s perception of difficulty performing each of the tasks before and after O&M training. Clients are also asked to rate their ability to get around (AGA). The DMQ is intended to be an indicator of O&M specialist effectiveness although this version is not reflective of a complete mobility program, but rather only aspects of one. The authors stated that they intend to investigate the feasibility of developing a briefer version of the DMQ for actual O&M program use validated on a larger sample of participants.
Second, in a later study, the DMQ was tested as an outcome measure for Dog Guide instruction involving 49 participants (La Grow, Towers, Dae Kim, and Haneline, 2015). The authors did acknowledge the few limitations of the study, for example, their use of convenience sampling and the variation of intervention in the content between first time dog guide users and those training with a replacement dog.
In 2011, a small project team at Guide Dogs NSW/ACT commenced the development of a client evaluation tool (Appendix A). The major motivation for its development was to increase results-based accountability that would feed into the organisation’s key performance indicators, assist strategic direction, and increase service outcomes for clients. The initial project team comprised an external consultant, five clients, four O&M specialists, and an allied health specialist with all specialists having 20 or more years of experience, who were also managers, and researchers. The project team deduced that an effective tool must be administered pre- and post-training, apply to all types of O&M programs, take into account the judgement and feedback of O&M specialists and clients; be quick and easy to use; and consider the complexity of clients who have additional disabilities, and the complexity of the training environment (Deverell, 2011). Information about these components was to be gathered during the client assessment and/or during the initial demonstration of mobility skills by clients prior to a formal O&M training program. This information could be confirmed via client medical reports, a declaration by the client, or through O&M specialist observation.
The major components of the tool follow:
Vision impairment: includes low vision, legally blind, or blind. Definitions for each are provided to guide O&M specialists.
Client complexity that impacts the service request objective (or the goal of the O&M program): many clients experience an additional disability other than vision impairment which might impact their O&M program. For example, if a client experiences memory problems as well as vision impairment, then it might take that client longer to complete their O&M program; they might need to incorporate additional strategies into their program; or it might prevent the client from completing the program. This information is helpful to assist service planning, reflective practice, and O&M specialist training.
Environmental complexity of the training environment: the rating is based on the highest level of environmental complexity in which training will occur (Deverell, 2011). This rating might change from pre- to post-program when, for example, a client’s performance exceeds expectation and the client decides to use the GPS aid in more complex environments rather than just in the quieter home suburb as originally planned.
Skill and confidence rating: there is no standardised measure available to rate overall client performance in O&M (Virgili & Rubin, 2006). Only various aspects of O&M have been successfully measured such as walking speed, and avoiding or contacting objects and drop-offs on a path of travel (Clark-Carter, Heyes, & Howarth, 1986; Kim, Wall Emerson, & Curtis, 2010; Tellevik, Martinsen, Storlilokken, & Elmerskog, 2000). However, it has been suggested that a person’s level of confidence and skill is linked to performance outcome (Beggs, 1992; Kanazawa, 2004, 2010; Oney & Oksuzoglu-Guven, 2015). Oney and Oksuzoglu-Guven (2015) discussed ‘specific self-confidence’ being based on specific experience. Specific self-confidence is defined as a person’s self-stated confidence in his abilities in a specific context. Specific confidence can increase when a person has repeated success in a specific skill (e.g., long cane training) which has the effect of reducing the person’s anxiety. Thus, skill and confidence appears to be a reliable indicator of O&M performance.
The first three versions of the CET (from Jan to Sept 2012) included the Personal Wellbeing Index – Adult Version (PWI-A) (International Wellbeing Group, 2006). The rationale for its inclusion was that O&M training might have the added advantage of influencing a client’s feeling of wellbeing or satisfaction with life. However, pilot results indicated that this was not the case and the PWI was a cause of distress for a majority of clients. A majority of clients opted not to complete the PWI commenting it was intrusive and irrelevant to their O&M program. The few clients who did attempt to complete the PWI-A objected to or commented on the intrusive nature of the questions, in particular: ‘how satisfied are you with your personal relationships’ and ‘how satisfied are you with your future security?’ Therefore, this measure was deleted from the CET. The majority of clients appeared happy to complete their section of the CET as long as it was quick and easy and did not ask questions unrelated to their O&M training. It might be important to note that the majority of clients had been vision impaired for some time and had not recently experienced sudden vision loss from a stroke or other cause. It appeared clients had adjusted, to varying extents, to their vision impairment and were building on their existing mobility skills. Perhaps those people who have suddenly lost their vision, who have not yet developed strategies to cope, or skills to get around, would most likely report increases in quality of life pre- and post-O&M training. To date, no research has been conducted to identify under what conditions, if any, lifestyle scales measuring quality of life indicate O&M intervention effectiveness.
A total of three CET pilot evaluations occurred between 2012 and 2013 including testing for validity and reliability. Each version of the tool was refined based on pilot data outcome, and client and O&M specialist feedback. The pilot details follow:
(i) Jan-March 2012 Pilot 1: Version 1: over three months, involving four O&M specialists and eight clients, Sydney
(ii) April-June 2012 Pilot 2: Version 2: over six weeks, involving seven O&M specialists and 11 clients, metro and regional areas
(iii) July-Sept 2012 Pilot 3: Version 3: over six months involving seven O&M specialists and 30 clients, metro and regional areas
(iv) April-Sept 2013 Pilot 4: Version 4: over six months, involving all O&M specialists in all regions and 89 clients (Gallimore, Tinsley, Keay, Borkowski, & Hill, 2015). This version included a single-item measure of the ability to get around (AGA) taken from the World Health Organization Quality of Life BREF (WHO, 1996).
(v) October 2014 (Final) Version 5: Included Version 4 without the AGA measure.
The final version of the CET comprises five pages (Appendix A). Pages 1-4 are completed by the O&M specialist. Page 1 includes demographic information about the client, the region in which the O&M specialist works, and details related to the training program. It also includes a brief description of the CET. Page 2 requires information about the client’s level of vision impairment and client complexity. Page 3 requires the rating of the highest environmental complexity in which the training is to be conducted. If the environment changes in complexity from pre to post-measure, then it is noted on this page. Page 4 requires a rating by the specialist about the client’s skill and confidence level as well as a rationale for this rating. Page 5 requires a client rating of their skill and confidence level as well as a rationale for this rating. Detailed instructions for using the tool are also available.
The final version of the tool was rolled out across Guide Dogs NSW/ACT in June 2014. A one-year analysis of data occurred from July 2014 to July 2015.
While this evaluation tool was developed as part of quality assurance activities at Guide Dogs NSW/ACT (2012-2014), approval was sought and granted for a research study to validate the CET through the University of Sydney Human Research Ethics Committee (Project number: 2014/915).
In this research study 24 O&M specialists administered the CET to 40 clients participating in a variety of O&M programs. Construct validity was examined against the global question ‘how well are you able to get around?’ that is a standard global measure (La Grow, Alpass, Stephens, & Towers, 2011; Yeung, La Grow, Towers, Alpass, & Stephens, 2011).
Pre-program, specialists applied the CET twice with clients (with 3-5 days in between each CET as a test for repeatability) and at the completion of the program to evaluate test-retest agreement. After each CET was completed, it was posted to the data administrator so that each CET result could not be compared.
CET results for all adult clients receiving an O&M program during one year from Guide Dogs NSW/ACT specialists was used to investigate the responsiveness of the instrument in evaluating O&M programs. The impact of client specific and environmental factors was investigated.
The test-retest repeatability for the four items in the CET was evaluated using agreement and a weighted Kappa statistic. The Kappa statistic was interpreted according to Viera and Garrett (2005) where 0.61-0.80 represents substantial agreement and 0.81-0.99 almost perfect agreement. Construct validity was investigated by comparing change in the CET items against the global question from the WHOQOL BREF. The responsiveness of the CET was assessed using the Wilcoxon Signed Rank sum test to compare pre/post-program measures for each item separately. The predictors of significant improvement (2 points or more on 5-point scale) for each item were modelled using logistic regression. Any factors significantly associated (p<0.20) were considered in a multivariate model and removed using step-wise backward elimination to determine a final multivariate model predicting improvement.
There were 40 clients in the validity study, 18 men and 22 women, aged 52 years on average (range 18-85). There were 11 clients (28%) with low vision, 23 (58%) that were legally blind and six (15%) with no vision or light perception only. Agreement was 90-100% and the weighted Kappa statistic was >0.80 for all items on the CET, indicating almost perfect agreement, high repeatability, and that specialist and client ratings were consistent.
There were measurable improvements (Wilcoxon Signed Rank sum test p<0.0001) in instructor-rated confidence (1.35±0.92) and skill (2.88±1.11) and client-rated confidence (1.60±1.10) and skill (1.93±1.33). These changes in client confidence and skill levels pre- and post-program were highly significant, suggesting O&M goals were attained resulting in increased levels of client confidence.
There was moderate correlation as shown by the Pearson’s Correlation Coefficient between improvements on the global question ‘how well are you able to get around?’ and improvement on the items on the CET: instructor-rated skill (0.52) and confidence (0.57) and client-rated skill (0.66) and confidence (0.52) indicating construct validity. Importantly, the CET measures two specific constructs (confidence and skill) whereas the ‘ability to get around’ is subjective to interpretation by the client.
All adult clients (18+ years) requiring O&M training were included in the analysis (n=361). Clients were both existing clients requiring an O&M program to learn a new skill, as well as people new to the service requiring formal O&M training. Only those clients whose programs were completed were included in the analysis. Fifty-seven O&M specialists provided the O&M training. The analysis involved clients and specialists from all seven office regions throughout NSW and ACT. The characteristics of the group are summarised in Table 1.
One post-program evaluation was missing therefore, graphs show data for 361 clients pre-program and 360 post-program. On average, the instructor-rated skill score improved by 1.6 points on the 5-point scale (1.6±1.3, mean ± standard deviation p<0.0001) and the instructor-rated confidence score by 0.8 points (0.8±1.0, p<0.0001). Thus, the improvement in skill and confidence rating was statistically significant for pre- and post comparisons with approximately two-thirds of clients being rated as independent and half ‘very good’ for confidence (Figure 1).
There were 22 clients who did not rate their skill or confidence on at least one of the assessments so data are presented for 344 clients before their program and 345 after their program. The clients rated their skill as on average 1.3 points higher (1.3±1.3, p<0.0001) and confidence 1.0 points higher (1.0±1.3, p<0.0001) after the program. Thus, the improvement in skill and confidence rating was statistically significant for pre and post comparisons and around 50% used the highest rating on the 5-point scale for these items after the program (Figure 2).
The majority of programs took place in the larger metro regions. The majority of clients trained in dynamic environments participating in a wide variety of O&M programs (Table 2).
Client characteristics and environmental factors were considered as candidate predictive factors. The data are odds ratios and their 95% confidence intervals (Table 3) represent the likelihood of improvement in the CET items, where an odds ratio <1 implies that the factor reduces the likelihood of a 2-point improvement and an odds ratio >1 increases the likelihood of an improvement. The odds ratios that are statistically significant have 95% confidence intervals that do not overlap 1 and are highlighted in bold font (Table 3).
Data are odds ratios with 95% confidence intervals. An odds ratio <1 implies that the factor reduces the likelihood of a 2-point improvement. An odds ratio >1 increases the likelihood of an improvement. The odds ratios that are statistically significant have 95% confidence intervals that do not overlap 1 and are highlighted in bold font (Table 3).
Age was independently predictive of a 2-point improvement on the instructor-rated scale for skill whereby older clients were 6% less likely to have this level of improvement per 5 years older (OR 0.94, 95% CI 0.89-0.995). The other factor predicting 2-point improvement was client complexity whereby more complex clients were more likely to make substantial gains in skills. The 2-point improvement in instructor-rated confidence was 30% less likely in male clients (OR 0.70, 95% CI 0.54-0.91) but three times more likely for the small proportion of clients (5%) with communication difficulty (OR 3.02, 95% CI 1.17-7.80).
For the client-rated scores, there were no client specific or environmental factors which influenced the chance of a 2-point improvement on skill but complex environments reduced the likelihood of 2-point improvement in confidence by approximately 85% (OR 0.85, 95% CI 0.74-0.98).
Of the 361 clients there were partially more females than males (male 44%; female 56%) with an average age of 57 years. The majority of clients were either legally blind (45%) or blind (35%), with 22% of clients having one complexity that impacted the training program and 11% who had two or more complexities, for example, a physical impairment (e.g., affected limbs, arthritis, pain), memory or problem solving issues, or another condition (e.g., hearing impairment, musculoskeletal disease, diabetes, Usher syndrome).
Clients participated in 16 types of O&M programs with the majority of clients requiring orientation training (32%) to shops and shopping centres, across roads, new workplaces, universities, and meeting places (e.g., community centres). The majority of clients requiring orientation training were independently using a primary mobility aid (long cane or guide dog). The other major training programs were guide dog (16%), long cane (14%), and support cane (8%). The majority of the programs were taught in complex and dynamic environments (88%) in metropolitan regions (75%).
Pre-program, specialists rated client skill at a lower level than did clients. A majority of specialists indicated that clients would need verbal prompting and observation to acquire the O&M skill. Specialists used their experience to make the rating and were perhaps more objective in their rating than clients. Clients who had previously received O&M training programs over two or more years rated their skill level to achieve their current O&M goal as ‘average’ or ‘good’. These clients had previously experienced successful O&M programs and believed they had O&M skills that would enable them to achieve the program goal. For example, a specialist commented that:
“the client will be able to cross the intersection once they are orientated to the light pole and understand the complexities of the crossing.”
The client commented:
“if I tried to cross the intersection now I’d probably be good at it because I’ve crossed heaps of roads before. I just need you to describe the crossing and where things are so I understand it – and you [the O&M specialist] watch me do it a few times.”
Post-program, specialists rated client skill as slightly higher than clients rated themselves with the majority of clients reporting to have ‘good’ or ‘very good’ skills. Specialists observed that the majority of clients were able to perform the O&M skill independently with only some needing or wanting further observation. Further observation was often provided by partners or friends, who would eventually fade out from the program once clients believed they no longer required observation. A few clients liked being accompanied by their partners when in public despite performing the O&M skill independently (e.g., crossing roads, or travelling on public transport). A client commented:
The confidence level of clients improved significantly from pre- to post-training. The rating by specialists and clients were almost the same. The majority of clients had ‘good’ or ‘very good’ levels of confidence. A client commented:
“I can now go on the bus to the shopping centre – I don’t have any problems at all. I feel great and feel very confident.” “We [guide dog and handler] are able to get around the Uni[versity] now really easily – I feel great and very confident we won’t have any hassles …” “You’ve shown me how to use the Trekker and I feel really good about it. I feel very confident.”
It is interesting to find that client and specialist pre- and post-program rating of skill and confidence were consistent to one another. That is, pre-training clients rated their skill and confidence levels as similar and post-program similar again. Specialist ratings followed the same pattern. For instance, post-training, about 48% of clients rated their skill as ‘very good’, and 40% as ‘good’. Similarly, post-training about 51% of clients rated their confidence as ‘very good’, and 37% as ‘very good’. Further, post-training, the minority of clients who rated their skill as ‘poor’ (1%), also rated their confidence as ‘poor’ (1%) which replicated the rating of their O&M specialists. This finding seems to infer a strong correlation between skill and confidence as an indicator of O&M performance supporting the claims of Beggs (1992) and Oney and Oksuzoglu-Guven (2015).
A cross analysis of 11 variables was conducted to investigate whether or not these affected client and specialist rating levels of skill and confidence (Table 3). There might be numerous reasons for the results and a brief rationale is offered. The six statistically significant findings follow:
As clients become older, it was less likely that O&M skills would improve significantly from pre- to post-program. Many O&M specialists reported that older clients preferred to be accompanied when travelling post-program rather than travel a route by themselves independently. Clients commented that they preferred to travel with a friend or partner for company, as many of their outings were socially-based to shopping centres, or entertainment destinations. Few older clients were working full-time and did not have the desire to travel independently to and from work. Specialists tended to rate clients who preferred to be accompanied as ‘needs some observation’ rather than as ‘independent’. Comments from specialists included:
If the client had an additional complexity that impacted training (e.g., physical impairment, hearing impairment) then there was a significant improvement in skill level. This could indicate that clients with additional complexities restrict their O&M activity so that they remain less skilled than other clients who might try to compensate in other ways (e.g., by asking others for assistance, or trying to problem solve by themselves). For example, a 48-year-old male with a hearing impairment said:
“I was having trouble crossing that busy junction because I couldn’t hear all the cars coming but when I used the tactile beeper at the traffic lights – I could cross fairly easily.”
Before training his specialist rated his skill level as ‘needing total assistance’ as the client was unaware that an audio-tactile crossing existed on a crossing close-by. Post-training, the client was crossing independently without any difficulty.
Similarly, a 64-year-old female long cane user with limited spatial skills wanted to walk to her local shop though was fearful travelling the route unaccompanied in case she became disorientated. The O&M specialist taught her to consistently shoreline using many tactile landmarks to assist her orientation. Prior to the program, the specialist rated her skill level as ‘needing physical prompting’ though post-training she was independent on this route.
If a client has more than one complexity, then skill levels improve significantly. This might infer that the more additional disabilities a person has, the less skilled they are in mobility when commencing O&M programs. For instance, a 70-year-old male long cane user who is legally blind, depended on his partner when travelling (he was guided while using his long cane). He had hearing impairment, diabetes, and osteoarthritis. After the death of his partner, he wanted to continue to travel the routes he once had. He was (first) orientated on a route to his local shop. Prior to training he needed total assistance to learn orientation, however, after training he was able to travel to the shop safely and independently.
Male clients experienced less confidence than females post-program (even though both males and females experienced significant improvement in confidence post-program). Many O&M specialists rated male confidence levels as ‘very poor’ or ‘poor’ pre-program and as ‘average’ or ‘good’ post-program compared to many females rated by specialists as having ‘average’ confidence levels pre-program and ‘good’ or ‘very good’ confidence post-program. For example, an O&M specialist rated a 65-year-old man’s confidence pre-long cane training as ‘very poor’. She commented that:
“the client wants to learn to use a long cane but is very nervous and unsure about it. He feels very embarrassed and concerned what his friends and family will think when they see him use it.”
Post-training the specialist commented that:
“his confidence is good. He uses the cane well and can travel to his local shopping centre without difficulty. He is still a little worried about what his friends think of him.”
This might infer that the males were a little more self-conscious than females, or even a little more reluctant to travel without being accompanied.
Clients with communication difficulties experienced significant increases in confidence post-program. Examples of communication difficulties included having a condition that restricted speech and/or being understood by others; having a hearing impairment and not being able to hear what others were saying. Such clients typically appear to have low levels of confidence at the commencement of training. The confidence increase post-program seemed very high compared to their starting condition. Improvement in confidence might be the result of learning communication strategies as part of their O&M programs. For instance, a 72-year-old client with a significant hearing impairment commented that:
“I can’t believe I can use a mobile device to communicate with café staff and they respond so well!”
A 23-year-old woman said:
The more complex the environment the less confidence clients felt (many clients noting their confidence level as ‘average’ rather than ‘good’ or ‘very good’). It might be reasonable to assume that when clients are learning in a dynamic, uncontrolled traffic environment which includes movement by vehicles and traffic that might not give way to pedestrians – they might be less confident than a client travelling in a less complex environment. As one client commented:
“I’m able to walk to work in the city, but I still feel a little stressed about it – just because I hope drivers follow the road rules.”
A minority of clients did not complete their O&M programs during the year for numerous reasons. Some clients were unwell, some had moved to another location, and some wanted to change their O&M goal. This was noted on the CET and the clients often rated themselves at a lower level of confidence to signify they had not completed the programs.
There was a limitation in the development of the CET that requires acknowledgement. There were no control groups to ensure that the change in ratings between pre- and post-CET were the direct result of O&M intervention rather than the result of time or other factors. However, applying the CET pre- and post across a large sample of clients (n=361) most likely indicates the effect of O&M intervention.
The CET is intended to provide an outcome measure for adult O&M programs which also takes into consideration the complexities of the training environment and additional disabilities of the client that might affect the training program. The CET tool is administered before the commencement of the O&M program and at its completion. The key components are client O&M skill and confidence that appear to relate to O&M performance outcome. The CET is quick and easy to use taking less than 15 minutes to administer. The pre- and post-CET results can be compared immediately without the need for statistical analysis. Guide Dogs NSW/ACT is currently transferring the CET to a research electronic data capture system for use on mobile devices and it is being used to measure ongoing O&M performance.
Sincere thanks to Guide Dogs NSW/ACT CEO Dr Graeme White for his support throughout this project. The CET was developed by a project team that comprised clients, managers, and O&M specialists at Guide Dogs NSW/ACT. The major contributors to the development and/or piloting of the CET were Jacqueline Amos, Carolyn Bates, Ewa Borkowski, Karen Carrigan, Lisa Dillon, Sue Gosewinckel, Jeremy Hill, Jennifer Jedovnicky, Jessie Jiang, Lisa Middlemiss, Elaine Neal, Kerry Peirce, Kelly Prentice, Nicola Riley, Doug Ritchie, Christiana Savas, Jessica Taylor, Matthew Walker, and Ying Wah Wan. Thank you to all of the O&M specialists at Guide Dogs NSW/ACT who positively engaged in CET training and assisted to successfully roll out and implement the CET.