Investigating the Inter-Rater Reliability and Usability of Telehealth for Home Hazard Falls-Risk Assessment After Stroke
-
Husna Ahmad Ainuddin
,
Muhammad Hibatullah Romli
,
Tengku Aizan Hamid
,
Mazatulfazura SF Salim
,
Lynette Mackenzie
Abstract
Introduction: Stroke survivors are more likely to fall at home. A home hazard assessment may be beneficial to reduce the risk of falling; however, it is resourceful and time-intensive. This study examines the inter-rater reliability and usability of telehealth for a hazard assessment to address the risk of falls.
Materials and Methods: Two occupational therapists accessed the telehealth platform from different locations and simultaneously rated participants’ home environment using the home falls and accident screening Tool. Stroke survivors and their caregivers answered the telehealth usability questionnaire.
Results: A total of 36 stroke survivors and 31 caregivers participated in the study. Gwet’s AC1 was used for agreement analysis. The overall AC1 value for the inter-rater reliability was 0.93 (95% confidence interval [CI], 0.66%, 1.00%). There was a moderate correlation between the raters (r=0.57, P=0.000). Bland and Altman graph plot showed a mean difference of -0.61 and 97.2% of the difference score fell within the limits of agreement (95% CI, -5.67%, 4.39%). The overall mean score of the telehealth usability questionnaire was 5.62 out of 7.
Conclusion: Telehealth technology is a potential medium that provides an opportunity for synchronous practitioner-client interaction in evaluating home hazards. Some challenges were noted during the telehealth sessions, thus requiring a brief protocol to navigate the system.
Donnan GA, Fisher M, Macleod M, Davis SM. Stroke. Lancet. 2008; 371(9624):1612-23. [DOI:10.1016/S0140-6736(08)60694-7] [PMID]
Langhorne P, Stott D, Robertson L, MacDonald J, Jones L, McAlpine C, et al. Medical complications after stroke: A multicenter study. Stroke. 2000; 31(6): 1223-9. [DOI:10.1161/01.STR.31.6.1223] [PMID]
Schmid AA, Rittman M. Consequences of poststroke falls: Activity limitation, increased dependence, and the development of fear of falling. The American Journal of Occupational Therapy. 2009; 63(3):310-6. [DOI:10.5014/ajot.63.3.310] [PMID]
Batchelor FA, Mackintosh SF, Said CM, Hill KD. Falls after stroke. International Journal of Stroke. 2012; 7(6):482-90. [DOI:10.1111/j.1747-4949.2012.00796.x] [PMID]
Stubbs B, Brefka S, Denkinger MD. What works to prevent falls in community-dwelling older adults? Umbrella review of meta-analyses of randomized controlled trials. Physical Therapy. 2015; 95(8):1095-110. [DOI:10.2522/ptj.20140461] [PMID]
Ahmad Ainuddin H, Romli MH, Hamid TA, Salim MSF, Mackenzie L. Stroke rehabilitation for falls and risk of falls in Southeast Asia: A scoping review with stakeholders’ consultation. Frontiers in Public Health. 2021; 9:611793. [DOI:10.3389/fpubh.2021.611793] [PMID]
Simpson LA, Miller WC, Eng JJ. Effect of stroke on fall rate, location and predictors: A prospective comparison of older adults with and without stroke. Plos One. 2011; 6(4):e19431.[DOI:10.1371/journal.pone.0019431] [PMID]
Pighills A, Ballinger C, Pickering R, Chari S. A critical review of the effectiveness of environmental assessment and modification in the prevention of falls amongst community dwelling older people. British Journal of Occupational Therapy. 2016; 79(3): 133-43. [DOI:10.1177/0308022615600181]
Maggi P, de Almeida Mello J, Delye S, Cès S, Macq J, Gosset C, et al. Fall determinants and home modifications by occupational therapists to prevent falls: Facteurs déterminants des chutes et modifications du domicile effectuées par les ergothérapeutes pour prévenir les chutes. Canadian Journal of Occupational Therapy. 2018; 85(1):79-87. [DOI:10.1177/0008417417714284] [PMID]
Atwal A, Spiliotopoulou G, Stradden J, Fellows V, Anako E, Robinson L, et al. Factors influencing occupational therapy home visit practice: A qualitative study. Scandinavian Journal of Occupational Therapy. 2014; 21:40-7. [DOI:10.3109/11038128.2013.821162] [PMID]
Drummond AE, Whitehead P, Fellows K, Sprigg N, Sampson CJ, Edwards C, et al. Occupational therapy predischarge home visits for patients with a stroke (HOVIS): Results of a feasibility randomized controlled trial. Clinical Rehabilitation. 2012; 27(5):387-97. [DOI:10.1177/0269215512462145] [PMID]
Ninnis K, Van Den Berg M, Lannin NA, George S, Laver K. Information, and communication technology use within occupational therapy home assessments: A scoping review. British Journal of Occupational Therapy. 2018; 82(3):141-52. [DOI:10.1177/0308022618786928]
Ahmad Ainuddin H, Romli MH, Hamid TA, Sf Salim M, Mackenzie L. An exploratory qualitative study with older Malaysian stroke survivors, caregivers, and healthcare practitioners about falls and rehabilitation for falls after stroke. Frontiers in Public Health. 2021; 9:611814. [DOI:10.3389/fpubh.2021.611814] [PMID]
Wan Asyraf WZ, Ah Khan YK, Chung LW, Kee HF, Irene L, Ang CL, et al. Malaysia stroke council guide on acute stroke care service during COVID-19 Pandemic. The Medical Journal of Malaysia. 2020; 75(3):311-3. [PMID]
World Federation of Occupational Therapists. Occupational therapy and telehealth. London: World Federation of Occupational Therapists; 2021. [Link]
Hung Kn G, Fong KN. Effects of telerehabilitation in occupational therapy practice: A systematic review. Hong Kong Journal of Occupational Therapy. 2019; 32(1):3-21. [DOI:10.1177/1569186119849119] [PMID]
Romli MH, Mackenzie L, Lovarini M, Tan MP, Clemson L. The clinimetric properties of instruments measuring home hazards for older people at risk of falling: A systematic review. Evaluation & The Health Professions. 2018; 41(1):82-128. [DOI:10.1177/0163278716684166] [PMID]
Renda M, Lape JE. Feasibility and effectiveness of telehealth occupational therapy home modification interventions. International Journal of Telerehabilitation. 2018; 10(1):3-14. [DOI:10.5195/ijt.2018.6244] [PMID]
Sanford J, Butterfield T. Using remote assessment to provide home modification services to underserved elders. The Gerontologist. 2005; 45(3):389-98. [DOI:10.1093/geront/45.3.389] [PMID]
Sadasivam RS, Luger TM, Coley HL, Taylor BB, Padir T, Ritchie CS, et al. Robot-assisted home hazard assessment for fall prevention: A feasibility study. Journal of Telemedicine and Telecare. 2014; 20(1):3-10. [DOI:10.1177/1357633X13517350] [PMID]
Daniel H, Oesch P, Stuck AE, Born S, Bachmann S, Schoenenberger AW. Evaluation of a novel photography-based home assessment protocol for identification of environmental risk factors for falls in elderly persons. Swiss Medical Weekly. 2013; 143:w13884. [PMID]
Ritchey KC, Meyer D, Ice GH. Non-therapist identification of falling hazards in older adult homes using digital photography. Preventive Medical Reports. 2015; 2:794-7. [DOI:10.1016/j.pmedr.2015.09.004] [PMID]
Breeden LE. Occupational therapy home safety intervention via telehealth. International Journal of Telerehabilitation. 2016; 8(1):29-40. [DOI:10.5195/ijt.2016.6183] [PMID]
Romero S, Lee MJ, Simic I, Levy C, Sanford J. Development and validation of a remote home safety protocol. Disability and Rehabilitation. Assistive Technology. 2018; 13(2):166-72. [DOI:10.1080/17483107.2017.1300345] [PMID]
Stevens JA, Lee R. The potential to reduce falls and avert costs by clinically managing fall risk. American Journal of Preventive Medicine. 2018; 55(3):290-7. [DOI:10.1016/j.amepre.2018.04.035] [PMID]
Gately ME, Trudeau SA, Moo LR. Feasibility of telehealth-delivered home safety evaluations for caregivers of clients with dementia. Occupation, Participation and Health. 2020; 40(1):42-9. [DOI:10.1177/1539449219859935] [PMID]
Mackenzie L, Byles J, Higginbotham N. Designing the home falls and accidents screening tool (HOME FAST): Selecting the items. British Journal of Occupational Therapy. 2000; 63(6):260-9. [DOI:10.1177/030802260006300604]
Lai FHY, Yan EWH, Mackenzie L, Fong KNK, Kranz GS, Ho ECW, et al. Reliability, validity, and clinical utility of a self-reported screening tool in the prediction of fall incidence in older adults. Disability and Rehabilitation. 2019; 42(21):3098-105. [DOI:10.1080/09638288.2019.1582721] [PMID]
Romli MH, Mackenzie L, Lovarini M, Tan MP, Clemson L. The interrater and test-retest reliability of the Home Falls and Accidents Screening Tool (HOME FAST) in Malaysia: Using raters with a range of professional backgrounds. Journal of Evaluation in Clinical Practice. 2017; 23(3):662-9.[DOI:10.1111/jep.12697] [PMID]
Mackenzie L, Byles J, Higginbotham N. Reliability of the Home Falls and Accidents Screening Tool (HOME FAST) for identifying older people at increased risk of falls. Disability and Rehabilitation. 2002; 24(5):266-74. [DOI:10.1080/09638280110087089] [PMID]
Parmanto B, Lewis AN Jr, Graham KM, Bertolet MH. Development of the Telehealth Usability Questionnaire (TUQ). International Journal of Telerehabilitation. 2016; 8(1):3-10. [DOI:10.5195/ijt.2016.6196] [PMID]
Nielsen J. Usability 101: Introduction to usability [Internet]. 2003 [Updated 25 August 2003]. Available from: [Link]
ISO. Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 1: General introduction. Geneva: ISO;1992. [Link]
Hamzah N, Musa KI, Wee CX, Mohd Yusof NR, Wei LT, Subramaniam D, et al. Translation validity of three instruments on self-perception of technology-utilization for intervention. JCHS. 2021; 6(2).
Terwee CB, Mokkink LB, Knol DL, Ostelo RW, Bouter LM, de Vet HC. Rating the methodological quality in systematic reviews of studies on measurement properties: A scoring system for the COSMIN checklist. Quality of Life Research. 2012; 21(4):651-7. [DOI:10.1007/s11136-011-9960-1] [PMID]
Zaiontz C. Real Statistics Using Excel [Internet]. 2020. [Updated 27 August 2024]. Available from: [Link]
Gwet K. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Piedmont: Advanced Analytics, LLC; 2014. [Link]
Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology. 2013; 13:61. [DOI:10.1186/1471-2288-13-61] [PMID]
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1):159-74. [DOI:10.2307/2529310] [PMID]
Liu J, Tang W, Chen G, Lu Y, Feng C, Tu XM. Correlation and agreement: Overview and clarification of competing concepts and measures. Shanghai Archives of Psychiatry. 2016; 28(2):115-20. [PMID]
Giavarina D. Understanding bland altman analysis. Biochemia Medica. 2015; 25(2):141-51. [DOI:10.11613/BM.2015.015] [PMID]
Harvill L, Tennessee E, Uniuersit S. Standard error of measurement. Educational Measurement. 1991; 10(2):33-41. [DOI:10.1111/j.1745-3992.1991.tb00195.x]
Darter BJ, Rodriguez KM, Wilken JM. Test-retest reliability and minimum detectable change using the K4B2: Oxygen consumption, gait efficiency, and heart rate for healthy adults during submaximal walking. Research Quarterly for Exercise and Sport. 2013; 84(2):223-31. [DOI:10.1080/02701367.2013.784720] [PMID]
Nix J, Comans T. Home quick: Occupational therapy home visits using mHealth, to facilitate discharge from acute admission back to the community. International Journal of Telerehabilitation. 2017; 9(1):47-54. [DOI:10.5195/ijt.2017.6218] [PMID]
Ahmad Ainuddin H, Romli MH, Hamid TA, Salim MSF, Mat Din H, Mackenzie L. Cross-cultural adaptation and reliability of the Home Falls and Accidents Screening Tool (HOME FAST) in assessing falls-risk home hazards for stroke using technologies over a conventional home visit. Occupational Therapy International. 2022; 2022:6044182. [DOI:10.1155/2022/6044182] [PMID]
Struckmeyer LR, Pickens ND. Home modifications for people with Alzheimer’s disease: A scoping review. The American Journal of Occupational Therapy. 2016; 70(1):7001270020p1-9. [DOI:10.5014/ajot.2015.016089] [PMID]
Gately ME, Tickle-Degnen L, Voydetich DJ, Ward N, Ladin K, Moo LR. Video telehealth occupational therapy services for older veterans: National survey study. JMIR Rehabilitation and Assistive Technologies. 2021; 8(2):e24299. [DOI:10.2196/24299] [PMID]
| Issue | Vol 18 No 3 (2024) | |
| Section | Research Article(s) | |
| DOI | https://doi.org/10.18502/jmr.v18i3.16418 | |
| Keywords | ||
| Telehealth Usability Home environment Falls Stroke | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
