Inter-rater Reliability and Usability of Telehealth for Home Hazard Falls-Risk Assessment after Stroke
-
Husna Ahmad Ainuddin
,
Muhammad Hibatullah Romli
,
Tengku Aizan Hamid
,
Mazatulfazura SF Salim
,
Lynette Mackenzie
Abstract
Objective: Stroke survivors are more likely to fall at home. A home hazard assessment may be beneficial to reduce the risk of falling, but it is resource and time intensive. This study examined the inter-rater reliability and usability of telehealth for a hazard assessment to address risk of falls.
Methods: Two occupational therapists accessed the telehealth platform from different locations and simultaneously rated participants’ home environment using the Home Falls and Accident Screening Tool (HOME FAST). Stroke survivors and their caregivers answered the Telehealth Usability Questionnaire (TUQ).
Results: Thirty-six stroke survivors and 31 caregivers participated in the study. Gwet’s AC1 was used for agreement analysis. The overall AC1 value for the inter-rater reliability was 0.93 (95% CI: 0.66-1.00). There was a moderate correlation between the raters (r=0.57, p=0.000). Bland and Altman graph plot showed a mean difference of -0.61 and 97.2% of the difference score fell within the limits of agreement (95% CI, −5.67 to 4.39). The overall mean score of the TUQ was 5.62 out of 7.
Conclusions: Telehealth technology is a potential medium that provides an opportunity for synchronous practitioner-client interaction in evaluating home hazards. Some challenges were noted during the telehealth sessions, thus requiring a brief protocol to navigate the system.
[1] Donnan GA, Fisher M Macleod M, Davis SM. Stroke. Lancet. 2008;371(9624): 1612-1623. doi: 10.1016/S0140-6736(08)60694-7.
[2] Langhorne P, Stott D, Robertson L, MacDonald J, Jones L, McAlpine C, et al. Medical complications after stroke: a multicenter study. Stroke. 2000;31(6): 1223-1229. doi: 10.1161/01.str.31.6.1223.
[3] Schmid A, Rittman M. Consequences of poststroke falls: activity limitation, increased dependence, and the development of fear of falling. Am J Occup Ther. 2009;6: 310-316. doi: 10.5014/ajot.63.3.310.
[4] Batchelor FA, Mackintosh SF, Said CM, Hill KD. Falls after stroke. Int J Stroke. 2012;7(6): 482-490. doi: 10.1111/j.1747-4949.2012.00796.x.
[5] Stubbs B, Brefka S, Denkinger M. What works to prevent falls in community-dwelling older adults? Umbrella review of meta-analyses of randomized controlled trials. Phys Ther. 2015;95(8): 1095-1110. doi: 10.2522/ptj.20140461.
[6] Ainuddin HA, Romli MH, Hamid TA, Salim MSF, Mackenzie L. Stroke rehabilitation for falls and risk of falls in Southeast Asia: a scoping review with stakeholders’ consultation. Public Health Front. 2021;Mar 3(9): 611793. doi: 10.3389/fpubh.2021.611793.
[7] Simpson L, Miller W, and Eng J. Effect of stroke on fall rate, location and predictors: a prospective comparison of older adults with and without stroke. PLOS ONE. 2011; 6(4).
[8] Pighills A, Ballinger C, Pickering R, Chari S. A critical review of the effectiveness of environmental assessment and modification in the prevention of falls amongst community dwelling older people. Br J Occup Ther. 2016;79(3): 133-143. doi: 10.1177/0308022615600181.
[9] Maggi P, Mello JA, Delye S, Ces S, Macq J, Gosset C, et al. Fall determinants and home modifications by occupational therapists to prevent falls. Can J Occup Ther. 2018;85(1): 79-87. doi: 10.1177/0008417417714284.
[10] Atwal A, Spiliotopoulous G, Stradden J, Fellows V, Anako E, Robinson L, et al. Factors influencing occupational therapy home visit practice: A qualitative study. Scand J Occup Ther. 2014;21: 40-47. doi: 10.3109/11038128.2013.821162.
[11] Drummond AER, Whitehead P, Fellows K, Sprigg N, Sampson CJ, Edwards C, Lincoln NB, et al. Occupational therapy predischarge home visits for patients with a stroke (HOVIS): results of a feasibility randomized controlled trial. Clin Rehabil. 2012;27(5): 387-397. doi: 10.1177/0269215512462145.
[12] Ninnis K, Van Den Berg M, Lannin NA, George S, Laver K. Information, and communication technology use within occupational therapy home assessments: A scoping review. Br J Occup Ther. 2018;82(3): 141-152. doi: 10.1177/0308022618786928.
[13] Ainuddin HA, Romli MH, Hamid TA, Salim MSF, Mackenzie L. An exploratory qualitative study with older Malaysian stroke survivors, caregivers, and healthcare practitioners about falls and rehabilitation for falls after stroke. Public Health Front. 2021:9(611814). doi: 10.3389/fpubh.2021.611814.
[14] Wan Asyraf WZ, Khan YKA, Chung LW, Kee HF, Irene L, Ang CL, et al. Malaysia stroke council guide on acute stroke care service during COVID-19 Pandemic. Med J Malaysia. 2020;75(3): 311-313.
[15] World Federation of Occupational Therapists. (2021). Position Statement. Occupational
Therapy and Telehealth. Access from https://wfot.org/resources/occupational-therapy-and-
telehealth.
[16] Hung K, Fong K. Effects of telerehabilitation in occupational therapy practice: A systematic review. Hong Kong J Occup Ther. 2019;32(1): 3-21. doi: 10.1177/1569186119849119.
[17] Romli MH, Mackenzie L, Meryl L, Tan MP, Clemson L. The clinimetric properties of instruments measuring home hazards for older people at risk of falling: a systematic review. Eval Health Prof. 2018; 41(1): 82-128. doi: 10.1177/0163278716684166.
[18] Renda M, Lape J. Feasibility and effectiveness of telehealth occupational therapy home modification interventions. Int J Telerehabilitation. 2018;10(1): 3-14. doi: 10.5195/ijt.2018.6244.
[19] Sanford J, Butterfield T. Using remote assessment to provide home modification services to underserved elders. Gerontologist. 2005;45(3): 389-398. doi: 10.1093/geront/45.3.389.
[20] Sadasivam RS, Luger TM, Coley HL, Taylor BB, Padir T, Ritchie CS, et al. Robot-assisted home hazard assessment for fall prevention: a feasibility study. J Telemed Telecare. 2014;20(1): 3-10. doi: 10.1177/1357633X13517350.
[21] Daniel H, Oesch P, Stuck AE, Born S, Bachmann S, Schoenenberger AW. Evaluation of a novel photography-based home assessment protocol for identification of environmental risk factors for falls in elderly persons. Swiss Med Wkly. 2013;143(13884). doi: 10.4414/smw.2013.13884.
[22] Ritchey K, Meyer D, Ice G. Non-therapist identification of falling hazards in older adult homes using digital photography. Preventive Medical Reports. 2015;2: 794-797. doi: 10.1016/j.pmedr.2015.09.004.
[23] Breeden L. Occupational therapy home safety intervention via telehealth,” Int J Telerehabilitation. 2016;8(1): 29-40. doi: 10.5195/ijt.2016.6183.
[24] Romero S, Lee M, Simic I, Charles L, Jon S. Development and validation of a remote home safety protocol. Disabil Rehabil Assist Technol. 2018;13(2): 166-172. doi: 10.1080/17483107.2017.1300345.
[25] Stevens J, Lee R. The potential to reduce falls and avert costs by clinically managing fall risk. Am J Prev Med. 2018;55: 290-297. doi: 10.1016/j.amepre.2018.04.035.
[26] Gately M, Trudeau S, Moo L. Feasibility of Telehealth-Delivered Home Safety Evaluations for Caregivers of Clients with Dementia. OTJR. 2020;40(1): 42-49. doi: 10.1177/1539449219859935.
[27] Mackenzie L, Byles J, Higginbotham N. Designing the home falls and accidents screening tool (HOME FAST): selecting the items. Br J Occup Ther. 2000;63: 260-269. doi: 10.1177/030802260006300604.
[28] Lai F, Yan E, Mackenzie L, Fong KNK, Kranz GS, Ho ECW, et al. Reliability, validity, and clinical utility of a self-reported screening tool in the prediction of fall incidence in older adults. Disabil Rehabil. 2019;42(21): 3098-3105. doi: 10.1080/09638288.2019.1582721.
[29] Romli MH, Mackenzie L, Lovarini M, Tan MP, Clemson L. The interrater and test-retest reliability of the Home Falls and Accidents Screening Tool (HOME FAST) in Malaysia: Using raters with a range of professional backgrounds. J Eval Clin Pract. 2017; 23(3): 662-669, doi: 10.1111/jep.12697.
[30] Mackenzie L, Byles J, Higginbotham N. Reliability of the Home Falls and Accidents Screening Tool (HOME FAST) for identifying older people at increased risk of falls. Disabil Rehabil. 2002;24(5): 266-274. doi: 10.1080/09638280110087089.
[31] Parmanto B, Lewis A, Graham KM. Development of the Telehealth Usability Questionnaire (TUQ). Int J Telerehabilitation. 2016;8(1): 3-10, doi: https://doi.org/10.5195/ijt.2016.6196.
[32] Nielsen J. Usability 101: Introduction to usability. 2006. Available from: https://www.nngroup.com/articles/usability-101- introduction-to- usability/.
[33] ISO. Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 1: General introduction, ISO 9241-1:1992. 1992. Available from: http://www.iso.org/iso/iso catalogue/catalogueics/catalogue detail ics.htm?csnumber=16873.
[34] Hamzah N, Musa KI, Wee CX, Mohd Yusof NR, Wei LT, Subramaniam D, et al. Translation validity of three instruments on self-perception of technology-utilization for intervention. JCHS. 2021;6(2). doi: 10.24191/jchs.v6i2(S).
[35] Terwee C, Mokkink L, Knol D. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res. 2012;1(4): 651-657. doi: 10.1007/s11136-011-9960-1.
[36] Zaiontz C. Real Statistics Using Excel 2020. 2020. Available from: www. real- statistics.com.
[37] Gwet K, Handbook of Inter-rater Reliability. Gaithersburg (MD): Advanced Analytics, LLC; 2014.
[38] Wongpakaran N, Wongpakaran T, Wedding, D. et al. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13(61). doi: https://doi.org/10.1186/1471-2288-13-61
[39] Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1): 159-174.
[40] Liu J, Tang W, Chen G, Lu Y, Feng Y, Tu XM. Correlation and agreement: overview and clarification of competing concepts and measures. Shanghai Arch Psychiatry. 2016;28(2): 115-120. doi: 10.11919/j.issn.1002-0829.216045.
[41] Giavarina D. Understanding Bland Altman Analysis. Biochem Med. 2015;25(2): 141-151. doi: 10.11613/BM.2015.015
[42] Harvill L, Tennessee E, Uniuersit S. Standard Error of Measurement. EM: IP. 1991;10: 33-41.
[43] Darter B, Rodriquez K, Wilken J. Test–retest reliability and minimum detectable change using the K4B2: oxygen consumption, gait efficiency, and heart rate for healthy adults during submaximal walking. RQES. 2013;84(2): 223-231. doi: 10.1080/02701367.2013.784720.
[44] Nix J, Comans T. Home quick: Occupational therapy home visits using mHealth, to facilitate discharge from acute admission back to the community. Int J Telerehabilitation. 2017;9(1)-47-54. doi: 10.5195/ijt.2017.6218.
[45] Ainuddin HA, Romli MH, Hamid TA, Salim MSF, Mat Din H, Mackenzie L. Cross-cultural adaptation and reliability of the Home Falls and Accidents Screening Tool (HOME FAST) in assessing falls-risk home hazards for stroke using technologies over a conventional home visit. Occup Ther Int. 2022. doi: 10.1155/2022/6044182.
[46] Struckmeyer L, Pickens N. Home modifications for people with Alzheimer’s disease: a scoping review. Am J Occup Ther. 2016;70(1): 7001270020p1-9. doi: 10.5014/ajot.2015.016089.
[47] Gately M, Tickle-Degnen L, Voydetich DJ, Ward N, Ladin K, Moo LR. Video telehealth occupational therapy services for older veterans: national survey study. JMIR Rehab Assist Technol. 2021;8(2): e24299. doi: 10.2196/24299.
[2] Langhorne P, Stott D, Robertson L, MacDonald J, Jones L, McAlpine C, et al. Medical complications after stroke: a multicenter study. Stroke. 2000;31(6): 1223-1229. doi: 10.1161/01.str.31.6.1223.
[3] Schmid A, Rittman M. Consequences of poststroke falls: activity limitation, increased dependence, and the development of fear of falling. Am J Occup Ther. 2009;6: 310-316. doi: 10.5014/ajot.63.3.310.
[4] Batchelor FA, Mackintosh SF, Said CM, Hill KD. Falls after stroke. Int J Stroke. 2012;7(6): 482-490. doi: 10.1111/j.1747-4949.2012.00796.x.
[5] Stubbs B, Brefka S, Denkinger M. What works to prevent falls in community-dwelling older adults? Umbrella review of meta-analyses of randomized controlled trials. Phys Ther. 2015;95(8): 1095-1110. doi: 10.2522/ptj.20140461.
[6] Ainuddin HA, Romli MH, Hamid TA, Salim MSF, Mackenzie L. Stroke rehabilitation for falls and risk of falls in Southeast Asia: a scoping review with stakeholders’ consultation. Public Health Front. 2021;Mar 3(9): 611793. doi: 10.3389/fpubh.2021.611793.
[7] Simpson L, Miller W, and Eng J. Effect of stroke on fall rate, location and predictors: a prospective comparison of older adults with and without stroke. PLOS ONE. 2011; 6(4).
[8] Pighills A, Ballinger C, Pickering R, Chari S. A critical review of the effectiveness of environmental assessment and modification in the prevention of falls amongst community dwelling older people. Br J Occup Ther. 2016;79(3): 133-143. doi: 10.1177/0308022615600181.
[9] Maggi P, Mello JA, Delye S, Ces S, Macq J, Gosset C, et al. Fall determinants and home modifications by occupational therapists to prevent falls. Can J Occup Ther. 2018;85(1): 79-87. doi: 10.1177/0008417417714284.
[10] Atwal A, Spiliotopoulous G, Stradden J, Fellows V, Anako E, Robinson L, et al. Factors influencing occupational therapy home visit practice: A qualitative study. Scand J Occup Ther. 2014;21: 40-47. doi: 10.3109/11038128.2013.821162.
[11] Drummond AER, Whitehead P, Fellows K, Sprigg N, Sampson CJ, Edwards C, Lincoln NB, et al. Occupational therapy predischarge home visits for patients with a stroke (HOVIS): results of a feasibility randomized controlled trial. Clin Rehabil. 2012;27(5): 387-397. doi: 10.1177/0269215512462145.
[12] Ninnis K, Van Den Berg M, Lannin NA, George S, Laver K. Information, and communication technology use within occupational therapy home assessments: A scoping review. Br J Occup Ther. 2018;82(3): 141-152. doi: 10.1177/0308022618786928.
[13] Ainuddin HA, Romli MH, Hamid TA, Salim MSF, Mackenzie L. An exploratory qualitative study with older Malaysian stroke survivors, caregivers, and healthcare practitioners about falls and rehabilitation for falls after stroke. Public Health Front. 2021:9(611814). doi: 10.3389/fpubh.2021.611814.
[14] Wan Asyraf WZ, Khan YKA, Chung LW, Kee HF, Irene L, Ang CL, et al. Malaysia stroke council guide on acute stroke care service during COVID-19 Pandemic. Med J Malaysia. 2020;75(3): 311-313.
[15] World Federation of Occupational Therapists. (2021). Position Statement. Occupational
Therapy and Telehealth. Access from https://wfot.org/resources/occupational-therapy-and-
telehealth.
[16] Hung K, Fong K. Effects of telerehabilitation in occupational therapy practice: A systematic review. Hong Kong J Occup Ther. 2019;32(1): 3-21. doi: 10.1177/1569186119849119.
[17] Romli MH, Mackenzie L, Meryl L, Tan MP, Clemson L. The clinimetric properties of instruments measuring home hazards for older people at risk of falling: a systematic review. Eval Health Prof. 2018; 41(1): 82-128. doi: 10.1177/0163278716684166.
[18] Renda M, Lape J. Feasibility and effectiveness of telehealth occupational therapy home modification interventions. Int J Telerehabilitation. 2018;10(1): 3-14. doi: 10.5195/ijt.2018.6244.
[19] Sanford J, Butterfield T. Using remote assessment to provide home modification services to underserved elders. Gerontologist. 2005;45(3): 389-398. doi: 10.1093/geront/45.3.389.
[20] Sadasivam RS, Luger TM, Coley HL, Taylor BB, Padir T, Ritchie CS, et al. Robot-assisted home hazard assessment for fall prevention: a feasibility study. J Telemed Telecare. 2014;20(1): 3-10. doi: 10.1177/1357633X13517350.
[21] Daniel H, Oesch P, Stuck AE, Born S, Bachmann S, Schoenenberger AW. Evaluation of a novel photography-based home assessment protocol for identification of environmental risk factors for falls in elderly persons. Swiss Med Wkly. 2013;143(13884). doi: 10.4414/smw.2013.13884.
[22] Ritchey K, Meyer D, Ice G. Non-therapist identification of falling hazards in older adult homes using digital photography. Preventive Medical Reports. 2015;2: 794-797. doi: 10.1016/j.pmedr.2015.09.004.
[23] Breeden L. Occupational therapy home safety intervention via telehealth,” Int J Telerehabilitation. 2016;8(1): 29-40. doi: 10.5195/ijt.2016.6183.
[24] Romero S, Lee M, Simic I, Charles L, Jon S. Development and validation of a remote home safety protocol. Disabil Rehabil Assist Technol. 2018;13(2): 166-172. doi: 10.1080/17483107.2017.1300345.
[25] Stevens J, Lee R. The potential to reduce falls and avert costs by clinically managing fall risk. Am J Prev Med. 2018;55: 290-297. doi: 10.1016/j.amepre.2018.04.035.
[26] Gately M, Trudeau S, Moo L. Feasibility of Telehealth-Delivered Home Safety Evaluations for Caregivers of Clients with Dementia. OTJR. 2020;40(1): 42-49. doi: 10.1177/1539449219859935.
[27] Mackenzie L, Byles J, Higginbotham N. Designing the home falls and accidents screening tool (HOME FAST): selecting the items. Br J Occup Ther. 2000;63: 260-269. doi: 10.1177/030802260006300604.
[28] Lai F, Yan E, Mackenzie L, Fong KNK, Kranz GS, Ho ECW, et al. Reliability, validity, and clinical utility of a self-reported screening tool in the prediction of fall incidence in older adults. Disabil Rehabil. 2019;42(21): 3098-3105. doi: 10.1080/09638288.2019.1582721.
[29] Romli MH, Mackenzie L, Lovarini M, Tan MP, Clemson L. The interrater and test-retest reliability of the Home Falls and Accidents Screening Tool (HOME FAST) in Malaysia: Using raters with a range of professional backgrounds. J Eval Clin Pract. 2017; 23(3): 662-669, doi: 10.1111/jep.12697.
[30] Mackenzie L, Byles J, Higginbotham N. Reliability of the Home Falls and Accidents Screening Tool (HOME FAST) for identifying older people at increased risk of falls. Disabil Rehabil. 2002;24(5): 266-274. doi: 10.1080/09638280110087089.
[31] Parmanto B, Lewis A, Graham KM. Development of the Telehealth Usability Questionnaire (TUQ). Int J Telerehabilitation. 2016;8(1): 3-10, doi: https://doi.org/10.5195/ijt.2016.6196.
[32] Nielsen J. Usability 101: Introduction to usability. 2006. Available from: https://www.nngroup.com/articles/usability-101- introduction-to- usability/.
[33] ISO. Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 1: General introduction, ISO 9241-1:1992. 1992. Available from: http://www.iso.org/iso/iso catalogue/catalogueics/catalogue detail ics.htm?csnumber=16873.
[34] Hamzah N, Musa KI, Wee CX, Mohd Yusof NR, Wei LT, Subramaniam D, et al. Translation validity of three instruments on self-perception of technology-utilization for intervention. JCHS. 2021;6(2). doi: 10.24191/jchs.v6i2(S).
[35] Terwee C, Mokkink L, Knol D. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res. 2012;1(4): 651-657. doi: 10.1007/s11136-011-9960-1.
[36] Zaiontz C. Real Statistics Using Excel 2020. 2020. Available from: www. real- statistics.com.
[37] Gwet K, Handbook of Inter-rater Reliability. Gaithersburg (MD): Advanced Analytics, LLC; 2014.
[38] Wongpakaran N, Wongpakaran T, Wedding, D. et al. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13(61). doi: https://doi.org/10.1186/1471-2288-13-61
[39] Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1): 159-174.
[40] Liu J, Tang W, Chen G, Lu Y, Feng Y, Tu XM. Correlation and agreement: overview and clarification of competing concepts and measures. Shanghai Arch Psychiatry. 2016;28(2): 115-120. doi: 10.11919/j.issn.1002-0829.216045.
[41] Giavarina D. Understanding Bland Altman Analysis. Biochem Med. 2015;25(2): 141-151. doi: 10.11613/BM.2015.015
[42] Harvill L, Tennessee E, Uniuersit S. Standard Error of Measurement. EM: IP. 1991;10: 33-41.
[43] Darter B, Rodriquez K, Wilken J. Test–retest reliability and minimum detectable change using the K4B2: oxygen consumption, gait efficiency, and heart rate for healthy adults during submaximal walking. RQES. 2013;84(2): 223-231. doi: 10.1080/02701367.2013.784720.
[44] Nix J, Comans T. Home quick: Occupational therapy home visits using mHealth, to facilitate discharge from acute admission back to the community. Int J Telerehabilitation. 2017;9(1)-47-54. doi: 10.5195/ijt.2017.6218.
[45] Ainuddin HA, Romli MH, Hamid TA, Salim MSF, Mat Din H, Mackenzie L. Cross-cultural adaptation and reliability of the Home Falls and Accidents Screening Tool (HOME FAST) in assessing falls-risk home hazards for stroke using technologies over a conventional home visit. Occup Ther Int. 2022. doi: 10.1155/2022/6044182.
[46] Struckmeyer L, Pickens N. Home modifications for people with Alzheimer’s disease: a scoping review. Am J Occup Ther. 2016;70(1): 7001270020p1-9. doi: 10.5014/ajot.2015.016089.
[47] Gately M, Tickle-Degnen L, Voydetich DJ, Ward N, Ladin K, Moo LR. Video telehealth occupational therapy services for older veterans: national survey study. JMIR Rehab Assist Technol. 2021;8(2): e24299. doi: 10.2196/24299.
| Issue | Articles in Press | |
| Section | Research Article(s) | |
| Keywords | ||
| Telehealth; usability; home hazard; falls; stroke | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
How to Cite
1.
Ahmad Ainuddin H, Romli MH, Hamid TA, SF Salim M, Mackenzie L. Inter-rater Reliability and Usability of Telehealth for Home Hazard Falls-Risk Assessment after Stroke. jmr. 2024;(-).
