Data Fusion of Human Skeleton Joint Tracking Using Two Kinect Sensors and Extended Set Membership Filter
-
摘要: 以Kinect为代表的深度图像传感器在肢体康复系统中得到广泛应用.单一深度图像传感器采集人体关节点数据时由于肢体遮挡、传感器数据错误和丢失等原因降低系统可靠性.本文研究了利用两台Kinect深度图像传感器进行数据融合从而达到消除遮挡、数据错误和丢失的目的，提高康复系统中数据的稳定性和可靠性.首先，利用两台Kinect采集患者健康侧手臂运动数据；其次，对两组数据做时间对准、Bursa线性模型下的坐标变换和基于集员滤波的数据融合；再次，将融合后的健康侧手臂运动数据经过“镜像运动”作为患侧手臂运动指令；最后，将患侧运动指令下发给可穿戴式镜像康复外骨骼带动患者患侧手臂完成三维动画提示的康复动作，达到患者主动可控康复的目的.本文通过Kinect与VICON系统联合实验以及7自由度机械臂控制实验验证了数据融合方法的有效性，以及两台Kinect可有效解决上述问题.
-
关键词:
- 镜像康复 /
- Kinect深度图像传感器 /
- Bursa线性模型 /
- 集员滤波
Abstract: Kinect-like depth sensors are widely used in rehabilitation systems. However, using a single depth sensor is less reliable due to limb blocking, data-loss or error. This paper uses two Kinect sensors and a data fusion algorithm to solve these problems. Firstly, two Kinect sensors are used to capture the motion data of the healthy side arm of the hemiplegic patient. Secondly, the data is processed with time alignment, then coordinates are transformed with Bursa transform and data fusion is done using extended set membership filter (ESMF) successively. Then, the motion data is mirrored by the middle plane, namely "mirror motion". In the end, the mirrored motion data controls the wearable robotic arm to drive the patient's paralytic side arm so as to interactively and initiatively complete a variety of recovery actions prompted by computer with 3D animation games. The effectiveness of the proposed approach is validated by both experiments on Kinect sensors & VICON and a 7 DOF manipulator. Also, two Kinect sensors can solve those problems effectively.1) 本文责任编委 程龙 -
图 2 镜像运动康复系统的Kinect布置示意图及各坐标系
Fig. 2 Coordinate system and the layout of Kinect sensors of the mirror movements rehabilitation system
图 5 副Kinet#2的坐标系与主Kinect#1沿其$Y$轴旋转90$^{\circ}$之后的坐标系之间满足小角度旋转假设
Fig. 5 The coordinate of Kinect#2 should meet the small angle rotation condition with the coordinate of Kinect#1 after it rotates 90$^{\circ}$ according to its $Y$ axis
图 6 Kinect传感器追踪的三个手臂关节点及其三个关节角
Fig. 6 3 joints and the 3 joint angles tracked by Kinect sensors
图 10 实验现场对两台Kinect做空间对准的场景
Fig. 10 Experiment scene of registration of the two Kinect sensors
图 11 VICON和Kinect同时追踪肢体动作
Fig. 11 VICON and Kinect sensors tracking the limb joints synchronously
图 13 VICON和Kinect采集的手掌点运动数据对比
Fig. 13 The movement data of hand from VICON system and Kinect sensor
图 14 系统误差补偿后的Kinect数据与VICON数据
Fig. 14 The data from Kinect sensor and data from VICON system using system error compensation
表 1 Kinect人体关节点数据误差
Table 1 The error of Kinect sensor for joints tracking
坐标 均值(mm) 方差(mm) 区间(mm) 直角系x -15.4 45.6 [-200 200] 直角系y -36.1 38.7 [-150 150] 直角系z 16 54.1 [-200 200] 球面系r 14.3 54.5 [-200 200] 球面系φ 0.01 0.03 [-0.1 0.1] 球面系θ 0.19 0.6 [-1 1] 
下载: 导出CSV
表 2 数据融合后Kinect人体关节点数据误差
Table 2 The error of Kinect sensor after data fusion
坐标 均值(mm) 方差(mm) 直角系x 0.0 36.7 直角系y 0.0 34.9 直角系z 0.0 40.6 球面系r 0.8 38.4 球面系φ 0.00 0.02 球面系θ 0.02 0.60
下载: 导出CSV
-
[1] Roy A K, Soni Y, Dubey S. Enhancing effectiveness of motor rehabilitation using Kinect motion sensing technology. In:Proceedings of the 2013 IEEE Global Humanitarian Technology Conference. Trivandrum, India:IEEE, 2013. 298-304 [2] Chang C Y, Lange B, Zhang M, Koenig S, Requejo P, Somboon N, Sawchuk A A, Rizzo A A. Towards pervasive physical rehabilitation using Microsoft Kinect. In:Proceedings of the 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops. San Diego, USA:IEEE, 2012. 159-162 [3] Chang Y J, Chen S F, Huang J D. A Kinect-based system for physical rehabilitation:a pilot study for young adults with motor disabilities. Research in Developmental Disabilities, 2011, 32(6):2566-2570 doi: 10.1016/j.ridd.2011.07.002 [4] Nguyen K D, Chen I M, Luo Z, Yeo S H, Duh H B L. A wearable sensing system for tracking and monitoring of functional arm movement. IEEE/ASME Transactions on Mechatronics, 2011, 16(2):213-220 doi: 10.1109/TMECH.2009.2039222 [5] Kong K, Tomizuka M. Control of exoskeletons inspired by fictitious gain in human model. IEEE/ASME Transactions on Mechatronics, 2009, 14(6):689-698 doi: 10.1109/TMECH.2009.2032685 [6] Han J G, Shao L, Xu D, Shotton J. Enhanced computer vision with Microsoft Kinect sensor:a review. IEEE Transactions on Cybernetics, 2013, 43(5):1318-1334 doi: 10.1109/TCYB.2013.2265378 [7] Khoshelham K, Elberink S O. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors, 2012, 12(2):1437-1454 https://www.ncbi.nlm.nih.gov/pubmed/22438718 [8] Choi S M, Jeong J C, Chang J. Dense 3D depth map with DOE pattern. In:Proceedings of the 9th International Conference on Ubiquitous Robots and Ambient Intelligence. Daejeon, Korea:IEEE, 2012. 34-37 [9] Dutta T. Evaluation of the KinectTM sensor for 3-D kinematic measurement in the workplace. Applied Ergonomics, 2012, 43(4):645-649 doi: 10.1016/j.apergo.2011.09.011 [10] Stommel M, Beetz M, Xu W L. Inpainting of missing values in the Kinect sensor's depth maps based on background estimates. IEEE Sensors Journal, 2014, 14(4):1107-1116 doi: 10.1109/JSEN.2013.2291315 [11] Newcombe R A, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison A J, Kohi P, Shotton J, Hodges S, Fitzgibbon A. KinectFusion:real-time dense surface mapping and tracking. In:Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality. Basel, Switzerland:IEEE, 2011. 127-136 [12] 杨东方, 王仕成, 刘华平, 刘志国, 孙富春.基于Kinect系统的场景建模与机器人自主导航.机器人, 2012, 34(5):581-589 doi: 10.3724/SP.J.1218.2012.00581Yang Dong-Fang, Wang Shi-Cheng, Liu Hua-Ping, Liu Zhi-Guo, Sun Fu-Chun. Scene modeling and autonomous navigation for robots based on Kinect system. Robot, 2012, 34(5):581-589 doi: 10.3724/SP.J.1218.2012.00581 [13] Xia L, Chen C C, Aggarwal J K. Human detection using depth information by Kinect. In:Proceedings of the 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Colorado, USA:IEEE, 2011. 15-22 [14] Zhang Z, Liu W H, Metsis V, Athitsos V. A viewpoint-independent statistical method for fall detection. In:Proceedings of the 21st International Conference on Pattern Recognition. Tsukuba, Japan:IEEE, 2012. 3626-3630 [15] Obdrzálek S, Kurillo G, Ofli F, Bajcsy R, Seto E, Jimison H, Pavel M. Accuracy and robustness of Kinect pose estimation in the context of coaching of elderly population. In:Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. San Diego, USA:IEEE, 2012. 1188-1193 [16] Štrbac M D, Malešević N M, Čobeljić R, Schwirtlich L. Feedback control of the forearm movement of tetraplegic patient based on Microsoft Kinect and multi-pad electrodes. Journal of Automatic Control, 2013, 21(1):7-11 doi: 10.2298/JAC1301007S [17] Edward V, Peña C, Pardo A. Educational robotic platform of low cost for brain and servovisual control practice. Recent Researches in Automatic Control, Systems Science and Communications. Porto:WSEAS Press, 2012. 127-133 [18] Megalingam R K, Saboo N, Ajithkumar N, Unny S, Menon D. Kinect based gesture controlled robotic arm:a research work at HuT Labs, In:Proceedings of the 2013 IEEE International Conference in MOOC Innovation and Technology in Education. Jaipur, India:IEEE, 2013. 294-299 [19] Tao G, Archambault P S, Levin M F. Evaluation of Kinect skeletal tracking in a virtual reality rehabilitation system for upper limb hemiparesis. In:Proceedings of the 2013 International Conference on Virtual Rehabilitation. Philadelphia, USA:2013. 164-165 [20] Li S Y, Pathirana P N, Caelli T. Multi-Kinect skeleton fusion for physical rehabilitation monitoring. In:Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Chicago, USA:IEEE, 2014. 5060-5063 [21] 朱特浩, 赵群飞, 夏泽洋.利用Kinect的人体动作视觉感知算法.机器人, 2014, 36(6):647-653 http://www.cnki.com.cn/Article/CJFDTOTAL-JQRR201406002.htmZhu Te-Hao, Zhao Qun-Fei, Xia Ze-Yang. A visual perception algorithm for human motion by a Kinect. Robot, 2014, 36(6):647-653 http://www.cnki.com.cn/Article/CJFDTOTAL-JQRR201406002.htm [22] 王文利, 程传录, 陈俊英.常用坐标转换模型及其实用性研究.测绘信息与工程, 2010, 35(5):37-39 http://www.cnki.com.cn/Article/CJFDTOTAL-CHXG201005017.htmWang Wen-Li, Cheng Chuan-Lu, Chen Jun-Ying. Coordinate transformation models and their practicability. Journal of Geomatics, 2010, 35(5):37-39 http://www.cnki.com.cn/Article/CJFDTOTAL-CHXG201005017.htm [23] 李微晓, 沈云中, 李博峰.顾及2套坐标误差的三维坐标变换方法.同济大学学报(自然科学版), 2011, 39(8):1243-1246 http://www.cnki.com.cn/Article/CJFDTOTAL-TJDZ201108028.htmLi Wei-Xiao, Shen Yun-Zhong, Li Bo-Feng. Three-dimensional coordinate transformation with consideration of coordinate errors in two coordinate systems. Journal of Tongji University (Natural Science), 2011, 39(8):1243-1246 http://www.cnki.com.cn/Article/CJFDTOTAL-TJDZ201108028.htm [24] 周波, 钱堃, 马旭东, 戴先中.一种新的基于保证定界椭球算法的非线性集员滤波器.自动化学报, 2013, 39(2):150-158 http://www.aas.net.cn/CN/abstract/abstract17846.shtmlZhou Bo, Qian Kun, Ma Xu-Dong, Dai Xian-Zhong. A new nonlinear set membership filter based on guaranteed bounding ellipsoid algorithm. Acta Automatica Sinica, 2013, 39(2):150-158 http://www.aas.net.cn/CN/abstract/abstract17846.shtml [25] Gu F, He Y Q, Han J D. Active persistent localization of a three-dimensional moving target under set-membership uncertainty description through cooperation of multiple mobile robots. IEEE Transactions on Industrial Electronics, 2015, 62(8):4958-4971 doi: 10.1109/TIE.2015.2403798 -
点击查看大图
计量
- 文章访问数: 2160
- HTML全文浏览量: 432
- PDF下载量: 720
- 被引次数: 0
