摘要：【中文】本发明涉及机械臂技术领域，具体而言，涉及一种手眼触屏机械臂。一种手眼触屏机械臂，包括底座、底座关节、臂关节、肘关节、腕关节、传感器和末端触屏指针以及设置在末端触屏指针外侧的摄像头，所述底座通过所述底座关节连接所述臂关节，所述臂关节通过所述肘关节连接所述腕关节，所述腕关节与所述末端触屏指针相连所述传感器包括第一关节力矩传感器、第二关节力矩传感器、第三关节力矩传感器、第四关节传感器、第五传感器。本发明的增益效果是:本技术方案采用多轴联动，通过机械臂与摄像头相互配合调度，实现操作效率高，触控位置精准，超程过载都会自动停止执行，保证不损伤其他设备，使用寿命长。 【EN】The invention relates to the technical field of mechanical arms, in particular to a hand-eye touch screen mechanical arm. The hand-eye touch screen mechanical arm comprises a base, a base joint, an arm joint, an elbow joint, a wrist joint, a sensor, a terminal touch screen pointer and a camera arranged on the outer side of the terminal touch screen pointer, wherein the base is connected with the arm joint through the base joint, the arm joint is connected with the wrist joint through the elbow joint, the wrist joint is connected with the terminal touch screen pointer, and the sensor comprises a first joint torque sensor, a second joint torque sensor, a third joint torque sensor, a fourth joint sensor and a fifth sensor. The multi-axis linkage type touch control device has the advantages that the multi-axis linkage is adopted, the mechanical arm and the camera are matched for scheduling, the high operation efficiency is realized, the touch control position is accurate, the overtravel overload can automatically stop execution, other equipment is guaranteed not to be damaged, and the service life is long.

摘要：【中文】本发明涉及图像处理技术和视觉伺服技术领域，特别涉及基一种电厂集控室深度学习和视觉伺服的触屏控制操作方法。本操作方法包括：获取集控室电厂需要监测的界面图像并建立相应图像库；利用训练好的卷积神经网络识别界面局部的字符信息来判断当前所停留的工作页面；采用图像的模板匹配方法检测需要后续机械臂点击的字符按钮；利用摄像头采集显示屏的RGB图像，再通过卷积神经网络确定显示屏的位姿信息，结合操作界面的图像识别与定位，驱动机械臂末端完成操作。本发明采用深度学习技术，实现集控系统状况的快速、智能判断；引入视觉伺服技术，实现机械臂对控制界面的精准操作，安全性得到了大幅提高，保证电厂生产的信息安全。 【EN】The invention relates to the technical field of image processing technology and visual servo, in particular to a touch screen control operation method based on deep learning and visual servo of a power plant centralized control room. The operation method comprises the following steps: acquiring an interface image to be monitored of a power plant in a centralized control room and establishing a corresponding image library; identifying local character information of an interface by using the trained convolutional neural network to judge the current staying working page; detecting a character button needing to be clicked by a subsequent mechanical arm by adopting an image template matching method; the camera is used for collecting RGB images of the display screen, the pose information of the display screen is determined through the convolution neural network, and the tail end of the mechanical arm is driven to complete operation by combining image recognition and positioning of an operation interface. The invention adopts the deep learning technology to realize the quick and intelligent judgment of the condition of the centralized control system; a visual servo technology is introduced, so that the mechanical arm can accurately operate a control interface, the safety is greatly improved, and the information safety of power plant production is ensured.