Overview of the project

The main aim of the project is designing an ambidextrous robotic hand actuated by air muscles and controlled over internet with intuitive control interface allowing it to be operated by users without additional training.

The main application areas of the ambidextrous hand are rehabilitation and physiotherapy after injuries or strokes and management of phantom pains for amputees by learning to control a robotic prosthesis.

Other applications are possible such as a biomedical or robotic research platform, teleoperation, remote educational platform (see chromeweblab project) as well as a number of novelresearch or artistic applications which might beinspired by implementing control interface over social media websites.

To facilitate ease of deployment and keep the total price of the system competitive the emphasis was placed on Open Source software and hardware platforms from early stages of the project. Using TCP/IP as a main control backbone protocol insures minimal installation requirements and specialized control infrastructure needed for places like hospitals where the system will be installed to be used by patients remotely.




System architecture


Electromyography

Electromyography is a technique for evaluating and recording the electrical activity produced by skeletal muscles. EMG is performed using an instrument called an electromyograph, to produce a record called an electromyogram. This technique can be a way to reduce phantom limb pain that is to say the sensation that an amputated or missing limb (even an organ, like the appendix) is still attached to the body and is moving appropriately with other body parts. The signal is acquired from muscles before being processed and connected to the robotic interface. Here is the global scheme of the process :



Diagram of electromyography

Image processing

Hand detection

A web browser plugin contains an application able to detect the shape of a hand. An approximate polygon is created according to the fingers´ contour and angles are analyzed according to the gesture adopted by the user.





Some outputs of the image processing

Points tracking

To improve the hand tracking, it was later decided to filter the input image before starting the gesture detection. The first filter is the background subtraction. The program first captures the background, then removes the background for each following input frame. Therefore only the foreground remains after this operation. In our case the user’s hand is the foreground. Then with the help of the second filter, the skin segmentation, the remaining noise is taken off resulting from the background substraction. To do so, the HSV color space is used to recognize the human skin as accurate as possible. This color space allows us to recognize a wide range of color skin. The purpose of these two filters is to get an image of a hand as clean as possible.






From original video to contour drawing via background substraction


Interfacing of different modules

The control means of the robot hand are multiple, with modules running on different platforms (linux, mac os, windows, ...) with their own communication protocol and program codes written in different languages. Thus the set of modules must be unified through a server which makes them all compatible. This is why the modules are connected to Robot Operating System (ROS), which is a multi-language and multi-platform meta-operating system used as a middleware from an operating system.



Connection between the different modules


Designs of mechanical structure

Concept by Michal Simko

The first concept of the ambidextrous robotic hand is displayed in the following figures. The concept is designed to be manufactured mainly from ABS plastic by using Rapid Prototyping technique. The plastic parts are displayed in light grey color and machinable parts to be made from aluminum are displayed in light green color.




Ambidextrous Hand - Conceptual Design


Each finger of the conceptual hand is controlled via four sets of pneumatic muscles that are positioned in the forearm and connected via tendons to each finger. The thumb is controlled by 5 muscles and it can rotate 150 degrees enabling right or left hand operation.



Left versus Right hand thumb position


Additionally the forefinger, ring and little fingers are able to move laterally, while the middle finger is fixed to hand as shown in Figure 11. The movement is performed via 2 muscles, 1 used for forefinger and 1 for ring/little fingers.



Fingers lateral movement and linkage set up (right)



Concept by Luke Kavanagh

The finger design centers around the concept of keeping all tendons routed along the center line of each phalanx in order to avoid tendon stretching due to the movement of preceding phalanxes.




CAD designs


Movement in the phalanxes is produced as a result of pressure exerted on the circumference of the guiding pulleys on the left hand side of each phalanx. These pulleys are rigidly connected to their respective phalanx. Pulleys located centrally within the proximal phalanx (green) are to free to rotate to reduce friction with the tendons and are also used to keep tendons centrally aligned.



Location of pulleys


Preliminary thumb design- thumb rotates about an axis located roughly in line with the middle finger to give arcing movement of the thumb similar to hand. As with finger design, tendons are located centrally, this is especially important with the wide range of movement and ambidextrous thumb experiences when rotating about the palm when entering and exiting the palm.



Preliminary thumb design


Since the thumb is only required to flex and extend in one direction profiled pulleys have been used for the joints of each phalanx to allow the tendons to be located along the central axes of them, again mid-point pulleys are used to aid tendon location. The palm has not yet been developed.
Remote-control

The hand can be commanded through internet from Facebook or from a software application. An embedded webcam streams the video in real-time and different instructions can be sent by clicking on buttons.



The user interface from the software application


Development of control system hardware (by Anthony Huynh)
For a number of reasons, a modular approach has been taken for the control of this robotic hand. This means that we will use more than one Microcontroller board to process the control of this robot. One of the reasons is for “expandability“; the modular approach does not limit the amount of Inputs and Outputs that can be used, not just because of physical pin limits but also because we can get better loop rates.
We will able to accommodate any new features that may be applied to the hand with a dedicated microcontroller communicating with the rest of the robot. One such feature that is currently being worked on is a remote controlled ability of the hand using Ethernet.
Cheaper, smaller boards can be used and therefore are able to be placed where we want; each board can be in completely different locations if necessary. If the hand is extended into an arm then we do not need to replace the boards with a larger one for control of the whole system, but only have to add microcontrollers for the arm and have them communicating.

Modelling in Ambidextrous robotic hand project (by Leonid Paramonov)
Early in the project it was identified that there is no available modeling software for modeling tendon driven systems as well as those modeling air muscles. A modeling library has been developed in Matlab which allows in silico investigation of different physical parameters of the mechanical structure of the ambidextrous finger, number of muscles, types of tendon routing. The model allows modeling of elastic contact forces between different objects and therefore allows for investigation of efficiency of gasping which can be achieved by the given mechanical design. The model describes fundamental motions of the finger design in 2D due to given tendon routing and mechanical parameters.



Three phalanx finger design driven by 2 “passive“ and 4 “active“ tendons


In this design the Distal Phalanx is driven by two “passive tendons“, which are not activated by muscles themselves but are driven by change of relative position between Medial and Proximal phalanges effectively working as a mechanical constraint which links degrees of freedom DIP (Distal Interphalangeal joint) and MIP (Middle Interphalangeal joint) joints. Similar constraints are often applied in robotic hand designs and similar to mechanical action of human hand. This mechanical constraint allows for reduction of total number of muscles required to fully control the mechanical structure of ambidextrous finger to reproduce motions of a human hand.

Mechanical structure validation for ambidextrous behavior

In order to identify if it is feasible to construct and control ambidextrous hand, Meccano tools have been used at initial stage of project development. Below are shown the preliminary results on design and control of ambidextrous hand structure.

Finger´s behavior

The flexion and extension of the finger are driven by two pairs of antagonist muscles, one for the proximal phalange and the other one for the intermediate and distal phalanges which always move in synchronization.





Maximum ranges of a prototype of an ambidextrous finger


The wheels at joints´ level permit to fix an angle allowing the finger to keep a human behavior when the structure switches from one side to another. Sockets are implemented to prevent the tendons to become slack when the proximal phalange is moving. The left behavior is controlled from right and inversely for the same reason, so the interaction with the distal phalanges is still possible. Torsion springs allow these very phalanges to stay vertical when the muscles are relaxed.




Right and left behavior of the finger


Finger´s control

The sensors´ data feedback is dealt with a PID (Proportional, Integrative and Derivative) control, which calculates in real-time the difference between the target fixed by the program and the current values of the sensors. Muscles´ contractions are then controlled to make the finger reach the required force or position.

Angular sensors

The angular control can be used either to reach a specific angle with the proximal phalange or to copy one position from one sensor to another one.




Control of the angle


Force sensors

According to the target fixed by the program, the finger can stop bending when it detects a piece of paper or strengthen its grasp to hold a piece of metal.




Different forces applied by the finger: detection of paper and hold of a piece of metal



References

Both of these researches have been used to reach the results obtained by the image processing:

J. Jones, M. and M.Rehg., J. (1999) "Statistical color models with application to skin detection", International Journal of Computer Vision , vol. 46, pp. 274-275-280.

Molenaar, G. (2010) Sonic Gesture.