Just play your piano by moving your fingers in the air!

Taken from Unsplash

I recently visited my cousin and he had been trying to learn the piano for some time. However, due pandemic, his teacher could not come home and they held zoom meetings. That’s when I got the idea to make a virtual piano that both his teacher and he could use to learn music. When I thought about this, I wondered why I wouldn’t move outside the keyboard? Trying to make music from scratch? Let our creative mind flow and make an interaction that allows a person to play the piano just by moving their hands in the air ?! That’s when I decided to do it “Ilmapiano”.

Air Piano is a project made at the confluence of computer vision and human-computer interaction. To make the Air Piano, I have used the Python language and a specialized library called OpenCV. It is an open source computer vision and machine learning software library.

Another important library that will help us implement this project is the PyAutoGUI library. PyAutoGUI lets your Python scripts control your mouse and keyboard to automate interaction with other applications. PyAutoGUI has several features: moving and clicking the mouse or typing in windows of other applications, taking screenshots, etc.

  • The first step is to capture the user’s video stream.
  • After reading the input frame from the frame, we now need to change the BGR scale to the HSV scale so we can work better with colors.

Why do we convert to HSV color space? The simple answer is that, unlike RGB, HSV distinguishes the intensity of snow or image from chrome or color information. In computer view, you often want to distinguish color components from intensity for several reasons, such as resistance to changes in lighting or removal of shadows.

  • The hand is detected using the black color mask we created on the HSV scale. For this purpose, I decided to wear a pair of black gloves because it was relatively difficult to detect skin color and would have lost the generalization of the project.
  • Once the hand is observed, we find the outline, i.e. the boundary of our hand. We then draw a convex body to find the convex polygon surrounding it. From this polygon, we remove fingertips using convexity errors.

What is Convex Hull? Convex body, the smallest n-sided convex polygon that completely surrounds the object.

What are convexity errors? Any deviation of the shape from the convex body is called the convexity error.

  • There’s also a filter to get only the fingertips using the distance between the points, i.e. the fingertip and the joint. You can also use the angle between your fingers to achieve the same.
  • Before we start using the PyAutoGUI function, we draw the piano keys in a frame that becomes the “Air Weight Keys”.
  • The last section includes the use of the PyAutoGUI library, which allows you to perform keyboard functions according to the coordinates of hand movements (with your fingertips accurate). Now that this program is running, it tracks the position of the fingertips on the frame and automatically presses said keys. For this to work, we open another window with a link: https://www.onlinepianist.com/virtual-piano.
    This is a virtual piano that we control with our fingertips.

It was a detailed technical description of the Air Piano project, and I hope you learn a lot through this. Visit Github buy and check the entire code for a better understanding. The following are the resources I used to learn and helped me build this project successfully –

  1. Fingertip recognition
  2. Convex frame
  3. Convexity errors
  4. Hand recognition

The whole set of instruments evolving along similar lines can be made gesture-guided by the same principle as above, creating a highly interactive environment for learning music. We can add many other utilities and use the latest developments (such as MediaPipe library) more interesting for this project.


Please enter your comment!
Please enter your name here