Newly developed AI grades your piano skills by watching you play

An artificial neural network developed by the University of Nevada is said to be capable of determining the skill of a pianist by watching a video of their performance. The researchers suggest this could be a useful tool in remote learning, with COVID-19 resulting in a reduction of in-person tuition.

Led by Brendan Morris, the study is the first to explore the potential of computer vision in predicting a pianist’s skill level. The predictions are based on two attributes: player skill and music difficulty. Player skill is measured on a scale of 1-10, with grades 1-9 representing pre-collegiate level, and 10 representing a post-collegiate mastery of piano.

A trained pianist graded 61 piano performance videos found on YouTube, using the 1-10 scale. Then, the researchers used half of these graded videos to train an artificial neural network to assess piano skills using only visual cues. This might seem like a tough job, but the researches say that some technical skills and virtuosic skills are unique to pianists that sit in the 9 to 10 range on the grading scale.

Technical skills include the difficulty and speed of the scales and arpeggios included in the music. To visually represent this idea, Chopin’s //Etude Op 25 No 11// was used. The four-octave A harmonic minor scale performed at high speed immediately indicates a level 10 performance.

Virtuoistic skills may include high-speed performances with the pianist playing eight-note intervals using their first and third fingers. This would be a skill that is incredibly difficult for the average player.

Using this method, the AI’s accuracy in skill level prediction was 74 per cent when compared to the trained pianist’s judgement.

Interestingly, the AI was then set the task of judging pianist’s skill levels aurally, which is where it struggled to compete with the human judicator. It achieved just 64 per cent accuracy, due to its “unfamiliarity with variety in style, clarity and dynamics from song to song”, the study says.

However, combining both visual and aural assessments resulted in a 75 per cent accuracy rating. This is considered “near-human accuracy”, according to New Scientist. Although the study showed that the technology is not quite capable of accurately evaluating music performances and lessons, it could provide a useful second opinion in any conflicted assessments. The researcher’s hope is that the automated evaluation can also make music tuition more accessible to disadvantaged socioeconomic groups.

Read the full paper at arxiv.org.

For more music tech news, click here.