This post is authored by Nile Wilson, Software Engineer Intern at Microsoft.

Imagine Cup 2018 winning teams: smartARM (first place, front and center), iCry2Talk (second place, attired in pink), and Mediated Ear (third place, at the right).

Every year, Microsoft hosts the Imagine Cup, a global competition bringing together creative, bright, and motivated students to develop technologies that will shape how we live, work, and play. This year, tens of thousands of students from across the world registered for the competition, but only 49 teams were selected to compete in the World Finals. In addition to the first, second and third place winners, this year’s competition also awarded the top projects in Artificial Intelligence (AI), Big Data, and Mixed Reality.

Of the 49 finalists, team smartARM won the competition with their innovative, inexpensive, AI-enabled prosthetic hand. The team was comprised of Samin Khan from the University of Toronto and Hamayal Choudhry from the University of Ontario Institute of Technology. Although smartARM took home the top prize, all teams in the finals impressed the judges with their creativity and drive to have a positive impact on the world.

One other thing most all the winning teams had in common – they used AI as a core part of their solutions.

Recent developments have accelerated the application of machine learning technologies across a wide variety of fields, from self-driving cars to AI-guided disease detection. There was a palpable sense of excitement around the profound and untapped capabilities of AI among the teams that participated at this year’s Imagine Cup.

Winning AI Solutions

From helping farmers manage diseased plants to helping new parents identify the meaning of their babies’ cries, the Imagine Cup 2018 winners tackled a broad spectrum of problems. Although these solutions addressed different problems, their paths to success have a similar underlying structure – each team began by identifying a problem they were passionate about. Next, they carefully considered the resources available to them, including AI, and cleverly used those resources to build prototypes and solutions with potentially outsized impact. AI is truly in the mainstream now, empowering motivated individuals to turn their great ideas into reality and have impact on the world.

smartARM

Team SmartARM from Canada won the first place with their AI-infused robotic prosthetic arm, which is designed to provide users with a low-cost, multi-grip prosthesis. When individuals with upper-limb amputations decide whether they want to use a functional prosthesis, they are typically faced with the choice of purchasing a simple, single-grip arm or spending tens of thousands of dollars on a more advanced myoelectric arm. The high cost of advanced prostheses prevents many from having access to a device that could greatly aid in their day-to-day life. smartARM reduces this barrier by providing multi-grip functionality for approximately $100.

The smartARM on the stage of the Imagine Cup 2018 World Finals.

The trick to the low price achieved by SmartARM is the use of 3D printing and readily accessible low-cost cloud technology. The camera in the palm of the smartARM captures a video of whatever is in view of the arm. Once the user points the palm towards an object of interest, the video frames are sent to the Azure cloud where the Cognitive Services Custom Vision model helps to identify the object and returns most appropriate grip, based on the object’s size and shape. The grip determined by the model is then actuated on the smartARM once the user sends the activation signal.

This allows users to interact with a variety of objects, ranging from picking up house keys to holding a cup.

It’s a truly intriguing solution, the notion of integrating a camera and vision into a prosthetic arm. In a companion blog post, we plan to big a bit deeper into the smartARM story and the underlying technology. Meanwhile, to learn more about smartARM, you can visit their Imagine Cup team page and LinkedIn company page.

iCry2Talk

Team iCry2Talk from Greece won second place with their mobile app, which is designed to aid parents in identifying the needs of their crying baby. New parents and especially parents with hearing-impairments or mothers with postpartum depression often have difficulties when it comes to immediately identifying the needs of their crying baby. This result is unwanted stress for both the parents and the child, typically multiple times each day.

Promptly responding to the cry is essential for the healthy physiological and psychosocial development of the infant. Although babies cannot articulate their needs through the use of language, they do give hints through their cries and body language. The cries themselves contain information that could help parents identify what the baby’s immediate needs are, but these cues can be hard for untrained ears to pick up.

Team iCry2Talk strives to improve the quality of the communication and the relationship between parents and babies by helping parents understand their child’s cries.

Mother and child looking at the iCry2Talk app. Photo courtesy of the iCry2Talk team.

How does the app work? The parent records the cry of their baby through the app using a smart phone or a digital assistant, like Alexa or Cortana. This audio clip is then sent to the cloud, processed, and classified using their custom Deep Learning models. The result is then sent back to the phone or digital assistant within seconds, and the parent is notified of the meaning of the cry through text, image, voice feedback, and sign language.

The team takes a holistic approach by not only focusing on the technology, but by also building and engaging with a community consisting of parents and doctors, in addition to designing for inclusivity, accessibility, and personalization.

iCry2Talk continues to collect donated audio clips of cries from parents involved in their community effort and is constantly improving their models. Parents with babies up to 12 months old who wish to contribute to the database and join the community can contact iCry2Talk through their Facebook page or e-mail them directly at [email protected].

To learn more about iCry2Talk you can visit their Imagine Cup team page or their website.

Mediated Ear

Team Mediated Ear from Japan took the third place with their mobile app designed to help the hearing-impaired clearly hear the voice of specific persons in noisy environments. Individuals with hearing loss may use hearing aids to amplify sound, but often find it difficult to isolate single speakers in noisy environments. This can make conversations in public settings and meetings difficult as multiple people may be speaking at once. The team began to develop Mediated Ear when a friend of theirs with hearing loss talked about their challenges when communicating in noisy, multi-speaker environments.

Speaker registering their voice to be recognized by the Mediated Ear mobile app. Image courtesy of the Mediated Ear team.

Clearly isolating individual speakers from each other and from background noises is not a simple task. The team had to work hard to develop an approach that would reliably isolate individual speakers from a mixed audio source. Mediated Ear works as a smartphone application that listens to the current conversation and allows the user to adjust the volume for individual speakers played through their earphones.

To isolate a given speaker’s voice from other voices and background noise, the user hands the phone over to the speaker of interest and asks them to speak into the phone for one minute. After the speaker reads the passage displayed through the app, the audio file of the voice is sent into the cloud, where it is processed and fed into a modified WaveNet deep learning model. Once the model learns the speaker’s voice, the app allows the user to pick up on the speaker’s voice and selectively amplify it, making it easier for the user to understand what is being said and confidently engage in the conversation.

With Mediated Ear, people with hearing impairments have control over who they hear and have an easier time focusing on the people they want to listen to in noisy environments.

Want to learn more about Mediated Ear? Visit their Imagine Cup team page or their website.

SochWare

Team SochWare from Nepal won the AI Award for their mobile app to help farmers identify plant diseases and take steps to mitigate crop damage. Agriculture plays a critical role in the livelihood of Nepal and its people, but difficulties in farming have led to a decline in agriculture. When farmers notice a diseased plant, it is often difficult to recognize what kind of disease the plant is inflicted with. The challenge of disease identification makes it hard to properly address the situation. Farmers often suffer losses when they either do not treat crops or apply improper chemicals to handle disease. There are also situations where excessive chemical use on crops leads to negative health effects on consumers.

Local farmer checking plant disease status with the E-Agrovet mobile app. Photo courtesy of the SochWare team.

Understanding the vitality of agriculture to their country and their families, Team SochWare decided to focus their efforts on developing a solution for this problem. Their solution takes the form of E-Agrovet, a mobile app that uses computer vision to help farmers identify plant disease and learn the proper next-steps for treatment.

How does E-Agrovet work? The farmer takes a photo of the plant of interest through the app. This photo is then sent to the cloud, processed, and fed into their Cognitive Services Custom Vision model. A report is generated based on the results of the Deep Learning image classification model and is sent back to the notification hub on the mobile phone. This report informs the farmer of what the disease is, how to mitigate it, and may also connect the farmer with experts to allow them to act.

Through E-Agrovet, SochWare strives to aid farmers and reduce the use of unnecessary chemicals on crops, improving the quality of life for everyone.

Want to learn more about SochWare? Visit their Imagine Cup team page or their website.

DrugSafe

Team DrugSafe from India won the Big Data Award for their consumer-side and vendor-side apps that help fight the consumption and distribution of counterfeit drugs. The development all began when a friend was suddenly in a lot of pain. The team was surprised to find that their friend had unknowingly taken counterfeit drugs for his diabetes and suffering as a result. Upon further investigation, the team became aware of widespread drug counterfeiting, estimated to be a $75 billion industry in 2010, according to the World Health Organization (WHO). Realizing the widespread prevalence and severity of fake medicine, and its role in the unnecessary spread of disease and suffering, the team sought to develop a solution. Because this is such a large-scale problem – one that affects both consumers and legitimate drug vendors – the DrugSafe team decided to develop both a consumer-side and vendor-side solution via mobile apps and an online dashboard.

The Drugsafe mobile app (consumer side). Image courtesy of the DrugSafe team.

What do these apps do?

The mobile app consists of a simple interface that allows users to take photos of drug labels to check for anomalies in the text and label color. The app sends the photo through a custom pipeline involving Azure Cosmos DB and Cognitive Services and notifies the user of the validity of the drug. Users have the option of reporting the drug if it is found to be illegitimate. Selecting to report the issue will take the user to the built-in chatbot to expedite the reporting process. The app also has a community component and can warn users about pharmacies that are seeing an increase in counterfeit drug selling.

The vendor app allows drug vendors to monitor the conditions of their deliveries using an MXChip IoT Board, to ensure that their drugs are delivered safely. The IoT board records information such as temperature, pressure, humidity, and acceleration, which can help vendors monitor the condition of their shipments.

Through the adoption and use of the DrugSafe app, the team hopes to reduce the spread of disease and help individuals gain access to authentic safe medication.

To learn more about DrugSafe you can visit their Imagine Cup team page.

The Impact of AI

The number and quality of AI-centric entries at this year’s Imagine Cup tells us that the next generation of developers recognize AI’s game-changing potential. Although each of these student projects addressed very different needs, teams took a holistic approach and appropriately infused their solutions with AI.

We are inspired by the boldness of these young innovators. Their work reinforces our own mission, to “empower every person and every organization on the planet to achieve more”. We couldn’t be more excited to support the next generation of developers at future Imagine Cups and to see how their ideas – coupled with AI – will move and shake the world.

If you are interested to build your own intelligent solutions, we recommend getting started with the AI School – we have free tutorials that provide step-by-step instructions on how to build real world solutions on the Microsoft AI platform.

Nile