YouTube robotics influencer Dave Niewinski has developed robots for all the things from driveable La-Z-Boy chairs to an AI-guided cornhole tosser and horse-drawn chariot racing.
His current Interactive Animatronic GLaDOS venture was amongst 9 winners within the Hackster AI Innovation Problem. About 100 contestants vied for prizes from NVIDIA and Sparkfun by creating open-source tasks to advance the usage of AI in edge computing, robotics and IoT.
Niewinski gained first place within the generative AI purposes class for his modern robotic primarily based on the GLaDOS information from sport collection Portal, the first-person puzzle platform from online game developer Valve.
Different high winners included contestants Andrei Ciobanu and Allen Tao, who took first prize within the generative AI fashions for the sting and AI on the edge purposes classes, respectively. Ciobanu used generative AI to assist just about strive on garments, whereas Tao developed a ROS-based robotic to map the within of a house to assist discover issues.
Harnessing LLMs for Robots
Niewinski builds customized purposes for robotics at his Armoury Labs enterprise in Waterloo, Ontario, Canada, the place he makes use of the NVIDIA Jetson platform for edge AI and robotics, creating open-source tutorials and YouTube movies following his experiences.
He constructed his interactive GLaDOS robotic to create a private assistant for himself within the lab. It handles queries utilizing Transformer-based speech recognition, text-to-speech, and huge language fashions (LLMs) working onboard an NVIDIA Jetson AGX Orin, which interfaces with a robotic arm and digital camera for interactions.
GLaDOS can observe his whereabouts within the lab, transfer in several instructions to face him and reply rapidly to queries.
“I like doing issues with robots that individuals will have a look at and say it’s not what they’d instantly anticipated,” he mentioned.
He wished the assistant to sound like the unique GLaDOS from Portal and reply rapidly. Happily, the gaming firm Valve has put the entire voice strains from Portal and Portal 2 on its web site, permitting Niewinski to obtain the audio to assist practice a mannequin.
“Utilizing Jetson, your common question-and-answer stuff runs fairly fast for speech,” he mentioned.
Niewinski used NVIDIA’s open-source NeMo toolkit to fine-tune a voice for GLaDOS, coaching a spectrogram generator community known as FastPitch and HiFiGAN vocoder community to refine the audio high quality.
Each networks are deployed on Orin with NVIDIA Riva to allow speech recognition and synthesis that’s been optimized to run at many instances the real-time charge of speech, in order that it will possibly run alongside the LLM whereas sustaining a clean, interactive supply.
For producing sensible responses from GLaDOS, Niewinski makes use of a domestically hosted LLM known as OpenChat that he runs in Docker from jetson-containers, saying that it was a drop-in substitute for OpenAI’s API. All of this AI is working on the Jetson module, utilizing the most recent open-source ML software program stack constructed with CUDA and JetPack.
To allow GLaDOS to maneuver, Niewinski developed the interactions for a Unitree Z1 robotic arm. It has a stereo digital camera and fashions for seeing and monitoring a human talking and a 3D-printed GLaDOS head and physique shell across the arm.
Making an attempt on Generative AI for Style Match
Winner Ciobanu, primarily based in Romania, aimed to enhance the digital clothes try-on expertise with the assistance of generative AI, taking a high prize for his EdgeStyle: Style Preview on the Edge.
He used AI fashions equivalent to YOLOv5, SAM and OpenPose to extract and refine knowledge from pictures and movies. Then he used Steady Diffusion to generate the photographs, which he mentioned was key to attaining correct digital try-ons.
This technique taught the mannequin how garments match completely different poses on individuals, which he mentioned enhanced the realism of the try-ons.
“It’s fairly useful because it permits customers to see how garments would look on them with out really attempting them on,” mentioned Ciobanu.
The NVIDIA JetPack SDK supplied all of the instruments wanted to run AI fashions easily on the Jetson Orin, he mentioned.
“It’s super-helpful to have a steady set of instruments, particularly whenever you’re coping with AI tech that retains altering,” mentioned Ciobanu. “It actually lower down on the time and problem for us builders, letting us focus extra on the cool stuff we’re constructing as a substitute of getting caught on tech points.”
Discovering Misplaced Gadgets With Robotic Help
Winner Tao, primarily based in Ontario, Canada, created a robotic to reduce the burden of looking for issues misplaced round the home. His An Eye for an Merchandise venture took high honors on the Hackster problem.
“Discovering misplaced objects is a chore, and up to date developments in zero-shot object detection and LLMs make it possible for a pc to detect arbitrary objects for us primarily based on textual or pictorial descriptions, presenting a possibility for automation,” mentioned Tao.
Tao mentioned he wanted robotic computing capabilities to catalog objects in any unstructured atmosphere — whether or not a lounge or giant warehouse. And he wanted it to additionally carry out real-time calculations for localization to assist with navigation, in addition to working inference on bigger object detection fashions.
“Jetson Orin was an ideal match, supporting all performance from textual content and picture queries into NanoDB, to real-time odometry suggestions, together with leveraging Isaac ROS’ hardware-accelerated AprilTag detections for drift correction,” he mentioned.
Different winners of the AI Innovation Problem embrace:
- George Profenza, Escalator individuals tracker, 2nd place, Generative AI Functions class
- Dimiter Kendri, Cooking meals with a neighborhood AI assistant utilizing Jetson AGX Orin, third place, Generative AI Functions class
- Vy Phan, ClearWaters Underwater Picture Enhancement with Generative AI, 2nd place, Generative AI Fashions class
- Huy Mai, Realtime Language Phase Something on Jetson Orin, 2nd place, Generative AI Fashions class
- Fakhrur Razi, Autonomous Clever Robotic Procuring Cart, 2nd place, AI on the Edge Open class
- Group Kinetika, Counting for Inspection and High quality Management with TensorRT, third place, AI on the Edge Open class
Study extra about NVIDIA Jetson Orin for robotics and edge AI purposes. Get began creating your individual tasks on the Jetson AI Lab.
