multimodal

Overview

A collection of 7 multimodal AI research skills from Orchestra Research, providing structured engineering instructions for coding agents to work with vision, speech, and generative models. Part of 83 AI research engineering skills designed to enable coding agents to conduct AI research experiments.

Key Features

• CLIP — OpenAI's vision-language model for zero-shot classification (320 lines, 25k+ GitHub stars)
• Whisper — Robust speech recognition supporting 99 languages (395 lines, 73k+ stars)
• LLaVA — Vision-language assistant for image chat at GPT-4V level (360 lines)
• BLIP-2 — Vision-language pre-training with frozen image encoders and LLMs
• SAM — Segment Anything Model for universal image segmentation
• Stable Diffusion — Text-to-image generation via HuggingFace Diffusers with SDXL and ControlNet (380 lines)
• AudioCraft — Meta's audio generation framework for music, sound effects, and audio

Who is this for?

AI researchers and ML engineers working on multimodal applications who want their coding agent to help with vision, speech, and generative model experiments. Ideal for teams building products that combine text, image, audio, and video understanding.