BriefsNewsAvailable now
Exa speeds up AI embedding on AWS chips by 47 percent
Exa improved embedding throughput and cut costs using custom NKI kernels and agentic workflows on AWS hardware.
Summary
Exa optimized its web indexing embedding infrastructure on AWS Inferentia2 and Trainium2 hardware. By using custom NKI kernels, memory changes, and agentic workflows, the team achieved a 47 percent improvement in performance and cost per million documents embedded.
What changed
- Increased embedding throughput by 47 percent on AWS Inferentia2 and Trainium2 hardware.
- Wrote a custom weight-stationary fused SwiGLU MLP kernel to eliminate high-bandwidth memory stalls.
- Rewrote the attention kernel to cut FLOPS spent on padding and folded attention operations into single matmuls.
- Used an agentic development workflow with Flyte to verify correctness, measure performance, and profile workloads.
Availability
Available now“This post details how we optimized our embedding infrastructure for cheaper, more flexible inference workloads on AWS Neuron hardware ( AWS Inferentia2 and AWS Trainium2 ), and how an agentic development workflow drove the optimization.”
Founder takeaway
Engineering teams running large-scale vector embedding workloads can combine custom kernel development with agentic testing to use lower hardware inference costs.
Source
Researched and drafted with AI from the official sources above, then checked automatically before publishing. Spotted a mistake? Tell us.
Related
More briefs
Google integrates Duplex AI calling into Tasks in Gemini
Google's AI assistant can now make phone calls to businesses on your behalf to handle real-world tasks.
Google Research1 min read
aws launches amazon textract insights for generative ai document extraction
AWS released Amazon Textract Insights, a generative AI feature that extracts structured data from multi-document files using natural language JSON schemas.
AWS1 min read
Thinking machines lab releases inkling 2 mini AI model
Thinking Machines Lab introduces Inkling 2 Mini, a multimodal Mixture-of-Experts AI model built for local setups.
Hugging Face1 min read
One email, every Monday
Briefs are published here through the day. Monday Heist is the email: one useful AI workflow, the files to use it, and the catch. The first issue is being prepared; subscribers get it first. Free, and you can leave in one click.
One email. Every Monday.