BriefsNewsAvailable now1 min read

Exa speeds up AI embedding on AWS chips by 47 percent

Exa improved embedding throughput and cut costs using custom NKI kernels and agentic workflows on AWS hardware.

Exa speeds up AI embedding on AWS chips by 47 percent — image from AWS
Image: AWS

Summary

Exa optimized its web indexing embedding infrastructure on AWS Inferentia2 and Trainium2 hardware. By using custom NKI kernels, memory changes, and agentic workflows, the team achieved a 47 percent improvement in performance and cost per million documents embedded.

What changed

  • Increased embedding throughput by 47 percent on AWS Inferentia2 and Trainium2 hardware.
  • Wrote a custom weight-stationary fused SwiGLU MLP kernel to eliminate high-bandwidth memory stalls.
  • Rewrote the attention kernel to cut FLOPS spent on padding and folded attention operations into single matmuls.
  • Used an agentic development workflow with Flyte to verify correctness, measure performance, and profile workloads.

Availability

Available now

“This post details how we optimized our embedding infrastructure for cheaper, more flexible inference workloads on AWS Neuron hardware ( AWS Inferentia2 and AWS Trainium2 ), and how an agentic development workflow drove the optimization.”

Founder takeaway

Engineering teams running large-scale vector embedding workloads can combine custom kernel development with agentic testing to use lower hardware inference costs.

Source

Researched and drafted with AI from the official sources above, then checked automatically before publishing. Spotted a mistake? Tell us.

SharePost on XLinkedIn

Related

More briefs

All briefs
  1. Google Research releases Deep Review, an agentic AI tool for checking documents — image from Google Research

    NewsAvailable now

    Google integrates Duplex AI calling into Tasks in Gemini

    Google's AI assistant can now make phone calls to businesses on your behalf to handle real-world tasks.

    Google Research1 min read

  2. aws launches amazon textract insights for generative ai document extraction — image from AWS

    NewsAvailable now

    aws launches amazon textract insights for generative ai document extraction

    AWS released Amazon Textract Insights, a generative AI feature that extracts structured data from multi-document files using natural language JSON schemas.

    AWS1 min read

  3. Thinking machines lab releases inkling 2 mini AI model — image from Hugging Face

    NewsAvailable now

    Thinking machines lab releases inkling 2 mini AI model

    Thinking Machines Lab introduces Inkling 2 Mini, a multimodal Mixture-of-Experts AI model built for local setups.

    Hugging Face1 min read

One email, every Monday

Briefs are published here through the day. Monday Heist is the email: one useful AI workflow, the files to use it, and the catch. The first issue is being prepared; subscribers get it first. Free, and you can leave in one click.

One email. Every Monday.