BriefsToolAvailable now
Amazon SageMaker AI adds GPU partitioning for real-time endpoints
Amazon SageMaker AI now supports GPU partitioning via NVIDIA MIG technology to run multiple models on a single accelerator.
Summary
Amazon SageMaker AI inference now supports GPU partitioning, letting users run multiple models on a single GPU instead of dedicating a full accelerator to each model. Using NVIDIA Multi-Instance GPU technology, SageMaker divides a GPU into up to seven isolated partitions with dedicated memory and compute. Users can configure partitioning when creating an endpoint hosting inference components without changing application code.
What changed
- Amazon SageMaker AI inference now supports GPU partitioning to run several models on a single GPU.
- It uses NVIDIA Multi-Instance GPU technology to divide a GPU into as many as seven isolated partitions.
- Users can configure GPU partitioning when creating an endpoint that hosts inference components.
Availability
Available now“GPU partitioning is available on P4d, P5, G7, and G7e instances in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt), and Europe (Ireland).”
Founder takeaway
Consider using GPU partitioning on supported instances if you serve smaller fine-tuned models, embeddings, or classifiers and want to avoid paying for unused GPU capacity.
Source
Researched and drafted with AI from the official sources above, then checked automatically before publishing. Spotted a mistake? Tell us.
Related
More briefs
AWS adds session endpoints to amazon bedrock agentcore runtime
Amazon bedrock agentcore runtime now features session endpoints to preview web apps live as coding agents build them.
AWS1 min read
nvidia releases ai cluster runtime for dynamo inference
NVIDIA released the AI Cluster Runtime tool to configure, deploy, and validate Kubernetes infrastructure for NVIDIA Dynamo AI inference.
NVIDIA1 min read
AWS adds web search to Kiro in AWS GovCloud US
AWS brings real-time internet search capabilities to the Kiro AI agent in AWS GovCloud US regions.
AWS1 min read
One email, every Monday
Briefs are published here through the day. Monday Heist is the email: one useful AI workflow, the files to use it, and the catch. The first issue is being prepared; subscribers get it first. Free, and you can leave in one click.
One email. Every Monday.