Autonomous operations for GPU infrastructure.
Built for GPU cloud providers and HPC operators.
Find the root cause. Automate the fix. Escalate what remains.
Built by engineers from
- Meta
- AMD
- Amazon
- Klaviyo
Beyond monitoring
Monitoring raises the alert. YovaSense drives it to resolution.
Monitoring tells your team that a GPU disappeared or a job failed. Finding out why still takes an engineer, pulling evidence by hand from the scheduler, the servers, the GPUs, the network and the job.
What monitoring tells you
- A node is out of service.
- A GPU stopped reporting.
- A network link keeps dropping.
- A job failed.
What YovaSense adds
- One incident instead of four alerts.
- A named GPU and its likely cause.
- Capacity back in service, verified.
- A technician ticket only when needed.
How it works
From the first alert to verified recovery.
Every incident runs the same three steps, day or night.
- 01
Find the root cause
YovaSense reads your GPUs, servers, network, scheduler and workloads together. A burst of alerts becomes one incident with a named cause.
GPUs · servers · network · scheduler · workloads
- 02
Automate the fix
It runs the recovery your policy allows and asks a person before anything beyond it. Nothing closes until YovaSense has checked that the capacity is healthy again.
reset · restart · reschedule · verify
- 03 · to a person
Escalate what remains
Anything that needs hands on hardware becomes a prioritized ticket, with the evidence and the repair steps attached.
An engineer in the loop. Not an engineer on every alert.
YovaSense in Slack
From alert to resolved, in one Slack thread.
YovaSense posts what broke and why, runs the fixes your policy allows, asks before anything else, and confirms the recovery in the channel your team already watches.
Works with Slack
- YovaSense (app), 10:42 AM: New incident. A training job stalled on node r1n2. Status: resolved.
- YovaSense, 10:43 AM: Root cause found. GPU 4 on node r1n2 stopped responding, and the rest of the node is healthy. Proposed fix: reset GPU 4 and resume the job from its last checkpoint. Scope: one node, one job. Approval: required by your policy. Approved by Maya.
- Maya, 10:44 AM: Looks right. Go ahead.
- YovaSense, 10:46 AM: Fix applied. GPU 4 reset cleanly and the job picked up from its last checkpoint.
- YovaSense, 10:47 AM: Recovery verified. GPU 4 passed its health checks and node r1n2 is back in service.
- YovaSense, 10:48 AM: One item needs a person. GPU 4 should be inspected at the next maintenance window, and the ticket is open with the evidence attached. Ticket OPS-2047, inspect GPU 4 on node r1n2, hardware, normal priority. Maya reacted with a check mark.
The cause comes first
Each incident arrives as one message, with the root cause named.
Approve from the thread
A fix outside your policy waits for one click from your team.
Closed with proof
YovaSense confirms the recovery and tickets whatever is left.
What it handles
Six families of infrastructure incidents.
The same three steps run for every one, across training and inference.
GPU hardware failures
Faulty GPUs, memory errors and GPUs that drop off the node.
Network and interconnect
Slow or failing links between GPUs, servers and racks.
Performance degradation
Jobs that slow down without failing: stragglers, throttling, regressions.
Drivers and services
The drivers, firmware and services every GPU depends on.
Unavailable GPU nodes
Nodes pulled out of service, and the capacity stranded with them.
Power and cooling
Thermal and power faults, from one GPU to a whole rack.
FAQ
Common questions.
What does YovaSense take off my team's plate?
The investigation and the routine recovery. YovaSense works out why GPU capacity went down, runs the fixes your policy allows and checks that the capacity is healthy again, so your engineers only see the incidents that need them, with the evidence already gathered.
Does it replace our monitoring?
No. YovaSense works on top of the monitoring, schedulers and logs you already run. Monitoring raises the alert, and YovaSense carries it through to a verified recovery.
How much does it do on its own?
As much as you allow. Your policy sets which recovery actions run automatically, anything beyond it waits for a person to approve, and a limit caps how much capacity YovaSense can take out of service by itself.
How does it work with Slack?
YovaSense posts each incident to the channel you choose, with the root cause and the proposed fix. Your team approves or escalates from the message, and the thread keeps a record of every step through to recovery.
What infrastructure does it support?
NVIDIA and AMD GPUs, Slurm and Kubernetes, training and inference, on clusters you own or rent. One deployment covers several clusters, and each keeps its own policy.
Where does our data go?
YovaSense runs in your environment and reads operational telemetry, not your models or datasets. Incident updates go only to the Slack channels and ticketing tools you connect, and it has only the access you grant it.
Couldn't a general AI assistant do this?
Because diagnosing a GPU fleet takes context a chat agent doesn't have: how every job maps to its GPUs, servers, network links and racks, and what each kind of failure looks like across them. YovaSense pairs known failure signatures with AI reasoning, and verifies every fix before it closes an incident.
How do we evaluate it?
Write to us and we'll start with your fleet and the incidents that take the most of your team's time. YovaSense connects to the telemetry you already collect, and you decide what it may do on its own before it does anything.
Get access
See it on your own fleet.
Tell us about your fleet, and we'll show you what YovaSense would find, fix and escalate on it.
Or write to david@yovasense.com