AutoViz: GPU Trace Analysis on LLM Inference Workloads

April 2026 · Christopher Vu, AMD Silo AI Team

I share trace analysis architecture in this article. Specific results are omitted under NDA.

Context GPU traces are composed of millions of kernel events. Different vLLM builds may see kernels get renamed, split, or fused, making comparison difficult. In collaboration with AMD, I built a system that ingests two build traces, offers various modes of comparison, and attempts to highlight potential optimization opportunities. Data: Llama-3.1-405B-Instruct (FP8, tensor-parallel across 8 GPUs) and DeepSeek-V3 (MoE, TP=4), on two builds each. Each DeepSeek build is profiled a decode-heavy (1k in, 100 out) and a prefill-heavy (100k in, 1k out) workload.

Kernel Patterns

Kernel Patterns as Unit of Comparison Most kernels are part of some subsequence (say, a decoder step/forward pass) that repeats many times throughout the trace. Thus, I treat repeating kernel subsequences as the unit of comparison within a given build. We detect repeating kernel loops across all ranks via suffix-based loop detection alignment and annotate every kernel with its pattern and position.

Pattern detection schematic showing a repeating kernel subsequence bracketed as a pattern, with each event annotated by pattern name and position across multiple aligned ranks.

Cross Build Pattern Matching Patterns are detected independently for each build, so two patterns in different builds may have different functions. For a sensible notion of 1:1 comparison, we must find, match, and analyze similar detected patterns in builds A and B.

I thought it best to split this process into two stages.

Stage one: Analyze each cross-build pattern pair on 1) Jaccard similarity over the sets of operation names and 2) Levenshtein similarity over the ordered sequences, then score by weighted sum (70/30 by default). 'Successful' matches are determined with a greedy bipartite assignment picking the best non-conflicting matches above a threshold, with similarity scores (+Jaccard/Levenshtein) shared in case other pairs need to be inspected.
Stage two: On any set of pairs that the analyst wishes to inspect, a suite of procedural comparisons are generated (per-pattern timing delta, a per-operation duration diff table, and a waterfall chart of per-op deltas).

Results This setup matched high confidence (>0.78) scores on almost all traces (even in MoE builds), and revealed >70k kernels present in some build patterns but absent in others. AutoViz^[1] became the standard trace ingestion method for our 8 person team, processing over 12M kernels.

Procedural and Learned Anomaly Detection

Procedural Baseline With matched patterns, regression detection can be operationally defined in two parts: find abnormally slow patterns, and localize blame to specific kernels. I define a run as one contiguous pattern execution on a given rank. As a baseline, runs are flagged beyond 2σ of their (pattern, rank) group mean (flag rates ran between 0.1–2.9% per combo), then each kernel duration is compared against the median of σ-normal runs at the same position.

AutoEncoder This provides a good baseline, but kernel runs can be anomalous while having features inside their 2σ threshold. Catching these requires modeling the normal ‘shape’ of a run rather than its summary statistics. Thus, we train a 1d convolutional autoencoder on run-level data, using reconstruction error to attribute individual kernels.

Two-stage matching architecture: build A and build B patterns scored by Jaccard over op sets and Levenshtein over op sequences into a 70/30 weighted score, greedy bipartite assignment producing candidate pairs, then analyst review feeding stage two timing deltas, per-op diff table, and waterfall comparison.

Input Representation: Each run becomes a two channel kernel sequence: log1p duration, log1p idle time.
Scope and Norm: One model per (build, scenario, pattern), trained on σ-normal runs. Channels are MAD scaled (due to the heavy tails), fit only on σ-normal runs.
Mask: Most patterns won't take up the full AE input, and thus no loss/error is computed on the padding kernels.
Architecture: Strided Conv1d encoder (2→16→32 channels, k=5, stride=2) followed by a 1 channel bottleneck (32→1; ~8x compressed latent space. strided rather than max pooling, as the latter leads to outlier kernels reconstructing too accurately). The decoder mirrors the encoder with NN upsampling. 9k parameters was the sweet spot in testing for fitting patterns well without reconstructing outliers.
Loss: (Masked) reconstruction error over the two channels, with runs flagged on the maximum per-position standardized error (max not mean, so that extreme kernels aren't diluted by normal ones). As a result of our input representation, the anomalous kernel is easy to identify, as well as whether its run time or idle time is anomalous. Theshold is set to the 2% validation FPR mark.

Results Unfortunately, there's no ground truth for truly 'anomalous' runs, but we can get close with synthetic patterns. Using real trace patterns, 500 of each of the following anomalous pattern types were generated and compiled into an evaluation, yielding promising results from the autoencoder: magnitude spike (pick one kernel at random and quintuple^[2] its length, recall 100%), idle stall (add 5 microseconds to a kernel's idle time, recall 98.4%), swap (swap two kernels 3 positions apart, recall 100%), broad drift (multiply all kernel runtimes by 1.1x, recall 69.4%).