Six Arm Instances, 23 Profiling Runs: A Lab Notebook

A working index of every profiling run I made with Arm Performix across two clouds, what each one produced, and where the limits are. Total cost: about $9.55.

The long-form write-up with the full reasoning is in Profiling aarch64 in the Cloud. This page is the shorter version: what ran, on what, and what it showed.

The instances

Graviton4 and Axion report the same MIDR (0x410fd4f1), so those two are genuinely the same core behind different hypervisors. Resizing c8g.xlarge → c8g.4xlarge did not change the counter budget; AWS documents that *8g needs 24xlarge for full PMU event support.

Which recipes actually ran

23 runs: 13 succeeded, 10 failed. The failures are as informative as the successes.

Three of nine recipes run on a normal cloud VM. The other six need bare metal, a larger instance, or a newer kernel.

The split isn’t arbitrary, which took me a while to notice. Everything that works on a VM gets its data from the kernel or a timer: system_utilization reads /proc, code_hotspots falls back to a software timer, and syscall_trace_summary uses strace, which needs no PMU whatsoever because ptrace intercepts syscalls at a boundary the kernel already mediates. Every recipe that fails needs the PMU or SPE, which is real hardware the hypervisor decides whether to hand over. The dividing line is hardware observability against software observability.

Who each answer is actually for

Reading back over the runs, they sort into three groups depending on who’d care about the result.

Fleet-level Arm-versus-x86 decisions

Procurement across thousands of instances, where a few points of performance per watt compound into real money. That needs characterisation rigorous enough to defend a capital commitment, and it’s the case where bare metal earns its cost.

On c8g.metal-24xl, memory_access produced roughly 140 columns per function. Latency histograms bucketed from 0 to 4096+ cycles, and load-source attribution with both cost and count across L1C, L2C, Peer, Local Cluster, LLC, Peer Cluster, Remote and DRAM. Load source is the column that changes what you do next: served from a peer core’s cache means fix your thread pinning, served from DRAM means fix your data layout. Same slow load either way.

asct mapped the hierarchy and independently corroborated my own benchmark:

My pointer chase measured 126 ns/access under load. Two unrelated methods agreeing to within the queueing delay you’d expect is what makes either number defensible when somebody senior asks where it came from.

→ The full bare-metal results

Migrating an existing workload from x86

This is the case I find most interesting, because it’s the one where the tooling and the reality diverge most.

Picture someone who has run on x86 for years, hears the price-performance story, moves to Graviton, and goes from having no comparison at all to suddenly having one. They changed the ISA, the OS, the instance family, the disks and the networking in a single step. Something shifted, maybe for the better, maybe not, and the whole job is working out what.

That’s a variable-isolation problem before it’s a profiling problem. My test workload was built so I already knew the answer: a pointer chase where one load stalls ~126 ns on DRAM and everything around it is free. Then two binaries compiled from identical source, differing only in whether the vectoriser ran, which gave 4.17 → 9.65 GFLOP/s on Graviton4 and 1.72× on Graviton2, which has NEON but no SVE.

Here’s the divergence. Anyone migrating a workload is, by definition, running on the instances they deploy on. Those are VMs. So the six recipes that would tell them the most are the six they can’t reach.

There’s a narrower gap too. instruction_mix doesn’t answer the question its name implies, because it’s a static whole-image count including every linked library. The binary under test contributed 284 instructions to a total of 394,990; the rest was libc, libm and the loader. So it can’t tell you whether your hot loop vectorised, which is more or less the first thing anyone asks after a migration.

→ Field notes on what breaks

Everything that isn’t a server

Arm ships in rather more places than data centres, and the same profiling problem presumably follows it into cars, robots and edge devices. None of my runs touched any of that, so I’ll leave it as a guess: the constraint pattern probably transfers, because the interesting analysis needs hardware access that production deployments rarely grant.

The headline result: interrupt sampling blames the wrong instruction

The single most useful thing I measured. Three instructions in a loop; the middle one stalls ~126 ns on DRAM, the other two are free.

1
subs w0, w0, #0x1     free
2
ldr  x19, [x19]       ~126 ns DRAM stall  ← the actual cost
3
b.ne loop             free

The bottom two rows are the same machine, the same binary and the same workload, differing only in sampling mechanism.

Interrupt-based sampling is how perf and most profilers work on every architecture, and it credits the free branch while missing the expensive load. Arm’s Statistical Profiling Extension, on the same machine against the same binary, puts zero samples on the branch.

The Axion row is the one that closes off the obvious objection. Google exposes no PMU at all, so perf falls back to a software timer, and skid persists anyway. So it isn’t an artifact of how PMU overflow interrupts get delivered. It’s inherent to interrupt-driven sampling.

→ The full experiment

Two bugs

The first is the one I’d actually file. On Axion, cpu_microarchitecture reports success and returns an empty table. Run feece99fdfaa: functions-capture-metrics.csv came back with a header row and nothing under it, 2,818 bytes of column names promising IPC, frontend stalls, Topdown L1 and L1I MPKI, all blank. The same recipe on AWS bare metal produced 23 populated rows.

The comparison is what makes it a bug rather than a limitation. On AWS with two counters it fails loudly and correctly, with INSUFFICIENT_PMU_COUNTERS. On Axion with zero it goes quiet. And memory_access on that same target fails cleanly with SPE_NOT_CONFIGURED, so the capability check clearly exists and works. It just isn’t wired into that one path.

The second is smaller but odder. That same run can’t be rendered at all, and the error points somewhere unrelated: “incompatible schema version 1.1 for component sl-collect-symbols; must be 1.1 or above”. I compared both manifests, and the run that renders fine and the run that fails both declare schema 1.1. The version is a red herring. The real cause is that there’s no symbol data to load.

A green result with no data is worse than a red one. At least a failure tells you to go and look.

What I’d check next

• Whether the empty-table case also occurs at exactly 1 counter or only at 0, which would locate the boundary properly.
• cache_sharing never appeared on any target, including metal, and I never worked out why.
• cmn_analysis needs the CMN interconnect PMU, which on Amazon Linux 2023 wants kernel 6.12 or newer. I was on the default 6.1.176, so a kernel upgrade would probably have given me the recipe.
• Bare metal on Google Cloud, which needs two separate quota increases and would mostly confirm that the same core behaves the same way behind a different hypervisor.

The workload is about seventy lines of C and it’s printed in full in the long-form post, so anyone can rebuild these numbers on their own instance. It compiles with cc -O2 -g -fno-omit-frame-pointer -o memlat memlat.c and runs as ./memlat chase 10. On an M4 Pro laptop it gives 118 ns/access for the chase against 1.02 ns for the sequential walk, which is the same shape as the Graviton and Axion figures above even though neither of those machines is a Mac.