From zero to Nsight

Kubernetes basics, GPU basics, then profiling a real GPU on our stg cluster. No prior knowledge assumed.

How to use this Do the checkboxes in order (your progress is saved in this browser). Every code block has a Copy button. All cluster commands target stg; nothing here touches prod.

1. What is Kubernetes?

You have software (a model server) that must run on many machines, restart if it crashes, and be found by other services. Kubernetes (“k8s”) is the system that does that bookkeeping. You tell it what you want (“run 3 copies of this model server, each with 1 GPU”) and it finds machines for them and keeps them running.

Term	Plain meaning	In our world
Cluster	A pool of machines managed together, plus a “control plane” (the brain that decides what runs where).	is the stg online-serving cluster in AWS us-east-1.
Node	One machine in the cluster (here, an AWS EC2 instance).	e.g. a `g7e.8xlarge` (1 GPU, ~32 vCPU) or a `c8i.16xlarge` (CPU only).
Pod	The smallest thing Kubernetes runs: one or more containers that share a network address and run together on one node.	One copy of a model server.
Container	A packaged program plus everything it needs (a Docker/OCI image), running isolated.	`model-server`, `model-runtime` (Triton+vLLM).
Namespace	A folder-like partition inside a cluster so teams don’t collide.	holds the serving pods.
Deployment	“Keep N identical pods alive.” Replaces pods that die.	Each model (like `…llmqu-v1-s2`) is a deployment of pods.
Scheduler	The component that picks a node for each new pod.	It matches the pod’s CPU / memory / GPU asks to nodes with room.
kubectl	The command-line tool you use to talk to a cluster.	Everything in this tutorial.

2. Pods & containers, the details that matter

Containers in one pod share localhost, so the gateway-side model-server can call Triton at localhost:port with no network hop.
Requests are what the scheduler reserves for you; limits are the cap. Our pods set them equal (e.g. 6 CPU / 24Gi).
Init containers run to completion before the app starts (download model weights, fetch secrets). A native sidecar is an init container with restartPolicy: Always, which keeps running beside the app. In our pod, model-runtime (Triton+vLLM) is exactly that.
nodeSelector says “only schedule on nodes with this label”. Taints/tolerations keep ordinary pods off special nodes; GPU nodes are tainted nvidia.com/gpu and GPU pods tolerate it.

3. Namespaces and contexts

A context is a saved shortcut in ~/.kube/config: which cluster + which login + default namespace. You have many (dev/stg/prd, many regions). A kubectl command without --context hits whichever one is “current”, so on this machine that is a prd control-plane context. That’s why this tutorial always passes --context explicitly.

# set once per terminal; $K then means "kubectl, aimed at stg online, namespace "
K="kubectl --context  -n "

# list every context you have
kubectl config get-contexts

# which one is current (don't rely on it)
kubectl config current-context

Safety habit Before any command that changes something (apply, delete, exec into something running traffic), read the --context in the command. Read-only commands (get, describe, logs) are safe to explore with.

4. GPUs in Kubernetes

Kubernetes doesn’t know what a GPU is out of the box. On each GPU node a program called the NVIDIA device plugin tells Kubernetes “this node has nvidia.com/gpu: 1”. A pod asks for one by putting nvidia.com/gpu: 1 in its limits. Two consequences that explain a lot of the GPU-cost conversation:

GPUs are whole numbers. You can’t ask for half. One pod with the request holds the entire card, even if it uses 5% of it.
So “GPUs per model” = “pods per model” unless you add sharing: MIG (hardware slices), MPS (shared processes), or several models in one Triton. That is what “heterogeneous packing” is about.

5. Explore our stg cluster (read-only)

Set $K (above), then work through these. Tick each one when you’ve run it.

List the machines, with their type.

kubectl --context  get nodes -L node.kubernetes.io/instance-type | head -30

You’ll see hundreds of CPU nodes (c8i.*) plus a few dozen g6.*, g7.* and g7e.*. Those are the GPU ones.

Count nodes per instance type.

kubectl --context  get nodes -L node.kubernetes.io/instance-type --no-headers | awk '{print $NF}' | sort | uniq -c | sort -rn

List the pods in the serving namespace.

$K get pods -o wide | head -20

Columns: READY (containers up / total), STATUS, RESTARTS, AGE, and which NODE each runs on.

Look inside the pod’s spec.

P=$($K get pods -o name | grep llmqu-v1-s2 | head -1)
$K get $P -o jsonpath='{range .spec.initContainers[*]}{.name}{"  "}{.restartPolicy}{"  "}{.resources.limits}{"\n"}{end}'

Find the line for model-runtime: restartPolicy: Always (sidecar) and nvidia.com/gpu: 1. That’s the pod’s GPU.

See what the node says about its GPU.

NODE=$($K get $P -o jsonpath='{.spec.nodeName}')
kubectl --context  describe node $NODE | grep -E 'nvidia|instance-type|gpu-name|Allocatable' -A1 | head -30

You should see karpenter.k8s.aws/instance-gpu-name=rtx-pro-server-6000 and nvidia.com/gpu: 1 under Allocatable.

6. What a GPU is, in five minutes

A GPU has thousands of small cores grouped into SMs (streaming multiprocessors). It is good at doing the same arithmetic on huge amounts of data at once.
A kernel is a function that runs on the GPU. The CPU launches kernels; each launch costs a few microseconds of overhead.
Device memory (GDDR/HBM) is big but slower; L2 cache and on-chip memory are small but much faster. Moving data is often the real cost.
Python/PyTorch lines like a @ w become one or more kernels on the GPU. Profilers show you those kernels.

7. Memory-bound vs compute-bound

Every kernel is limited by either math speed (FLOPS) or data movement (bytes/second). Which one tells you what to optimize.

	Memory-bound	Compute-bound
Typical op	Elementwise ops, LLM decode (one token at a time)	Big matrix multiplies, LLM prefill
Limited by	Memory bandwidth (GB/s)	Tensor-core throughput (TFLOPS)
Helped by	Smaller weights (INT4), fusing ops, faster memory	Better tiling, lower precision math, more SMs

Why this matters for Generating a token reads all model weights once but does little math per byte. That makes decode memory-bound, so the memory-bandwidth gap between GPUs decides latency. It’s the reason the L4 lost to the RTX PRO 6000 for this model.

8. Our hardware: g6 vs g7e

Instance	GPU	GPU memory	Memory bandwidth (approx.)	Role
`g6.*`	NVIDIA L4	24 GB	~300 GB/s	Cheaper; fine for small encoders; too slow for (420–570 ms in earlier tests)
`g7e.*`	RTX PRO 6000 Blackwell	96 GB (shown as 98 GB in nvidia-smi MiB)	~1.8 TB/s	Production and upcoming experiments

The nvidia-smi MiB figure is what you’ll see in the pod: 97887 MiB.

9. Create your lab pod

We’ll run your own pod with a GPU and the same image the production sidecar uses (it ships nsys, ncu, nvcc, PyTorch and Triton). The manifest below is already saved as pod.yaml next to this file.

apiVersion: v1
kind: Pod
metadata:
  name: nsight-lab
  namespace: 
  labels: {owner: seanb, purpose: nsight-learning}
spec:
  activeDeadlineSeconds: 14400        # self-destructs after 4 hours
  restartPolicy: Never
  nodeSelector:
    ap.apple.com/node-group: -2-gpu-workers
    node.kubernetes.io/instance-type: g7e.8xlarge
  tolerations:
  - {key: nvidia.com/gpu, operator: Exists}
  containers:
  - name: lab
    image: //triton-vllm-container:1.0-1
    command: ["sleep", "infinity"]     # just keep the pod alive; we exec in
    resources:
      limits:   {cpu: "4", memory: 24Gi, nvidia.com/gpu: "1"}
      requests: {cpu: "4", memory: 24Gi, nvidia.com/gpu: "1"}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities: {drop: ["ALL"]}
      seccompProfile: {type: RuntimeDefault}

Read it top to bottom: a pod, in this namespace, that dies after 4h, lands only on a g7e GPU node, and asks for 1 GPU.

Create it (changes stg, uses one GPU).

kubectl --context  apply -f ~/workplace/nsight-lab/pod.yaml
$K get pod nsight-lab -w          # Ctrl-C once READY is 1/1

If it stays Pending: no free g7e GPU. Run $K describe pod nsight-lab and read “Events” at the bottom; the scheduler says why. Stg has only a handful of free GPUs at any time.

Open a shell inside it and look at the GPU.

$K exec -it nsight-lab -- bash
# now you are INSIDE the pod:
nvidia-smi
which nsys ncu nvcc python3

Expect “NVIDIA RTX PRO 6000 Blackwell Server Edition”, ~97887 MiB total and ~1.4 GB used.

Copy the exercise script in (run this on your Mac, not in the pod).

$K cp ~/workplace/nsight-lab/lesson1.py nsight-lab:/tmp/lesson1.py
$K cp ~/workplace/nsight-lab/sweep.py   nsight-lab:/tmp/sweep.py

A tar: Exiting with failure status message is harmless here; check the file landed with $K exec nsight-lab -- ls /tmp.

10. Your first profile with Nsight Systems

Nsight Systems (nsys) records a timeline: which GPU kernels ran, how long each took, when the CPU launched them, and memory copies. Think “a stopwatch on everything”. First look at the script:

import torch, torch.cuda.nvtx as nvtx
x = torch.randn(1 << 28, device="cuda", dtype=torch.float16)  # 512MB: bandwidth-bound
w = torch.randn(8192, 8192, device="cuda", dtype=torch.float16)
a = torch.randn(8192, 8192, device="cuda", dtype=torch.float16)
for _ in range(3): x.mul_(1.0001); a @ w   # warmup
torch.cuda.synchronize()
with nvtx.range("memory_bound_scale"):
    for _ in range(20): x.mul_(1.0001)
with nvtx.range("compute_bound_gemm"):
    for _ in range(20): a @ w
torch.cuda.synchronize()

x.mul_(1.0001) multiplies 268M numbers by a constant: almost no math, lots of memory traffic → memory-bound.
a @ w is an 8192×8192 matrix multiply → compute-bound.
nvtx.range adds labelled regions so you can find them in the timeline.
The warm-up loop makes the first-use costs (CUDA init, kernel selection) not pollute the measurement.

Profile it (inside the pod).

cd /tmp
nsys profile -o l1 -f true -t cuda,nvtx python3 lesson1.py

-o l1 output name, -f true overwrite, -t cuda,nvtx trace CUDA calls and our labelled ranges. This writes /tmp/l1.nsys-rep.

Gotcha: re-profilingIf you profile again to the same -o name, nsys stats prints only a warning about an “older” .sqlite and no table. Add --force-export=true to the stats command (the commands below already do).

Ask for reports.

nsys stats --force-export=true -r cuda_gpu_kern_sum l1.nsys-rep   # time per kernel
nsys stats -r nvtx_sum         l1.nsys-rep      # time per labelled range
nsys stats -r cuda_gpu_trace   l1.nsys-rep | head -40   # every launch, in order
nsys stats -r cuda_gpu_mem_time_sum l1.nsys-rep # memcpy time

11. Reading the output

This is what we got on stg (your numbers should be within a few percent):

Time (%)  Total (ns)  Instances  Avg (ns)   Name
 78.4     62,548,294     23     2,719,491  cutlass_80_tensorop_f16_s16816gemm_relu_f16_256x128_32x3_nn_align8
 20.9     16,686,361     23       725,494  at::native::vectorized_elementwise_kernel<...Half...>
  0.7        553,216      3       184,405  distribution_elementwise_grid_stride_kernel (torch.randn)

Turn times into rates

Kernel	Work	Time	Rate
Elementwise (scale)	2^28 × 2 B = 512 MiB read + 512 MiB write ≈ 1.07 GB	725 µs	1.07 GB ÷ 725 µs ≈ 1.48 TB/s (~80% of the ~1.8 TB/s peak)
GEMM (matmul)	2 × 8192³ ≈ 1.1 TFLOP	2.72 ms	1.1 TFLOP ÷ 2.72 ms ≈ 400 TFLOPS

The lesson The scale kernel is already near the memory ceiling, so no cleverness will make it much faster: you can only move fewer bytes. The GEMM is limited by math throughput. Same GPU, two different bottlenecks. A profile tells you which one you’re in before you spend a week optimizing the wrong thing.

Note the instance count is 23 for both, not 20: that’s the 3 warm-up calls plus 20 timed ones. If you want only the timed region, filter the stats by NVTX range (nsys stats -r nvtx_gpu_proj_sum).

12. Exercise: the bandwidth sweep

Same elementwise op at different sizes, timed with CUDA events (no profiler needed). Run it and predict the shape before you look at the result.

import torch

for p in (16, 20, 22, 24, 26, 28, 30):
    n = 1 << p
    x = torch.randn(n, device="cuda", dtype=torch.float16)
    for _ in range(5): x.mul_(1.0001)
    s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
    s.record()
    for _ in range(50): x.mul_(1.0001)
    e.record(); torch.cuda.synchronize()
    t = s.elapsed_time(e) / 50 / 1e3        # seconds per call
    print(f"2^{p:<2} elems  {n*2/2**20:8.1f} MiB  {t*1e6:9.1f} us  {2*n*2/t/1e12:5.2f} TB/s")
    del x

Run it. $K exec nsight-lab -- python3 /tmp/sweep.py

Our result (don’t peek until you’ve predicted)

2^16 elems       0.1 MiB        5.9 us   0.04 TB/s
2^20 elems       2.0 MiB        5.8 us   0.73 TB/s
2^22 elems       8.0 MiB        5.7 us   2.96 TB/s
2^24 elems      32.0 MiB       12.3 us   5.45 TB/s
2^26 elems     128.0 MiB       88.3 us   3.04 TB/s
2^28 elems     512.0 MiB      733.9 us   1.46 TB/s
2^30 elems    2048.0 MiB     2939.7 us   1.46 TB/s

Three regimes:

Tiny (≤ ~8 MiB): launch-bound. Time sits at a ~5.8 µs floor no matter the size. That is the fixed cost of launching a kernel, not data movement. This is why LLM serving uses CUDA graphs and fused kernels: thousands of tiny launches add up.
Medium (~32 MiB): faster than DRAM allows. 5.45 TB/s is above the ~1.8 TB/s memory spec, so the data is being served from the on-chip L2 cache, not device memory. (I’m inferring this from the numbers; confirming it needs ncu cache counters.)
Large (≥ 512 MiB): DRAM-bound. It settles at ~1.46 TB/s, the real memory bandwidth ceiling for this card. Weights for an LLM are far bigger than the cache, so this is the regime that governs decode.

Stretch: change torch.float16 to torch.float32 and rerun. Predict the TB/s first. (Bytes per element double; if the op is bandwidth-bound, TB/s should stay the same and time should double.)

13. Nsight Compute (`ncu`): currently blocked

Nsight Compute zooms into one kernel and reports hardware counters: percent of peak DRAM bandwidth, percent of peak compute, cache hit rates, occupancy. It is the tool that would confirm the L2-cache explanation above.

# what you'd run
ncu -k regex:vectorized_elementwise -c 1 --section SpeedOfLight python3 /tmp/lesson1.py

On our stg nodes this fails ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters. The node’s GPU driver only allows counter access to admins. Two ways forward: ask the platform team for a profiling-enabled g7e node (preferred), or run the pod with CAP_SYS_ADMIN, which weakens isolation, so do it only with their OK.

14. Clean up

Delete the pod so the GPU goes back to the pool.

$K delete pod nsight-lab

It would auto-delete after 4 hours anyway, but stg GPUs are scarce, so release it when you’re done.

Where to go next

Profile a real decode. Run a small vLLM generation inside the lab pod under nsys and find the per-token kernel loop and the gaps between launches.
Roofline math for your model. Weight bytes ÷ 1.46 TB/s = the floor on one decode step. Compare it with the measured per-step time.
Write a Triton kernel (the Python-embedded GPU language already in the image) for the same scale op and see if you can match PyTorch’s 1.46 TB/s.
Understand the “heterogeneous packing” proposal. You now know why GPUs are integer, why a 96 GB card sits mostly idle for a 1B INT4 model, and where the tokenizer CPU limit comes in.

Glossary

Cluster / Node / Pod / Container	See section 1.
Context	A saved (cluster, user, namespace) in your kubeconfig.
Sidecar	A container that runs alongside the main one in the same pod.
Device plugin	Node agent that advertises GPUs to Kubernetes.
MIG	Multi-Instance GPU: hardware partitioning of one GPU into isolated slices.
MPS	Multi-Process Service: lets several processes share one GPU concurrently.
Triton	NVIDIA’s inference server; here it hosts and batches the models.
vLLM	LLM inference engine with paged KV-cache; preallocates most GPU memory.
KV cache	Stored attention state for generated tokens; why vLLM fills the card.
Kernel	A function running on the GPU.
SM	Streaming multiprocessor; the GPU’s basic compute block.
L2 cache	On-chip cache shared by all SMs; far faster than device memory.
Roofline	A model: performance is capped by min(compute peak, bandwidth × arithmetic intensity).
NVTX	Labels you add to code so profilers show named regions.
nsys / ncu	Nsight Systems (timeline) / Nsight Compute (per-kernel counters).

Command cheat-sheet

K="kubectl --context  -n "

$K get pods -o wide                       # what's running, and where
$K describe pod NAME                      # spec + Events (why Pending?)
$K logs NAME -c CONTAINER                 # container output
$K exec -it NAME -- bash                  # shell inside a pod
$K cp LOCAL NAME:/tmp/FILE                # copy file in (reverse to copy out)
$K delete pod NAME                        # remove a pod

# inside the lab pod
nvidia-smi
nsys profile -o OUT -f true -t cuda,nvtx python3 SCRIPT.py
nsys stats -r cuda_gpu_kern_sum OUT.nsys-rep