AI-Infra-Auto-Driven-SKILLS
Agent-ready playbooks for LLM serving benchmarks, SGLang model Day-0 support, capacity planning, torch-profiler triage, pipeline analysis, compute simulation, SGLang/vLLM optimization, human code review, production incidents, and model PR intelligence.
This repository is built for AI infrastructure engineers who want agents to do real work, not recite generic prompts.
It gives an agent the operational memory needed to benchmark SGLang, vLLM, TensorRT-LLM, and TokenSpeed fairly; turn a new SGLang model architecture into an auditable Day-0 support and release plan; explain serving capacity from startup logs; split prefill and decode profiler evidence; inspect traces at layer and kernel level; estimate operator FLOPs and MFU; review SGLang patches against real maintainer discussion patterns; run Humanize-governed SGLang and vLLM SOTA loops; triage SGLang production incidents from a replay; and keep model-family optimization history close to the code that actually changed.
For standalone kernel campaigns and kernel evidence tools, see the sibling project KDA-Pilot.
For model-by-model image, video, audio, and 3D optimization runs, see the Markdown-only SGLang Diffusion optimization flow library. Every model and component flow is self-contained and intentionally repeats its environment, download, benchmark, profiling, optimization, and acceptance instructions.
If this saves you one stale model-support assumption, one misleading profiler trace, or one late-night benchmark loop, a star helps more AI-infra engineers find it.
Core Skills
| Skill | Use it when |
|---|---|
llm-serving-auto-benchmark | You need a fair, bounded serving benchmark search for SGLang, vLLM, TensorRT-LLM, TokenSpeed, or another OpenAI-compatible stack. |
llm-serving-capacity-planner | You need to explain SGLang or vLLM startup memory, KV cache budget, request capacity, or OOM pressure from logs. |
llm-torch-profiler-analysis | You need a three-table profiler report that keeps extend/prefill and decode evidence separate. |
llm-pipeline-analysis | You need forward-pass, layer, and kernel-level timing from a torch profiler trace, including anchor boundaries and Perfetto ranges. |
model-compute-simulation | You need operator shapes, FLOPs, MFU estimates, kernel-to-op mapping, or parallelism what-if analysis for an LLM serving shape. |
sglang-model-day0-support | You need to turn a new SGLang model architecture into a public Day-0 PR DAG, parallel/kernel adaptation plan, seven-gate validation matrix, release lock, and sanitized evidence bundle. |
sglang-humanize-review | You need SGLang code-review findings grounded in full human PR review episodes from project start through the latest refresh (June 2026), including inline code context, top-level discussion, review summaries, and multi-round replies. Every review opens with a PR comprehension pass — a change summary plus a Mermaid execution flowchart with the diff's modified steps marked — so the reviewer sees how the PR runs before the findings. |
sglang-sota-humanize-loop | You want one model-level Humanize RLCR loop that owns SGLang gap decisions against a selected comparison framework set, profiler triage, required layer-pipeline deep dives, SGLang patches, optional ncu-report-skill evidence, and real-model revalidation after the fixed fair benchmark. |
vllm-sota-humanize-loop | You want one model-level Humanize RLCR loop that owns gap decisions, profiler triage, required layer-pipeline deep dives, vLLM patches, optional ncu-report-skill evidence, and real-model revalidation after the fixed fair benchmark. |
sglang-prod-incident-triage | You need to turn queue growth, timeouts, wrong outputs, crashes, or distributed stalls into a replay and next debug step. |
model-architecture-diagram | You need original public architecture diagrams for popular LLM, VLM, MoE, OCR, and diffusion model families. |
SGLang SOTA Performance Loop
sglang-sota-humanize-loop always patches SGLang, while the competitor set is
caller-controlled. By default the comparison framework set can include vLLM,
TensorRT-LLM, and TokenSpeed; a prompt can also narrow it, for example:
Use sglang-sota-humanize-loop for <model>.
comparison_frameworks: [vllm]
Do not consider TensorRT-LLM or TokenSpeed; record them as user-excluded.
Model PR History Catalog
The model optimization layer is now one knowledge base:
model-pr-optimization-history. It contains
66 PR-driven history dossiers and a small query helper. These are not
per-model runbook skills; they preserve diff-backed model evolution records for
SGLang, vLLM, TensorRT-LLM, and TokenSpeed so SOTA loops can read prior source
and PR evidence before patching.
| Framework | PR histories |
|---|---|
| SGLang | 31 |
| vLLM | 31 |
| TensorRT-LLM | 2 |
| TokenSpeed | 2 |
Covered families include:
DeepSeek V3/R1/V3.1/V3.2/V4, Qwen3, Qwen3-Coder, Qwen3-Next,
Qwen3.5/Qwen3.6, Qwen VLM/Omni/ASR, GLM 4.5/4.6/4.7/5,
Kimi, MiniMax, Llama 4, Mistral Small 4, Mixtral, Nemotron,
Gemma, Ernie 4.5, Intern-S1, InternVL, Hunyuan, MOSS-VL,
GPT-OSS, Step 3.5, Mimo, and model-specific MoE/quantization paths.
Each model-family history is designed to answer practical questions:
- Which PRs changed this model path?
- Was the PR merged, closed, or still open?
- Which files and symbols moved?
- What optimization or correctness risk should be checked before touching it?
- Which upstream idea should be compared before writing a new kernel or fusion?
Query examples:
cd model-pr-optimization-history
python3 scripts/query.py --list
python3 scripts/query.py --framework sglang --model qwen3-core --paths-only
python3 scripts/query.py --framework vllm "qwen3 fused qk norm"
python3 scripts/query.py --framework tokenspeed --model qwen35 qk rmsnorm
Open PR freshness is tracked separately from merged history cards:
python3 tools/check_open_pr_watch.py --format markdown \
--output model-pr-optimization-history/open-pr-watch.md
This report uses the GitHub pulls API with an anonymous REST fallback when
gh api is rate-limited. If every repo fetch fails, the tool exits non-zero
instead of writing a misleading empty report.
Evidence Standards
The repo is opinionated about evidence because performance work gets noisy fast.
- Benchmark rows should include model, framework, GPU count, workload, request rate or concurrency, SLA status, launch command, benchmark command, and raw artifacts.
- Profiler reports should keep prefill and decode separate, then emit the same three tables: kernel table, overlap-opportunity table, and fuse-opportunity table.
- SOTA claims should be scoped to the exact model, hardware, framework commits, precision, workload, and SLA used in the run.
- SGLang human review should use the full PR episode corpus: inline review threads for line-local findings, PR conversations for design/test/repro negotiation, and review submissions for blocking maintainer summaries.
- Humanize SOTA loops should keep only the fixed fair benchmark outside the patch loop; gap decisions, profiler triage, required layer-pipeline deep dives, kernel evidence, target-framework code changes, and revalidation all stay inside one model-level RLCR loop.
- Kernel-local fixes inside that loop should use
ncu-report-skillwhen Nsight Compute counter evidence is needed, store NCU digests, and still pass the same real-model benchmark/profile gate. - Incident triage should start from replayable evidence instead of changing code from symptoms alone.
- Model optimization histories should point back to PRs, files, diffs, and risk surfaces rather than vague summary text; they live as one PR-driven knowledge base, not per-model skills.
- Root-level
update_prompt.mdcaptures the full refresh and validation workflow for updating this repo again without relying on memory from a previous run.
Install
This repository is not Codex-only. The skills are plain SKILL.md directories
and can be installed into Claude Code, Codex, Kimi, or another compatible agent
runtime.
Claude Code (one-shot plugin install)
The repository ships a .claude-plugin/ manifest so the whole skill set can be
installed as a single Claude Code plugin via the built-in marketplace flow:
/plugin marketplace add BBuf/AI-Infra-Auto-Driven-SKILLS
/plugin install ai-infra-auto-driven-skills@ai-infra-auto-driven-skills
/reload-plugins
After reload, the 11 skills appear namespaced as
ai-infra-auto-driven-skills:<skill-name> (for example
ai-infra-auto-driven-skills:sglang-sota-humanize-loop). Update later with
/plugin marketplace update ai-infra-auto-driven-skills.
Claude Code (per-skill symlink, legacy)
Prefer this when you only want a subset of the skills, or when developing against a local checkout. Symlink is recommended for local development because updates to this checkout are picked up immediately:
git clone https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS.git
cd AI-Infra-Auto-Driven-SKILLS
mkdir -p ~/.claude/skills
ln -s "$PWD/skills/llm-serving-auto-benchmark" ~/.claude/skills/llm-serving-auto-benchmark
ln -s "$PWD/skills/llm-serving-capacity-planner" ~/.claude/skills/llm-serving-capacity-planner
ln -s "$PWD/skills/llm-torch-profiler-analysis" ~/.claude/skills/llm-torch-profiler-analysis
ln -s "$PWD/skills/llm-pipeline-analysis" ~/.claude/skills/llm-pipeline-analysis
ln -s "$PWD/skills/model-compute-simulation" ~/.claude/skills/model-compute-simulation
ln -s "$PWD/skills/model-optimization/sglang-model-day0-support" ~/.claude/skills/sglang-model-day0-support
ln -s "$PWD/skills/sglang-humanize-review" ~/.claude/skills/sglang-humanize-review
ln -s "$PWD/skills/sglang-sota-humanize-loop" ~/.claude/skills/sglang-sota-humanize-loop
ln -s "$PWD/skills/vllm-sota-humanize-loop" ~/.claude/skills/vllm-sota-humanize-loop
ln -s "$PWD/skills/sglang-prod-incident-triage" ~/.claude/skills/sglang-prod-incident-triage
ln -s "$PWD/skills/model-architecture-diagram" ~/.claude/skills/model-architecture-diagram
ln -s "$PWD/model-pr-optimization-history" ~/.claude/skills/model-pr-history-knowledge
Restart Claude Code after installing. The skills can then be invoked by name,
for example [$llm-serving-auto-benchmark],
[$llm-serving-capacity-planner], [$llm-torch-profiler-analysis],
[$llm-pipeline-analysis], [$model-compute-simulation],
[$sglang-model-day0-support],
[$sglang-humanize-review],
[$sglang-sota-humanize-loop], or [$vllm-sota-humanize-loop].
If you prefer copies instead of symlinks, replace ln -s with cp -R. Copy
model-pr-optimization-history only when you want the agent to query the
PR-driven model knowledge base locally. It replaces the old per-model runbook
skill layout with one shared knowledge root.
Generic Agent Skill Directory
For Codex, Kimi, or another compatible runtime, copy or symlink the same directories into that runtime's skill directory:
cp -R skills/llm-serving-auto-benchmark <agent-skill-dir>/llm-serving-auto-benchmark
cp -R skills/llm-serving-capacity-planner <agent-skill-dir>/llm-serving-capacity-planner
cp -R skills/llm-torch-profiler-analysis <agent-skill-dir>/llm-torch-profiler-analysis
cp -R skills/llm-pipeline-analysis <agent-skill-dir>/llm-pipeline-analysis
cp -R skills/model-compute-simulation <agent-skill-dir>/model-compute-simulation
cp -R skills/model-optimization/sglang-model-day0-support <agent-skill-dir>/sglang-model-day0-support
cp -R skills/sglang-humanize-review <agent-skill-dir>/sglang-humanize-review
cp -R skills/sglang-sota-humanize-loop <agent-skill-dir>/sglang-sota-humanize-loop
cp -R skills/vllm-sota-humanize-loop <agent-skill-dir>/vllm-sota-humanize-loop
cp -R skills/sglang-prod-incident-triage <agent-skill-dir>/sglang-prod-incident-triage
cp -R skills/model-architecture-diagram <agent-skill-dir>/model-architecture-diagram
cp -R model-pr-optimization-history <agent-skill-dir>/model-pr-history-knowledge
How I Drive The Agents
These skills are exercised with coding agents in full-autonomy mode. For reproducibility, here is exactly how I launch them.
Claude Code — use the current Opus alias with Auto permission mode:
claude --model opus --permission-mode auto
opus tracks Claude Code's current Opus model instead of pinning a model number.
Auto mode can approve routine work while retaining the permission system.
bypassPermissions (or --dangerously-skip-permissions) should be reserved for
an isolated sandbox, ideally without internet access.
Codex — full-access, no approval prompts:
codex --sandbox danger-full-access --ask-for-approval never
The Codex command is intentionally unsandboxed and should likewise be used only inside an isolated environment with a scoped checkout and its own benchmark and correctness gates.
Repository Map
skills/
├── llm-serving-auto-benchmark/ # serving benchmark search and comparison
├── llm-serving-capacity-planner/ # startup memory and request capacity analysis
├── llm-torch-profiler-analysis/ # profiler capture and trace triage
├── llm-pipeline-analysis/ # forward/layer/kernel trace analysis
├── model-compute-simulation/ # operator FLOPs, tensor shapes, and MFU
├── sglang-humanize-review/ # human SGLang PR review corpus and workflow
├── sglang-sota-humanize-loop/ # Humanize-governed SGLang SOTA loop
├── vllm-sota-humanize-loop/ # Humanize-governed vLLM SOTA loop
├── sglang-prod-incident-triage/ # replay-first serving incident workflow
├── model-architecture-diagram/ # public architecture diagram resolver
└── model-optimization/
└── sglang-model-day0-support/ # model Day-0 PR and release gates
model-pr-optimization-history/
├── SKILL.md # knowledge-base usage instructions
├── scripts/query.py # local model/keyword query helper
├── sglang/ # 31 PR-driven SGLang model histories
├── vllm/ # 31 PR-driven vLLM model histories
├── tensorrt_llm/ # TensorRT-LLM competitor histories
└── tokenspeed/ # TokenSpeed competitor histories
prompts/
├── sglang-sota-b200-prompts.md # B200 SGLang SOTA task prompts
├── sglang-sota-b200-codex-goal-prompts.md
├── sglang-sota-h200-prompts.md # H200 SGLang SOTA task prompts
└── sglang-sota-h200-codex-goal-prompts.md
Related Projects
- Humanize provides the RLCR workflow that powers the Humanize-governed SGLang and vLLM SOTA loops.
- KDA-Pilot is the sibling home for standalone kernel loops, kernel knowledge, and NCU report workflows.