REA
Reverse engineer anything from your terminal or agent with one CLI and MCP server.
The Art of Debugging Open Book
作者提供的原文
如果你是作者,可把这段代码贴到 README。徽章会链接回本页。
[](https://mcp.popnx.com/en/skills/stas00-the-art-of-debugging)在同一工作中把 AI 连到服务的 MCP
Reverse engineer anything from your terminal or agent with one CLI and MCP server.
Codebase intelligence for TypeScript and JavaScript, exposed as structured MCP analysis tools.
MCP server for terminal commands, file operations, and process management
同类工作,排序与 Skill 列表相同
| 本项 the-art-of-debugging | rea | andrej-karpathy-skills | graphify | MoneyPrinterTurbo | |
|---|---|---|---|---|---|
| 作者 | stas00 | morluto | multica-ai | Graphify-Labs | harry0703 |
| Skill 数 | 1 | 1 | 1 | 1 | 1 |
| GitHub 星标 | 1,749 | 468,733 | 234,332 | 150,287 | 144,566 |
| 最近代码更新 | 2026年10月14日 | 2026年11月19日 | 2026年4月20日 | 2026年11月18日 | 2026年11月18日 |
| 许可证 | CC-BY-SA-4.0 | MIT | – | Apache-2.0 | MIT |
stas00 提供的 1 个 Skill(指令集),AI 读取后照做 · 不是 MCP server
MCP = 手
把 AI 连到外部服务或数据,例如读取 Notion 或开发票。需要有程序或服务器在运行。
Skill = 手册
教 AI 怎么做某项工作的文本文件,不连接任何服务。
有些会重叠:少数 Skill 带有调用工具的脚本,少数 MCP 只提供知识。请从你想完成的工作出发。
开发者未说明 · SKILL.md 形式的 Skill 可用于支持 Agent Skills 的应用,如 Claude Code、Codex 和 Cursor
GitHub 账号 stas00(个人),已创建 11 年
★ 1,749 星 · 代码最近更新 2026年10月14日
有 35 个可以在电脑上运行的脚本:build/check-new-links.sh, build/check-programs, pytorch/code/collective_mismatch.py, pytorch/code/collective_mismatch_analyze.py 另外还有 31 个文件
会联网的脚本:build/check-new-links.sh
没有发现 hook 或写入电脑设置的操作
根据仓库 commit 7e479bf 中的文件自动检查(只读取文件,没有运行任何内容)。这不是安全保证,请自己阅读后再决定
第一步:你用哪个 AI?
选择你用的应用,本页只显示该应用的安装方法
这个 Skill 使用 CC-BY-SA-4.0 许可证,不在我们可以转发的列表中,所以我们不在网站上提供 ZIP。请从 GitHub 下载并自己整理文件夹,或者用上面的"让 AI 帮你安装",更简单
解压后,只把 the-art-of-debugging 文件夹重新压缩。ZIP 的最上层必须是包含 SKILL.md 的 art-of-debugging 文件夹,不能直接放散乱的文件
+Create skillUpload a skill选择文件 art-of-debugging.zip
SettingsCapabilitiesCode execution and file creation
Turn on Code execution and file creation first (newer menus: Create and edit files) · personal plans: Settings → Capabilities · Team/Enterprise: an admin turns it on under Organization settings → Plugins & skills · every plan, including Free
Upload once; it works in claude.ai and Claude Desktop, and in recent Claude Code signed in with a Claude account (not with an API key)
Cowork and cloud sessions use the skills turned on in your claude.ai account, not ~/.claude/skills on your computer (Claude Desktop scheduled tasks run locally and still load it)
Checked against official docs 2026年11月12日 · Source: support.claude.com, support.claude.com, code.claude.com
复制这句话,粘贴到 Claude Code 的对话中
如果只用于一个项目,请粘贴到在该项目文件夹中打开的对话里 · AI 在安装前一定会先告诉你路径
AI 会先说明要做什么,每一步都会征求你的同意
重新打开 Claude Code,输入 /the-art-of-debugging,或直接提出相关任务,Claude 会自动选用
Checked against official docs 2026年11月12日 · Source: code.claude.com
Skill 是 AI 会照做的指令。安装前先阅读文件,只安装可信开发者的 Skill。
Mac:按 Command+空格输入 Terminal · Windows:下方命令在 PowerShell 中无法使用,请改用上方的“让 AI 帮你安装”
这条命令把 repo 下载到临时文件夹,只复制含 SKILL.md 的文件夹
git clone --depth 1 https://github.com/stas00/the-art-of-debugging.git /tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf && mkdir -p ~/.claude/skills && cp -R '/tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf/.' ~/.claude/skills/the-art-of-debuggingChatGPT 桌面应用内置 Codex,Codex 从 ~/.agents/skills 文件夹读取 Skill
在 ChatGPT 桌面应用中打开 Codex,把这句话粘贴到对话中
AI 在安装前一定会先告诉你路径
AI 会先说明要做什么,每一步都会征求你的同意
把 art-of-debugging 文件夹放到 ~/.agents/skills/art-of-debugging
~ 是你的主文件夹($HOME)
Skill 是 AI 会照做的指令。安装前先阅读文件,只安装可信开发者的 Skill。
Mac:按 Command+空格输入 Terminal · Windows:下方命令在 PowerShell 中无法使用,请改用上方的“让 AI 帮你安装”
这条命令把 repo 下载到临时文件夹,只复制含 SKILL.md 的文件夹
git clone --depth 1 https://github.com/stas00/the-art-of-debugging.git /tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf && mkdir -p ~/.agents/skills && cp -R '/tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf/.' ~/.agents/skills/the-art-of-debuggingChecked against official docs 2026年11月15日 · Source: learn.chatgpt.com
不能使用你自己安装的 Skill,只能使用目录中插件自带的 Skill
想在 ChatGPT 中使用这个 Skill,请按上面的方法使用 ChatGPT 桌面应用
Checked against official docs 2026年11月15日 · Source: learn.chatgpt.com
复制这句话,粘贴到 Codex 的对话中 · 适用于 ChatGPT 桌面应用中的 Codex、Codex CLI 和 IDE 扩展
如果只用于一个项目,请粘贴到在该项目文件夹中打开的对话里 · AI 在安装前一定会先告诉你路径
AI 会先说明要做什么,每一步都会征求你的同意
Codex 会自动发现新安装的 Skill;如果没有出现,请重启 Codex · 使用方法:在 ChatGPT 中输入 @ 并选择 Skill · 在 Codex CLI 或 IDE 扩展中运行 /skills,或输入 $ 加 Skill 名称
Checked against official docs 2026年11月15日 · Source: learn.chatgpt.com
Skill 是 AI 会照做的指令。安装前先阅读文件,只安装可信开发者的 Skill。
Mac:按 Command+空格输入 Terminal · Windows:下方命令在 PowerShell 中无法使用,请改用上方的“让 AI 帮你安装”
这条命令把 repo 下载到临时文件夹,只复制含 SKILL.md 的文件夹
git clone --depth 1 https://github.com/stas00/the-art-of-debugging.git /tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf && mkdir -p ~/.agents/skills && cp -R '/tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf/.' ~/.agents/skills/the-art-of-debugging复制这句话,粘贴到 Cursor 的对话中
如果只用于一个项目,请粘贴到在该项目文件夹中打开的对话里 · AI 在安装前一定会先告诉你路径
AI 会先说明要做什么,每一步都会征求你的同意
在 Cursor 的 Agent 聊天中输入 / 并选择 the-art-of-debugging
Checked against official docs 2026年11月12日 · Source: cursor.com
Skill 是 AI 会照做的指令。安装前先阅读文件,只安装可信开发者的 Skill。
Mac:按 Command+空格输入 Terminal · Windows:下方命令在 PowerShell 中无法使用,请改用上方的“让 AI 帮你安装”
这条命令把 repo 下载到临时文件夹,只复制含 SKILL.md 的文件夹
git clone --depth 1 https://github.com/stas00/the-art-of-debugging.git /tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf && mkdir -p ~/.cursor/skills && cp -R '/tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf/.' ~/.cursor/skills/the-art-of-debugging适用于其他能读取 Skill 的应用。这句话列出了 Claude Code、Codex 和 Cursor 的路径,AI 会按你的应用选择 · 复制这句话,粘贴到应用的对话中
如果只用于一个项目,请粘贴到在该项目文件夹中打开的对话里 · AI 在安装前一定会先告诉你路径
AI 会先说明要做什么,每一步都会征求你的同意
Checked against official docs 2026年11月12日 · Source: code.claude.com, learn.chatgpt.com, cursor.com
其他支持 Agent Skills 的应用(如 Gemini CLI、GitHub Copilot)使用不同文件夹,请查看该应用的指南,再把含 SKILL.md 的文件夹复制过去。
已于 2026年11月11日 对照 Claude Code、Codex 和 Cursor 官方指南核实 skill 文件夹
✓ 命令已检查 2026年11月18日
以下文字来自开发者的 repo,不是 PopNx MCP 的指令
显示前 8,000 个字符 · 在 GitHub 打开完整文件
---
name: art-of-debugging
description: >-
Systematic methodology and concrete tool recipes for debugging Unix, Python,
and PyTorch programs - crashes, hangs, segfaults, wrong output, CUDA OOM, NaN/Inf,
slowness, and multi-node/multi-GPU issues. Use when a program crashes, hangs,
deadlocks, segfaults, runs out of memory (OOM), produces NaN/Inf or wrong
numbers, runs too slowly, or when the user mentions gdb, strace, py-spy,
core files, CUDA_LAUNCH_BLOCKING, ldd/nm/LD_PRELOAD, cProfile, or distributed
training hangs. Distilled from "The Art of Debugging", the latest version of which can be found at https://github.com/stas00/the-art-of-debugging
The latest SKILL.md version can be found at https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md
---
# The Art of Debugging
> Distilled from **The Art of Debugging Open Book** by Stas Bekman - source: https://github.com/stas00/the-art-of-debugging (CC BY-SA 4.0). This skill is a condensed index; each section links back to the full chapter for depth.
Actionable methodology + copy-paste recipes for debugging Unix / Python / PyTorch programs. Apply the general loop first; then jump to the domain cheatsheet for the failure at hand. For scaling this up to large-model training/inference on real clusters (compute/storage/network, SLURM, throughput/memory, instabilities, fault tolerance, inference), pair this with [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md).
## The debugging loop
The single most important idea: **most of the effort is in *locating* the cause; once you truly understand it, the fix is usually easy.** Optimize everything for reaching understanding faster.
1. **Reproduce reliably.** Get one command that triggers the bug every time. If it's flaky, pin the nondeterminism (seeds, ordering, timing, network, uninitialized memory) first - you can't debug what you can't repeat.
2. **Shrink the payload.** Make the repro *fast*: fewer layers, tiny model/data, one process, one CPU/GPU, one node. A 2-second repro beats a 2-minute one - you'll run it hundreds of times. See [methodology](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md).
3. **Localize.** Confirm you're editing the code that actually runs (the `die` trick), then bisect the search space: which commit, which file/function, which line, which input, which rank.
4. **Get a usable signal.** Turn a cryptic failure into a precise one: a real traceback (sync mode), a stack dump (py-spy/gdb), a syscall trace (strace), a printed value at the boundary, or a min/max/NaN check on a tensor.
5. **Change one thing, re-run, verify.** Fix on the fast repro, confirm, then re-widen to the full payload. Revert anything that didn't help.
### Make the loop fast and reliable
- **Atomic debug cycles.** Make each iteration a single self-contained, repeatable command (setup + run in one shot) so you never hand-redo multi-step state. See [atomic debug cycles](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#atomic-debug-cycles).
- **Automate diagnostics, minimize typing.** Alias the repro and your most-used commands; one keystroke to re-run. See [alias frequently used commands](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#alias-frequently-used-commands) and [automate diagnostics](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#automate-diagnostics-minimize-or-avoid-typing).
- **One-liner programs.** `perl`/`awk`/`python -c` to slice logs, extract fields, transform data on the fly instead of writing throwaway scripts. See [the power of one-liner programs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#the-power-of-one-liner-programs).
- **Juggle configs cleanly** when running many debug experiments so results don't get confused. See [juggling multiple sets of configs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#juggling-multiple-sets-of-configs-for-different-debug-experiments).
### Localization techniques
- **Am I editing the right file/class?** Insert a guaranteed break where you *think* execution goes; if the program doesn't die, you're in the wrong file/class/env:
```python
def suspect():
die # NameError -> proves this code runs; the traceback also names the caller
```
`traceback.print_stack()` shows callers without stopping (useful when the same function is reached via many paths). See [am I editing the right file and the right class?](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#am-i-editing-the-right-file-and-the-right-class).
- **Bisect a regression.** `git bisect start / bad / good <rev>` walks commits automatically to the one that broke things - script the test for `git bisect run`. See [finding a breaking commit by bisecting](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#finding-a-breaking-commit-by-bisecting-revisions).
- **Small/synthetic payload first.** Use tiny or synthetic inputs; switch to real data only when the bug is data-dependent. See [real vs random vs synthetic data](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#real-data-vs-random-data-vs-synthetic-data).
- **Race conditions.** Reordering/timing bugs hide under async; forcing synchronous execution can expose (or mask) them - note which. See [avoiding race conditions](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#avoiding-race-conditions) and [async vs sync mode](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#async-vs-sync-mode).
### Reproducing resource & environment issues
- **Cap resources on purpose** to test failure paths: emulate a nearly-full disk, limited CPU RAM, or limited GPU memory. See [running out of resources](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#running-out-of-resources-disk-space-cpu-memory-gpu-memory).
- **Watch resources live.** `watch -n1 nvidia-smi` / `free -h` / `df -h` in a second visible terminal to correlate a hang/OOM with what the machine is doing. See [watching and reproducing resource issues](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#watching-and-reproducing-resource-issues).
- **Inject `sleep`** to freeze a program at the interesting moment so you can attach a debugger or snapshot state. See [uses for sleep](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#uses-for-sleep).
- **HPC/SLURM:** keep the allocation and re-run with `srun` instead of re-`sbatch`-ing to cut per-iteration overhead. See [SLURM salloc and srun fast debug combo](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#slurm-salloc-and-srun-fast-debug-combo).
## Unix / shell
Full chapter: [Unix Tools for Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md).
- **Make shell scripts fail loudly and traceably:**
```bash
set -e # abort on first error
set -o pipefail # a failing command anywhere in a pipe fails the whole pipe
set -u # abort on undefined variables (catches typos)
set -x # trace: print each command with expanded values as it runs
set +x # turn tracing back off around a noisy region
```
Combine as `set -euo pipefail`. See [controlling script execution](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#controlling-script-execution).
- **`strace` - trace system calls** to see what a program actually does (files, network, why it's stuck):
```bash
strace python -c "print('hi')" # trace from the start
strace --pid PID # attach to a running/stuck process
strace -o log.txt -f torchrun ... # -f follows fork提问、分享技巧或报告无法使用。按“有帮助”数量排序。