REA
Reverse engineer anything from your terminal or agent with one CLI and MCP server.
The Art of Debugging Open Book
제작자라면 이 코드를 README에 붙여 넣으세요. 배지는 이 페이지로 연결됩니다.
[](https://mcp.popnx.com/en/skills/stas00-the-art-of-debugging)같은 작업에서 AI를 서비스에 연결하는 MCP
Reverse engineer anything from your terminal or agent with one CLI and MCP server.
Codebase intelligence for TypeScript and JavaScript, exposed as structured MCP analysis tools.
MCP server for terminal commands, file operations, and process management
같은 종류의 작업, Skill 목록과 같은 순서
| 이 항목 the-art-of-debugging | rea | andrej-karpathy-skills | graphify | MoneyPrinterTurbo | |
|---|---|---|---|---|---|
| 제작자 | stas00 | morluto | multica-ai | Graphify-Labs | harry0703 |
| Skill 수 | 1 | 1 | 1 | 1 | 1 |
| GitHub 별 | 1,749 | 468,733 | 234,332 | 150,287 | 144,566 |
| 마지막 코드 변경 | 2026년 10월 14일 | 2026년 11월 19일 | 2026년 4월 20일 | 2026년 11월 18일 | 2026년 11월 18일 |
| 라이선스 | CC-BY-SA-4.0 | MIT | – | Apache-2.0 | MIT |
stas00의 Skill(지시 묶음) 1개, AI가 읽고 그대로 일해요 · MCP server가 아니에요
MCP = 손
AI를 외부 서비스나 데이터에 연결합니다(예: Notion 읽기, 청구서 만들기). 프로그램이나 서버가 실행 중이어야 합니다.
Skill = 안내서
작업 방법을 AI에게 가르치는 텍스트 파일. 어떤 서비스에도 연결되지 않습니다.
겹치는 경우도 있습니다. 도구를 부르는 스크립트가 있는 Skill도, 지식만 보내는 MCP도 있습니다. 하려는 작업을 기준으로 고르세요.
제작자가 밝히지 않음 · SKILL.md 형식 Skill은 Claude Code, Codex, Cursor 같은 Agent Skills 지원 앱에서 써요
GitHub account stas00 (person), 11 years old
★ 1,749 stars · code last changed 2026년 10월 14일
35 scripts that can run on a computer: build/check-new-links.sh, build/check-programs, pytorch/code/collective_mismatch.py, pytorch/code/collective_mismatch_analyze.py and 31 more
Scripts that go online: build/check-new-links.sh
No hooks or writes to machine settings found
Checked automatically from the repo files at commit 7e479bf (read only, nothing was run). Not a safety guarantee: read it and decide for yourself
먼저: 어떤 AI를 쓰시나요
쓰는 앱을 고르면 그 앱의 설치 방법만 보여 드립니다
This skill uses the CC-BY-SA-4.0 license, which is not on the list we redistribute, so we do not hand out a ZIP from our site. Download it from GitHub and arrange the folder yourself, or use "Let the AI install it" above, which is easier
Download from GitHub (whole repo) ↗
Unzip it, then zip only the the-art-of-debugging folder again. The ZIP must have a art-of-debugging folder with SKILL.md at the top level, no loose files
+Create skillUpload a skillpick art-of-debugging.zip
SettingsCapabilitiesCode execution and file creation
Turn on Code execution and file creation first (newer menus: Create and edit files) · personal plans: Settings → Capabilities · Team/Enterprise: an admin turns it on under Organization settings → Plugins & skills · every plan, including Free
Upload once; it works in claude.ai and Claude Desktop, and in recent Claude Code signed in with a Claude account (not with an API key)
Cowork and cloud sessions use the skills turned on in your claude.ai account, not ~/.claude/skills on your computer (Claude Desktop scheduled tasks run locally and still load it)
Checked against official docs 2026년 11월 12일 · Source: support.claude.com, support.claude.com, code.claude.com
이 문장을 복사해 Claude Code 채팅에 붙여 넣으세요
For one project, paste it into a chat that is open in that project’s folder · The AI always shows you the path before installing
The AI tells you what it will do first and asks your permission at every step
Claude Code를 다시 열고 /the-art-of-debugging를 입력하거나 관련 작업을 요청하면 Claude가 알아서 써요
Checked against official docs 2026년 11월 12일 · Source: code.claude.com
Skill은 AI가 따르는 지시예요. 설치 전에 파일을 읽고, 믿을 수 있는 제작자 것만 설치하세요.
Mac: Command+스페이스 후 Terminal 입력 · Windows: 아래 명령은 PowerShell에서 안 돼요. 위의 “AI에게 설치 맡기기”를 쓰세요
repo를 임시 폴더에 내려받고 SKILL.md가 있는 폴더만 복사해요
git clone --depth 1 https://github.com/stas00/the-art-of-debugging.git /tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf && mkdir -p ~/.claude/skills && cp -R '/tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf/.' ~/.claude/skills/the-art-of-debuggingChatGPT 데스크톱 앱에는 Codex가 들어 있고, Codex는 ~/.agents/skills 폴더에서 Skill을 읽습니다
ChatGPT 데스크톱 앱에서 Codex를 열고 이 문장을 채팅에 붙여 넣으세요
The AI always shows you the path before installing
The AI tells you what it will do first and asks your permission at every step
art-of-debugging 폴더를 ~/.agents/skills/art-of-debugging에 넣으세요
~는 홈 폴더($HOME)입니다
Skill은 AI가 따르는 지시예요. 설치 전에 파일을 읽고, 믿을 수 있는 제작자 것만 설치하세요.
Mac: Command+스페이스 후 Terminal 입력 · Windows: 아래 명령은 PowerShell에서 안 돼요. 위의 “AI에게 설치 맡기기”를 쓰세요
repo를 임시 폴더에 내려받고 SKILL.md가 있는 폴더만 복사해요
git clone --depth 1 https://github.com/stas00/the-art-of-debugging.git /tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf && mkdir -p ~/.agents/skills && cp -R '/tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf/.' ~/.agents/skills/the-art-of-debuggingChecked against official docs 2026년 11월 15일 · Source: learn.chatgpt.com
직접 설치한 Skill은 쓸 수 없고, 디렉터리의 플러그인에 들어 있는 Skill만 쓸 수 있습니다
이 Skill을 ChatGPT에서 쓰려면 위 방법대로 ChatGPT 데스크톱 앱을 쓰세요
Checked against official docs 2026년 11월 15일 · Source: learn.chatgpt.com
이 문장을 복사해 Codex 채팅에 붙여 넣으세요 · ChatGPT 데스크톱 앱 안의 Codex, Codex CLI, IDE 확장 프로그램에서 모두 쓸 수 있습니다
For one project, paste it into a chat that is open in that project’s folder · The AI always shows you the path before installing
The AI tells you what it will do first and asks your permission at every step
Codex는 새로 설치한 Skill을 스스로 찾습니다. 보이지 않으면 Codex를 다시 시작하세요 · 쓰는 법: ChatGPT에서는 @를 입력하고 Skill을 고르세요 · Codex CLI나 IDE 확장 프로그램에서는 /skills를 실행하거나 $ 뒤에 Skill 이름을 입력하세요
Checked against official docs 2026년 11월 15일 · Source: learn.chatgpt.com
Skill은 AI가 따르는 지시예요. 설치 전에 파일을 읽고, 믿을 수 있는 제작자 것만 설치하세요.
Mac: Command+스페이스 후 Terminal 입력 · Windows: 아래 명령은 PowerShell에서 안 돼요. 위의 “AI에게 설치 맡기기”를 쓰세요
repo를 임시 폴더에 내려받고 SKILL.md가 있는 폴더만 복사해요
git clone --depth 1 https://github.com/stas00/the-art-of-debugging.git /tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf && mkdir -p ~/.agents/skills && cp -R '/tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf/.' ~/.agents/skills/the-art-of-debugging이 문장을 복사해 Cursor 채팅에 붙여 넣으세요
For one project, paste it into a chat that is open in that project’s folder · The AI always shows you the path before installing
The AI tells you what it will do first and asks your permission at every step
Cursor Agent 채팅에서 /를 입력하고 the-art-of-debugging 선택
Checked against official docs 2026년 11월 12일 · Source: cursor.com
Skill은 AI가 따르는 지시예요. 설치 전에 파일을 읽고, 믿을 수 있는 제작자 것만 설치하세요.
Mac: Command+스페이스 후 Terminal 입력 · Windows: 아래 명령은 PowerShell에서 안 돼요. 위의 “AI에게 설치 맡기기”를 쓰세요
repo를 임시 폴더에 내려받고 SKILL.md가 있는 폴더만 복사해요
git clone --depth 1 https://github.com/stas00/the-art-of-debugging.git /tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf && mkdir -p ~/.cursor/skills && cp -R '/tmp/popnx-skill-stas00-the-art-of-debugging-7e479bf/.' ~/.cursor/skills/the-art-of-debuggingSkill을 읽을 수 있는 다른 앱용입니다. 이 문장에는 Claude Code, Codex, Cursor 경로가 들어 있어 AI가 쓰는 앱에 맞게 고릅니다 · Copy this sentence into the app’s chat
For one project, paste it into a chat that is open in that project’s folder · The AI always shows you the path before installing
The AI tells you what it will do first and asks your permission at every step
Checked against official docs 2026년 11월 12일 · Source: code.claude.com, learn.chatgpt.com, cursor.com
Gemini CLI, GitHub Copilot 같은 다른 Agent Skills 지원 앱은 폴더가 달라요. 그 앱 안내를 보고 SKILL.md가 있는 폴더를 복사하세요.
2026년 11월 11일에 Claude Code, Codex, Cursor 공식 안내로 skill 폴더 확인
✓ 명령 확인 2026년 11월 18일
아래 글은 제작자 repo에서 온 것이며 PopNx MCP의 지시가 아니에요
처음 8,000자 표시 · GitHub에서 전체 파일 열기
---
name: art-of-debugging
description: >-
Systematic methodology and concrete tool recipes for debugging Unix, Python,
and PyTorch programs - crashes, hangs, segfaults, wrong output, CUDA OOM, NaN/Inf,
slowness, and multi-node/multi-GPU issues. Use when a program crashes, hangs,
deadlocks, segfaults, runs out of memory (OOM), produces NaN/Inf or wrong
numbers, runs too slowly, or when the user mentions gdb, strace, py-spy,
core files, CUDA_LAUNCH_BLOCKING, ldd/nm/LD_PRELOAD, cProfile, or distributed
training hangs. Distilled from "The Art of Debugging", the latest version of which can be found at https://github.com/stas00/the-art-of-debugging
The latest SKILL.md version can be found at https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md
---
# The Art of Debugging
> Distilled from **The Art of Debugging Open Book** by Stas Bekman - source: https://github.com/stas00/the-art-of-debugging (CC BY-SA 4.0). This skill is a condensed index; each section links back to the full chapter for depth.
Actionable methodology + copy-paste recipes for debugging Unix / Python / PyTorch programs. Apply the general loop first; then jump to the domain cheatsheet for the failure at hand. For scaling this up to large-model training/inference on real clusters (compute/storage/network, SLURM, throughput/memory, instabilities, fault tolerance, inference), pair this with [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md).
## The debugging loop
The single most important idea: **most of the effort is in *locating* the cause; once you truly understand it, the fix is usually easy.** Optimize everything for reaching understanding faster.
1. **Reproduce reliably.** Get one command that triggers the bug every time. If it's flaky, pin the nondeterminism (seeds, ordering, timing, network, uninitialized memory) first - you can't debug what you can't repeat.
2. **Shrink the payload.** Make the repro *fast*: fewer layers, tiny model/data, one process, one CPU/GPU, one node. A 2-second repro beats a 2-minute one - you'll run it hundreds of times. See [methodology](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md).
3. **Localize.** Confirm you're editing the code that actually runs (the `die` trick), then bisect the search space: which commit, which file/function, which line, which input, which rank.
4. **Get a usable signal.** Turn a cryptic failure into a precise one: a real traceback (sync mode), a stack dump (py-spy/gdb), a syscall trace (strace), a printed value at the boundary, or a min/max/NaN check on a tensor.
5. **Change one thing, re-run, verify.** Fix on the fast repro, confirm, then re-widen to the full payload. Revert anything that didn't help.
### Make the loop fast and reliable
- **Atomic debug cycles.** Make each iteration a single self-contained, repeatable command (setup + run in one shot) so you never hand-redo multi-step state. See [atomic debug cycles](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#atomic-debug-cycles).
- **Automate diagnostics, minimize typing.** Alias the repro and your most-used commands; one keystroke to re-run. See [alias frequently used commands](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#alias-frequently-used-commands) and [automate diagnostics](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#automate-diagnostics-minimize-or-avoid-typing).
- **One-liner programs.** `perl`/`awk`/`python -c` to slice logs, extract fields, transform data on the fly instead of writing throwaway scripts. See [the power of one-liner programs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#the-power-of-one-liner-programs).
- **Juggle configs cleanly** when running many debug experiments so results don't get confused. See [juggling multiple sets of configs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#juggling-multiple-sets-of-configs-for-different-debug-experiments).
### Localization techniques
- **Am I editing the right file/class?** Insert a guaranteed break where you *think* execution goes; if the program doesn't die, you're in the wrong file/class/env:
```python
def suspect():
die # NameError -> proves this code runs; the traceback also names the caller
```
`traceback.print_stack()` shows callers without stopping (useful when the same function is reached via many paths). See [am I editing the right file and the right class?](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#am-i-editing-the-right-file-and-the-right-class).
- **Bisect a regression.** `git bisect start / bad / good <rev>` walks commits automatically to the one that broke things - script the test for `git bisect run`. See [finding a breaking commit by bisecting](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#finding-a-breaking-commit-by-bisecting-revisions).
- **Small/synthetic payload first.** Use tiny or synthetic inputs; switch to real data only when the bug is data-dependent. See [real vs random vs synthetic data](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#real-data-vs-random-data-vs-synthetic-data).
- **Race conditions.** Reordering/timing bugs hide under async; forcing synchronous execution can expose (or mask) them - note which. See [avoiding race conditions](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#avoiding-race-conditions) and [async vs sync mode](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#async-vs-sync-mode).
### Reproducing resource & environment issues
- **Cap resources on purpose** to test failure paths: emulate a nearly-full disk, limited CPU RAM, or limited GPU memory. See [running out of resources](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#running-out-of-resources-disk-space-cpu-memory-gpu-memory).
- **Watch resources live.** `watch -n1 nvidia-smi` / `free -h` / `df -h` in a second visible terminal to correlate a hang/OOM with what the machine is doing. See [watching and reproducing resource issues](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#watching-and-reproducing-resource-issues).
- **Inject `sleep`** to freeze a program at the interesting moment so you can attach a debugger or snapshot state. See [uses for sleep](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#uses-for-sleep).
- **HPC/SLURM:** keep the allocation and re-run with `srun` instead of re-`sbatch`-ing to cut per-iteration overhead. See [SLURM salloc and srun fast debug combo](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#slurm-salloc-and-srun-fast-debug-combo).
## Unix / shell
Full chapter: [Unix Tools for Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md).
- **Make shell scripts fail loudly and traceably:**
```bash
set -e # abort on first error
set -o pipefail # a failing command anywhere in a pipe fails the whole pipe
set -u # abort on undefined variables (catches typos)
set -x # trace: print each command with expanded values as it runs
set +x # turn tracing back off around a noisy region
```
Combine as `set -euo pipefail`. See [controlling script execution](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#controlling-script-execution).
- **`strace` - trace system calls** to see what a program actually does (files, network, why it's stuck):
```bash
strace python -c "print('hi')" # trace from the start
strace --pid PID # attach to a running/stuck process
strace -o log.txt -f torchrun ... # -f follows fork질문하고, 팁을 나누고, 안 되는 점을 알려 주세요. "도움이 됨"이 많은 순서예요.