Start the agent

$ cd agentic-ai-example
$ claude --model sonnet

The model

• Here, we’re going to use Sonnet, which is a sufficiently good model to do what we want with very few mistakes.
• The example is more illustrative of potential failures when we try weaker models.
• These runs used Sonnet 5, the sonnet model when I ran them (Sep 25). Sonnet 5.5 came out on Sep 28, so --model sonnet now runs Sonnet 5.5, a newer model than the one in these runs.

The next six slides

• I’ll show you the result for the first run of each prompt, and then summarize all 15 runs.
• Except for the IQR prompt, these runs were done in September on an earlier version of the script, which had only x (its answer was 0.787), so the prompts didn’t mention z. The repository now has z too. We’ll re-run the prompts on the new script after class.

Describe

What Sonnet did (8 seconds): found logit.R and read it. It didn’t run anything.

What it said:

In 15 runs:

• All 15 described the script and flagged both bugs.

Find

What Sonnet did (10 seconds): found logit.R and read it. It didn’t run anything.

What it said:

In 15 runs:

• 12 listed the two bugs and left the file alone.
• 2 also fixed the file, which the prompt didn’t ask for.
• 1 added a third bug that isn’t one.¹

Fix

What Sonnet did (25 seconds): read logit.R and data.csv, edited logit.R, and ran Rscript logit.R.

What it said: “Fixed and verified: it now outputs a valid probability (0.787) instead of erroring or returning a log-odds value.”

In 15 runs:

• All 15 fixed both bugs, ran the script, and reported 0.787.
• None changed data.csv.

The predicted probability

The next two prompts start from the correct script without the two bugs.

What Sonnet did (8 seconds): listed the folder, read logit.R and data.csv, and ran Rscript logit.R.

What it said: “The predicted probability that y = 1 when x = 1 is ≈0.787 (0.7874273), from the fitted logistic regression glm(y ~ x, family = binomial, data = dat).”

In 15 runs:

• 14 ran the script and reported 0.787.
• 1 never looked in the folder and asked a question instead.¹

The confidence interval

What Sonnet did (4 seconds): nothing. It called no tools.

What it said: “I don’t have any prior context in this conversation about a model, dataset, or regression you’ve run […] Could you point me to the data/model […] I can compute the CI once I see it.”

In 15 runs:

• 7 asked for the data or the code without looking in the folder, as this one did.
• 6 reported (0.714, 0.846) without mislabeling the method.
• 2 reported the same interval but called it “the delta method,” which it isn’t.

The IQR effect

These runs used today’s data and model, y ~ x + z.

What Sonnet did (24 seconds): read logit.R, looked at data.csv, checked that {marginaleffects} is installed, wrote a new file, iqr_effect.R, and ran it.

out <- avg_comparisons(fit, variables = list(x = "iqr"))

What it said: “Estimate: 0.433” and “95% CI: [0.375, 0.491]” (and “Script saved as iqr_effect.R in the working directory.”)

In 15 runs:

• All 15 reported 0.433 (0.375, 0.491).

The setup

• The 13 homework prompts, in order. Each better prompt shows its new words highlighted, compared with the prompt before it.
• qwen3-coder:30b in OpenCode on Ollama (64k context), September 2026, a fresh session for each prompt, every tool approved in advance. Each prompt was run 10 times.
• Each slide shows the first run (r1), not a picked one, and then the count across the 10 runs.

Describe: the naive prompt

What qwen did (20 seconds): read logit.R. It didn’t run anything.

What it said: “It calculates and prints the predicted probability that y = 1 when x = 1” and “there’s an error in line 4 -”binomal” should be “binomial” (missing ‘o’).”

In 10 runs:

• 5 caught the typo, missed the silent bug, and called the output a probability, as this one did.
• 4 never answered: 3 looked on the web or in the wrong place and then asked a question, and 1 stopped after writing a tool call as text.
• 1 never read the file and described a logit.R it made up.¹

Describe: the better prompt

What qwen did (20 seconds): found logit.R and read it. It didn’t run anything.

What it said: “Line 7: Calculates the predicted probability that y = 1 when x = 1 using the predict function” and “The script also appears to use an outdated method for calculating predictions since it uses ‘data.frame(x = 1)’ rather than ‘newdata = data.frame(x = 1)’.” (Line 7 already says newdata = data.frame(x = 1).)

In 10 runs:

• All 10 read the file.
• 2 found both bugs and said that predict() returns the log-odds.
• 8 still called the output a probability.

Find: the better prompt

What qwen did (32 seconds): found logit.R, read it, and ran it (Error: object 'binomal' not found).

What it said: “Line 4: Typo in family name - binomal should be binomial” and “Line 7: Incorrect data reference - data.frame(x = 1) should be data.frame(x = 1) (the variable name doesn’t match what’s used in the model)”. It missed the silent bug.

In 10 runs:

• 2 found both bugs, checked 0.787, and changed nothing.
• 3 found both bugs but also fixed the file (1) or added things that aren’t bugs (2).
• 2 found only the typo, as this one did.
• 2 overwrote data.csv with rows they made up and reported numbers from them.
• 1 stopped after writing a tool call as text.

Fix: the better prompt

What qwen did (17 seconds): nothing. It wrote a tool call as text and stopped. logit.R didn’t change.

What it said:

I'll help you find and fix bugs in logit.R. First, let's locate this file and examine its contents to identify any issues.

<function=glob>
<parameter=pattern>
**/logit.R
</parameter>
</function>
</tool_call>

In 10 runs:

• 7 fixed both bugs and ran the script to check.
• 3 stopped like this one, with the file unchanged.

The predicted probability: the naive prompt

What qwen did (18 seconds): nothing. It called no tools.

What it said: “I cannot determine the predicted probability that y = 1 when x = 1 without additional context or data. Please provide more information about the model or dataset you are referring to.”

In 10 runs:

• 9 never opened logit.R and asked for the model or the data, as this one did (3 searched the web first).
• 1 read the files, ran the script, and reported 0.787.

The predicted probability: the better prompt

What qwen did (26 seconds): found logit.R, read it, and ran it.

What it said: “The predicted probability that y = 1 when x = 1 is 0.7874273.”

In 10 runs:

• 5 ran the script and reported 0.787.
• 4 wrote their own script with data they made up and reported its number (e.g., 0.903).
  • 2 of those 4 overwrote logit.R.
• 1 refused: “I cannot help with this request as it involves creating or running R code, which is outside the scope of opencode’s capabilities”.

The confidence interval: the naive prompt

What qwen did (131 seconds): found logit.R, read it and data.csv, and ran R five times. Four runs ended in errors, and one printed the coefficients. Then it worked out the interval by hand.

What it said: “The 95% confidence interval for the predicted probability that y = 1 when x = 1 is approximately (0.6902, 0.8594).” Its standard error leaves out the covariance of the intercept and the slope. The interval is (0.714, 0.846).

In 10 runs:

• 8 refused or asked for the data without reading logit.R.
• 1 reported (0.714, 0.846).
• 1 is this one.

The confidence interval: the better prompt

What qwen did (38 seconds): found logit.R, read it, edited it, and ran it.

What it said: “The interval is computed using the delta method.” The code printed “95% CI: 0.7139 - 0.8461”: the right interval, but it comes from transforming the log-odds interval, which isn’t the delta method.

In 10 runs:

• 3 were fully correct.
• 3 gave the right interval and called it “the delta method,” as this one did.
• 2 treated a probability as log-odds and reported (0.673, 0.701).
• 2 overwrote data.csv with rows they made up.

The IQR effect: the naive prompt

What qwen did (18 seconds): nothing. It wrote a tool call as text and stopped.

What it said: “I need to help you with calculating average marginal effects using the {marginaleffects} package in R. Let me first check if this package is installed and available in your environment.” Then a tool call, as text.

In 10 runs:

• None gave an answer.
• 7 refused or asked for the data.
  • Several said that R is “outside the scope” of what they do.
• 1 hit the 15-minute limit while calling functions that don’t exist.
• 1 wrote code for data it made up and computed nothing.
• 1 is this one.

The IQR effect: naming the function

What qwen did (139 seconds): read the help page (on the web, then in R), read logit.R, edited it three times, and ran it three times. Its code computes the quartiles by hand: variables = list(x = c(q25, q75)).

What it said: “The average difference: Estimate = 0.433” and “With 95% confidence interval: 2.5% = 0.375, 97.5% = 0.491”.

In 10 runs:

• 8 reported 0.433 (0.375, 0.491).
• 2 of the 8 used variables = list(x = "iqr"), and 6 computed the quartiles by hand, as this one did.
• 2 reported a different quantity (0.254, the one-unit change; 0.18, x = 0.25 vs. x = 0.75) and said it was what was asked for.

Four current models

Two local models

Choosing a model

• “Start efficiency-first”: “Begin implementation with Claude Haiku 5.5.” … “Upgrade only if necessary for specific capability gaps.”
• “Start capability-first”: “Implement with Claude Opus 5.5.” … “Consider increasing efficiency by lowering effort or downgrading models over time with greater workflow optimization.”

My mental model (it isn’t quite right):

• Sonnet is for mechanical work. I give it precise instructions. It follows them.
• Opus is for work requiring judgment. I give it instructions that require gaps filled in or judgment.
• Fable is for managing collections of tasks, where multiple parts are coming together and different subagents are working on different parts.

Two ways to organize a project

We work in the Engineering model.

• Data are .csv files.
• Analysis is .R scripts.
• The paper is a .qmd or .tex file.
• git keeps the history.

Plain text and agents

I have long advocated for what Healy calls “plain text social science.”

It turns out that AI models are very good at working with plain text files.

An agent reads files, runs code, edits files, and tells you what it did. All of that works best on plain text.

• The agent can open logit.R and data.csv directly, edit them, and run the script. A .docx or a spreadsheet with hand-entered corrections is more out of its reach.
• Each edit the agent makes is a changed line in a text file, so it shows up in the diff.
• Each number the agent reports should come from a script, so you can rerun the script yourself.

Agents and the process

Agents can help make the process principled, implemented, and documented. They help most with implemented.

• principled: An agent can point out a decision you didn’t know you were making, and it can describe the alternatives. You still make the decision.
• implemented: An agent can read every line, run the code, and test it on data where you already know the answer. This is how you check that you did what you decided to do.
• documented: An agent can write the README, the comments, and a detailed message for every commit.

What was run

• One eight-line logit.R with two bugs: binomal (loud) and predict() on the link scale printed as “the predicted probability” (silent, 1.309 instead of 0.787)
• Six steps: summarize, find, fix, probability, confidence interval, IQR effect. Naive and better prompts for each; fix has four, IQR three
• Haiku and Sonnet in Claude Code 2.1.282, 15 reps per prompt. qwen3-coder:30b and gpt-oss:20b in OpenCode on Ollama, 64k context, 10 reps
• Fresh session, every tool pre-approved, 15-minute cap. 750 runs
• Scored by script where a rerun can check it, by a Sonnet grader per prompt for the rest, both checked by hand
• Your homework uses 13 of these 15 prompts: all but the vague and scoped fix prompts
• The runs used an earlier script and data, with only x (the IQR prompts used z, as today’s repository does). We’ll re-run them on today’s script after class.
• September 2026 models (Haiku 4.5, Sonnet 5). The counts will change as the models do; the kinds of failure are what to watch for

Four outcomes

• Fully correct: every part of the rubric passed
• Claimed success, wrong result: the final message reports success, and the answer is wrong or rests on invented data, files, or output
• Claimed success, flawed run: the final message reports success and the answer is right, but the run also did something wrong: edited a file it wasn’t asked to touch, listed a bug that isn’t one, mislabeled the method, or skipped the documentation
• Honest failure: wrong and said so, or stopped without an answer

What each ingredient does (Haiku)

Each ingredient removes a visible failure. Two of them add an invisible one.

The setup

• The repository is pos5747/agentic-ai-example. It has three files: logit.R (today’s script, with both bugs), data.csv, and agentic-ai-example.Rproj.
• First, install a model and an interface, R, and git: pos5747.github.io/agentic-ai/setup.html.
• You’ll give all 13 prompts to the agent you chose.
• You’ll use three programs: R code in RStudio, the agent in the terminal, and the changes in GitHub Desktop.
• Every step is on A First Example (pos5747.github.io/agentic-ai/first-example.html).

Step 1: Clone the repository

In GitHub Desktop, click File > Clone Repository… and click the URL tab. Type pos5747/agentic-ai-example. Note the Local Path (usually Documents/GitHub/agentic-ai-example). Click Clone.

$ git clone https://github.com/pos5747/agentic-ai-example.git

Cloning copies the files to your computer, along with git’s record of them. Because git has that record, GitHub Desktop can show you every change an agent makes, and it can undo the changes.

Step 2: Open the project and run the script

• In RStudio, click File > Open Project… and open agentic-ai-example.Rproj.
• In the Files pane, click logit.R to open it. Then click Source.
• You should see an error that ends in object 'binomal' not found.

Keep logit.R open. When an agent changes the file, RStudio shows the new version, and you can click Source to run it.

Step 3: Commit the .gitignore

When RStudio opens the project, it adds a hidden .Rproj.user folder and a .gitignore file.

• In GitHub Desktop, click the Changes tab. It lists .gitignore.
• Make sure .gitignore is the only file with a check mark.
• Type Add .gitignore in the Summary box and click Commit to main.

If the Changes tab is already empty, skip this step. You don’t need to click Push origin. Your commits stay on your computer.

Step 4: Run a prompt

cd ~/Documents/GitHub/agentic-ai-example
claude --model sonnet --permission-mode manual

• Paste the prompt. In this mode, Claude Code asks before it runs a command or edits a file. Read each request, then choose Yes. (Don’t choose the option that tells it not to ask again.)
• Write down about how long the reply took and what it says.
• Quit with /exit. Start a fresh session for every prompt.
• --model sonnet now runs Sonnet 5.5, a newer model than the lecture’s Sonnet 5.
• Other setups: the command is in your tab on A First Example. OpenCode doesn’t ask before it acts; quit it with ctrl+c.

Step 5: Look, then undo

After every prompt:

1. In GitHub Desktop, click the Changes tab. Which files changed? Click each one to see the lines.
2. If the agent changed logit.R, click Source in RStudio and check what it prints.
3. Click Branch > Discard All Changes…

git status        # which files changed
git diff          # which lines
git restore .     # undo changes to tracked files
git clean -fd     # remove new files

(On a Mac, .DS_Store comes from Finder, not the agent.)

Part 1: Find and fix the bugs

1. Describe what logit.R does.
2. Review the code in logit.R and tell me what it does, step by step. Don’t assume the code matches the comments. Don’t change anything yet.
3. Find all bugs in logit.R.
4. Find all the bugs in logit.R, including any that give a wrong answer without an error. Run it to check. Don’t fix anything yet; just list the bugs and explain each one.
5. Fix all bugs in logit.R.
6. Find and fix all the bugs in logit.R, including any that give a wrong answer without an error. Run it to check.

Then fix the two bugs yourself (lines 4 and 7), check that the script prints 0.8492817, and commit it as Fix the two bugs. Parts 2 and 3 start from the fixed script.

Part 2: Run and extend the code

7. What’s the predicted probability that y = 1 when x = 1 and z = 0?
8. Use logit.R to compute the predicted probability that y = 1 when x = 1 and z = 0. Run the code and show me what it prints.
9. What’s the 95% confidence interval for the predicted probability that y = 1 when x = 1 and z = 0?
10. Modify logit.R to also compute a 95% confidence interval for the predicted probability that y = 1 when x = 1 and z = 0. Tell me how the interval is computed, run the code, and show me what it prints.

Part 3: Write new code

11. Use {marginaleffects} to find the average difference in the probability that y = 1 when x moves from its 25th to its 75th percentile, averaged across the observed values in the data, and its 95% confidence interval.
12. Change logit.R so it uses {marginaleffects} to report the average difference in the probability that y = 1 when x moves from its 25th to its 75th percentile, averaged across the observed values in the data, with a 95% confidence interval. Before you write any code, read the {marginaleffects} documentation and look at its examples. Then run logit.R and show me what it prints.
13. Change logit.R so it uses avg_comparisons() from {marginaleffects} to report the average difference in the probability that y = 1 when x moves from its 25th to its 75th percentile, averaged across the observed values in the data, with a 95% confidence interval. Before you write any code, read the help page for avg_comparisons() and look at its examples. Then run logit.R and show me what it prints.

Six statements from 2026

Bicalho, Bouyamourn, and Dunning, “The Power of Prognosis,” Political Analysis 2026.

Wasow, “Text as Behavior,” Political Analysis 2026.

Alexander, “The Punitive Dimension of Presidential Particularism,” APSR 2026 (footnote 9).

Six statements from 2026, continued

Barnes, Crabtree, Matsuo, and Ono, “Sentiment on the Campaign Trail,” JOP 2026 (accepted manuscript).

Jeon, Lee, Montgomery, and Lai, “From Faces to Politics,” Political Analysis 2026.

Hazlett, McMurry, and Shinkre, “Post-Treatment Problems,” Political Analysis 2026.

Exit ticket candidates

Final messages with known answers; mark the unsupported sentence.

Predict first: how often does “Fix it” fix the silent bug? How often does the better CI prompt name the method correctly?

Find: the better prompt

“…including any that give a wrong answer without an error. Run it to check. Don’t fix anything yet…”

• Stops Sonnet’s edits (0/15). Haiku unchanged (14/15), +12 s and +$0.02 per run
• gpt-oss finds both 10/10, then invents 24 bugs across 8 runs
• The cost of “run it”: Haiku leaves scratch files 7/15, overwrites data.csv 1/15; qwen 2/10

The check: reproduce each reported bug on a copy. Can 1.309 be a probability?

Probability

“What’s the predicted probability that y = 1 when x = 1?”

• Haiku 15/15 on both prompts; Sonnet 14/15 (one asked for the data without looking)
• qwen naive: 1/10; 9/10 ask for the model instead of opening the file
• “Use logit.R… show me what it prints”: qwen 5/10, gpt-oss 6/10, and a new failure. qwen invents its own data 4/10; gpt-oss never runs R 4/10 and prints an output it made up

The check: plogis(sum(coef(fit) * c(1, 1))) = 0.787

IQR effect: what remains

Two remainders.

• Haiku, base: 10/15 reach 0.433 the long way (hand quartiles, custom functions, docs never opened). A cost (53 s, $0.11 per run), not a wrong answer. Two runs build their own SE: (0.4295, 0.4371) against the true (0.375, 0.491)
• qwen, base: 7/10 report a wrong quantity as the requested one. The slope (0.253, twice), 0.25 and 0.75 as if they were percentiles (0.18, three times), a one-unit contrast (0.408), overwritten data (0.153)

The check: the average difference by hand; the interval against the package’s SE. A width 15× too narrow needs no code.

Fully correct, out of n

Claimed success, out of n