A cheap weekly check for price drift on an AI tools directory

bestAIpacks lists a starting price for about a hundred AI tools. Vendors change pricing without telling anyone, and a directory that says $19/mo when the real number is $29 is worse than one that says nothing at all.

I didn't want to build a scraper farm. Vendor pricing pages are JavaScript-heavy, they change layout, and half of them block datacenter IPs anyway. What I wanted was a cheap weekly signal that says "this one probably moved, go look."

The check

For each tool, a script runs one web search through Tavily with a query like <tool name> pricing monthly plan cost 2026, takes the top three results at basic depth, and hands the excerpts to a language model along with the price we have listed. The model answers with one of OK, DRIFT, or UNSURE and a one-sentence reason.

A cron runs it weekly and emails a table with the DRIFT rows highlighted. The script never edits a review. Someone reads the email, opens the vendor page, and fixes the frontmatter in the CMS if the price really moved.

Where the first version went wrong

The first judge was gpt-4o-mini, because the job looked easy. It flagged drift all over the place, and when I dug into the false flags the causes had little to do with the model.

Monthly versus annual. Vendor pages default to the discounted "per month, billed yearly" figure. Our directory lists the monthly price billed monthly, because that's what someone with no commitment pays. So the search results would say $16, our page would say $20, and the model would call it drift on a number that hadn't changed. Same tier, different billing basis.

Quote-based vendors. Some tools don't publish a price. Aggregator sites publish guesses. The model would read a third-party guess, compare it to our blank, and flag drift on a number no vendor ever stated.

Both were fixable in the prompt. The judge prompt now spells it out: a monthly-versus-annual gap on the same tier is OK. A number that only shows up on a third-party site is UNSURE, never DRIFT. A DRIFT verdict has to compare monthly to monthly and the gap has to be more than about 15 percent.

With those rules in the prompt, gpt-4o-mini kept ignoring them. Swapping in Claude Sonnet as the judge fixed it. Writing the rules down was necessary and the model still had to be good enough to follow them.

The convention underneath

The check only works because every price on the site is on the same basis. That took a written convention:

• from is the monthly price billed monthly, never the annualized figure.
• If the annual discount is worth noting, it goes in parentheses with the exact phrase "billed annually" so the checker can strip it.
• Introductory pricing anchors on the ongoing rate, not the teaser.
• No public price means from: null. Nobody types in a guess.

That's documented in the repo. When an editor enters an annual figure by mistake, the checker throws a false flag the following week, and that flag is how we catch it.

Spending search credits where they matter

Checking a hundred tools every week burns credits on ninety that haven't changed. The default run now only checks tools whose lastVerified date is more than 30 days old, at basic search depth with three results. Passing a slug on the command line forces a check, so a suspected change can be confirmed on demand.

What it is and isn't

This is a nudge, not a tracker. The public AI tool price tracker on the site shows the verified dates so readers can see how old a number is, and the how we review page says a human checks before anything changes.

If you run a directory with prices in it, the convention did more work for me than the tooling. Pick one basis, write it down, and even a cheap check will tell you when something moved.