How Accurate Are AI Tool Directories? We Audited Our Own 366 Listings

Turning the ruler on ourselves#

A directory is only as credible as its details — a launch date, a price, a parent-company name — and readers quote them as fact. Almost nobody re-checks those fields: they are typed in once, indexed, copied onward, and the error sets. So we audited our own tool directory against public sources. This is what we found.

Finding one: we did not know most of our launch dates#

Of 366 listings at the start, 273 (75%) ended in "the first of some month", and 152 were exactly 1 January. Those were not birthdays but placeholders for "we only know the year". The site rendered them as full dates and passed the same invented precision to search engines through structured data.

The first fix was not to find dates but to stop claiming them: show a year when only a year is known, a month when a month is evidenced, a day only when a source states one. A product that genuinely launched on the 1st now reads one step less precisely, which is the harmless direction for an error. After that came the evidence hunt: 266 records have now been checked against a source (including ones we checked and deliberately left alone), with well over a hundred corrections. Some still only support a year, and we label them as such rather than guessing.

Finding two: the products move, the directory does not#

Six listings pointed at products that no longer exist and were removed: Phind (its domain answers 404 DEPLOYMENT_NOT_FOUND), Supermaven (a dated vendor sunset notice), Clockwise ("no longer be available starting on March 27, 2026", team to Salesforce), Socratic (the domain now renders a Google Lens page), GitHub Models (documentation states it was fully retired on 2026-07-30) and Youper (a vendor notice ending service on 2026-09-30 and deleting data). Two more were duplicate listings of a single product — Codeium, now Devin Desktop, and Bing Image Creator, now served inside Microsoft Designer — and were merged.

Renamed-but-alive products keep their old name in the blurb, so searching it still finds them. Three suspected renames were left untouched because a redirect proves relocation, not a rename date: DreamStudio, Galileo and INK.

Finding three: our related-coverage links were lying#

Tool pages matched article text by substring, which turns short names into skeleton keys: "Pi" matched every Anthropic story, "Exa" matched Alexa+, "INK" matched Thinking. Measured across the corpus, 28 of 131 article-tool mention pairs (21%) were false.

Switching to word-boundary matching removed the junk and exposed the real gap: only 40 of 364 listings have any coverage on this site at all. Those pages used to go silent after the description, or pad the shelf with false links; they now state plainly that the desk has not covered the tool and route the reader to the category index.

Method, and what remains#

Every change required a dated first-party source or a registry timestamp (an App Store releaseDate, a first PyPI or npm publish, a first GitHub release). An archive capture only bounds a date from below, so it can justify a year, never a month. Press coverage is enough to raise a question, never enough to rename a product — two agent-supplied conclusions were rejected on that basis. The audit state now lives in the repo as a checked-records list and a selector for what remains, so the work can be resumed by someone else instead of living in one person's memory.

The honest remainder: 324 tools we have never written about, and prices that drift — Zoom's assistant moved from "included" to a paid licence mid-audit. Any one-off verification expires; the claim "we checked" is only good until the next rename.